Governance end to end

By the end of this chapter you can govern one table of a data platform with code, end to end. You'll catalog it with an owning team and a classification for every column, capture the lineage of a pipeline that copies it, decide who may read which of its rows and enforce that decision on a query, and see a pull request fail because it added a column nobody classified. Each piece is small and runs in seconds, and five later chapters each make one of them solid.

This book is for data, analytics and platform engineers who already build pipelines and warehouses and have been asked to make them governed. You're comfortable with SQL, Python and Parquet or a lakehouse table, and you don't need governance theory: you need governance that runs, is tested, and is reviewed in pull requests like the rest of your code. By the end of the book, you can govern a data platform with code that runs in CI:

• a catalog where every table has an owner and every column a classification;
• personal data found automatically, with what the detector misses measured;
• lineage captured from pipelines;
• access policies as tested code, enforced as row filters and column masks;
• retention and erasure that remove data;
• an audit trail;
• checks that fail a pull request;
• the same rules applied to data that trains or grounds AI.

This book follows one case throughout: an online learning platform that sells professional courses to adult learners, which we call the platform. Its warehouse is a catalog of Iceberg tables, each under the namespace of the team that runs it: learners and instructors under crm, courses, enrollments, learning events and course reviews under learning, payments under billing, and support tickets under support. SQL pipelines build reporting tables from them. Support agents, instructors, marketing, finance, data scientists and the platform's course assistant, its AI, each need something different from those tables, and that's what makes the platform worth governing.

The platform's data, which every example in this book reads, is generated. A script makes it once from fixed seeds with Faker, a library for fake data, and the files are committed with a data card that says how they were made. No real person is in the data. Its email addresses are on the domains reserved for examples, so they can't reach anyone, and its IP addresses are in the ranges reserved for documentation, so they point at no one. Its phone numbers, card numbers and IBANs are random in a valid format, so one may match a number or an account in service: never call, message, charge or pay them. The card also lists what is planted for the examples to find, such as full card numbers in a legacy column and personal data in free text.

Every example runs offline, in one Python 3.12.15 process, on pinned libraries: PyIceberg 0.12.0 for the Iceberg catalog and its tables, DuckDB 2.0.0 for queries, sqlglot 30.24.0 to parse SQL, openlineage-python 1.54.0 for lineage, and regopy 1.5.2 for policies written in Rego, the policy language of Open Policy Agent (OPA). Six services that a production platform runs don't run here, and each has a stand-in:

• Trino, the SQL engine every tool would query through: DuckDB, behind the book's enforcement layer, which rewrites each query before DuckDB runs it.
• OPA's server, which Trino asks for access decisions (Trino 484, OPA access control): regopy, evaluating the same Rego inside the Python process.
• Spark, or whatever engine runs the pipelines at scale: DuckDB, running each pipeline's SQL file.
• A catalog server or a cloud catalog, serving table metadata through an API such as the Iceberg REST catalog's (REST catalog spec, 1.12.0): PyIceberg's SQL catalog, where a SQLite file records where each table's metadata is, and the metadata files sit beside the data on the local disk.
• A lineage server: a file of OpenLineage events, which the book's code reads back into a graph.
• The CI service and git: two folders, one for the main branch and one for a pull request's branch.

The examples teach the open standards those services work with, Iceberg's table metadata, OpenLineage and Rego, so what you build maps onto them.

On the platform, governance starts with a spreadsheet. Someone listed the columns of crm.learners, the table that holds every learner's name, email, phone number and address, with a classification beside each, and that's all there is. No team is named as the table's owner. A nightly pipeline copies learners' names and emails into a table for marketing, and nothing records that it does. Support agents each serve one region, but a query on the warehouse returns every region's learners. And the CRM team is about to add a column. This chapter answers each of those with the code that the later chapters build, at the smallest scale that works: one table, one pipeline, one rule and one check. Only one of the four is a policy engine; the others are metadata, lineage events and a test, kept in the same repository and reviewed the same way.

Five later chapters each make one piece solid. Chapter 2 keeps the catalog in step with schema changes and measures a local model's drafted metadata. Chapter 4 uses lineage for impact analysis and to propose classifications. Chapter 5 writes the whole policy, with its tests, and chapter 6 enforces it on every query. Chapter 10 runs the checks of chapters 2 to 7 in CI, with owners' approvals and waivers. Four chapters add what this one leaves out: finding the personal data the metadata misses (chapter 3), retention, consent and erasure (chapter 7), an audit trail (chapter 8), and the same rules for data that trains or grounds AI (chapter 9). The examples use the book's shared modules, files in examples/lib/ that hold the code many examples need, which you can open and read; each section names the ones it uses.

Catalog a table with its owner and classifications

The platform's first problem is its spreadsheet: a list of columns that nothing checks. A catalog lists a platform's tables and their schemas, and governance needs two facts the data doesn't carry: who answers for each table, and how each column must be protected. Kept in a spreadsheet or a catalog server's web page, those facts change without review, and no test can say whether they're complete. Kept in a file in the repository, beside the pipelines that create the tables, every change to them is a diff someone approves, and a test can compare the file with the catalog.

The platform's file is governance/metadata.toml. It gives each table an owner, a domain, the part of the business the table belongs to, and a description, and each column a description and a classification. Catalog tools mean different things by an owner, as chapter 2 compares, and we use one meaning throughout: the team accountable for the table, never a person, so the answer survives someone leaving. A classification is one of four values, from least to most protected:

• public: may be shown to anyone outside the platform;
• internal: for the platform's staff, and doesn't identify a person by itself;
• personal: identifies a person, or tells something about one;
• sensitive: data whose leak would directly harm someone, such as a card number.

Columns are keyed by their Iceberg field ID, which survives a rename, with the name beside it for people:

[tables."crm.learners"]
owner = "team-crm"
domain = "crm"
description = "Every learner who has signed up, with their contact details and consents. One row per learner."

[tables."crm.learners".columns]
3 = { name = "email", classification = "personal", description = "The email the learner signs in with." }

Three of the book's shared modules do the work. lib.catalog.open_catalog builds a fresh Iceberg catalog of the platform's tables in a temporary folder, which it removes when the example exits. lib.metadata loads the file, writes it into the catalog with apply, and checks the catalog against it with problems, which walks every table the catalog holds and reports a table with no metadata or no owner, and a column with no classification. Catalog crm.learners:

1"""Catalog crm.learners: its owner, domain and classifications come from governance/metadata.toml, the reviewed file."""2from collections import Counter34from lib import metadata5from lib.catalog import open_catalog67platform = open_catalog(["crm.learners"])            # a fresh Iceberg catalog holding the learners table8described = metadata.load()9learners = described["crm.learners"]10print("owner:", learners.owner)11print("domain:", learners.domain)12counts = Counter(column.classification for column in learners.columns.values())13print("columns classified personal:", counts["personal"])14print("personal columns:", ", ".join(c.name for c in learners.columns.values() if c.classification == "personal"))15print("columns classified internal:", counts["internal"])1617# Write the descriptions into the table's schema, and the owner and domain into its properties.18metadata.apply(platform, described)19table = platform.load_table("crm.learners")20print("table properties:", ", ".join(f"{key}={value}" for key, value in sorted(table.properties.items())))21print("email's description, in the Iceberg schema:", table.schema().find_field("email").doc)2223# The check: every table in the catalog has an owner, and every column one of the four classifications.24print("problems the check finds:", len(metadata.problems(platform, described)))

owner: team-crm
domain: crm
columns classified personal: 7
personal columns: full_name, email, phone, city, street_address, date_of_birth, notes
columns classified internal: 8
table properties: domain=crm, owner=team-crm
email's description, in the Iceberg schema: The email the learner signs in with.
problems the check finds: 0

The table's owner is team-crm, in the domain crm. Of its columns, 7 are personal: full_name, email, phone, city, street_address, date_of_birth, notes. notes is among them although its name says nothing, because support agents type whatever they like into it. The other 8 are internal: the learner's ID, two dates, their country, their two consent flags, their account's status and their support region. After apply, the table's properties read domain=crm, owner=team-crm, and email's description sits in the Iceberg schema, where any reader of the table finds it: "The email the learner signs in with." The classifications stay in the file, because that is where the access policy and the checks read them. And problems finds 0 problems: the catalog's one table has an owner, and every column one of the four classifications.

That zero is what a spreadsheet can't give you. It comes from walking the catalog, so a column the file doesn't describe would show up as a problem, as work_email does in the section on spreadsheets.

Working rule: Keep owners, domains, descriptions and classifications in one file in the repository, reviewed like the pipelines, with a team as every table's owner and one of a closed list of classifications on every column. Write the descriptions into the catalog, so every reader of a table sees them, and test the catalog against the file by walking the catalog, never the file. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it.

Capture one pipeline's column lineage

The catalog says what a table holds. Lineage says where its data came from and where it went, and governance needs both: a table built from crm.learners may or may not copy its emails. OpenLineage, an open specification for lineage, describes a job, such as a pipeline, its runs, and the datasets each run reads and writes. A run reports itself in events: a START when it begins, then one that ends it, such as COMPLETE or FAIL. Events carry facets, OpenLineage's name for blocks of metadata attached to a run, a job or a dataset.

The column lineage facet, on the output of a COMPLETE event, lists for each output column the input columns it came from, each with a type. DIRECT means the "output column value was somehow derived from inputField value", and INDIRECT that the "output column value is impacted by the value of inputField column, but it's not derived from it" (column lineage facet): a join or a filter deciding which rows exist, or a grouping, sort, window or condition.

The client library computes no lineage of its own, as chapter 4 finds in its source: it sends what you give it. So the book's module lib.lineage parses each pipeline's SQL with sqlglot, adds the INDIRECT inputs that sqlglot's own lineage leaves out, and wraps lib.pipelines.run, which runs one of the platform's SQL pipelines on DuckDB and writes its output to the catalog: START before the run, COMPLETE with the lineage after it. The events go through OpenLineage's file transport to one file, stamped with a fixed time and a run ID derived from that time and the job's name, so the example writes the same bytes every time.

The platform's marketing_audience pipeline is one create table ... as select ... statement that writes marketing.audience, the list of learners marketing may email. It selects each learner's ID, name, email and country from crm.learners, joins each learner to their favourite course category, ranked from their enrollments, and keeps the learners for whom l.marketing_consent and l.status = 'active' holds. Run it with lineage, read the lineage back from the events alone, as a lineage server would, and count what the table holds with lib.warehouse.connect, which gives DuckDB a view of each catalog table:

1"""Run the marketing_audience pipeline with OpenLineage run events, then read its column lineage back from the events."""23from lib import lineage, metadata, warehouse4from lib.catalog import open_catalog5from lib.platform import scratch67platform = open_catalog(["crm.learners", "learning.courses", "learning.enrollments"])8events_file = scratch() / "events.jsonl"9lineage.run(platform, "marketing_audience", lineage.client(events_file))10events = lineage.read_events(events_file)1112print("events:", ", ".join(f"{e['eventType']} of {e['job']['name']}" for e in events))13print("the run read:", ", ".join(dataset["name"] for dataset in events[0]["inputs"]))14print("the run wrote:", events[-1]["outputs"][0]["name"])1516graph = lineage.graph(events)17for column in ("full_name", "email"):18    inputs = [e for e in graph.edges if e.target == f"marketing.audience.{column}"]19    print(f"marketing.audience.{column} comes from:", ", ".join(f"{e.source} ({e.type} {e.subtype})" for e in inputs))20for subtype in ("FILTER", "JOIN"):21    shaping = [e.source for e in graph.edges if e.target == "marketing.audience" and e.subtype == subtype]22    print(f"its rows are shaped by, INDIRECT {subtype}:", ", ".join(shaping))23duckdb = warehouse.connect(platform)                  # DuckDB on the catalog's tables, to count what the join dropped24consenting = "select count(*) from crm.learners where marketing_consent and status = 'active'"25print("consenting, active learners in crm.learners:", f"{duckdb.sql(consenting).fetchall()[0][0]:,}")26print("rows in marketing.audience:", f"{duckdb.sql('select count(*) from marketing.audience').fetchall()[0][0]:,}")27reached = lineage.columns_reached(graph, lineage.downstream(graph, "crm.learners.email"))28print("a change to crm.learners.email reaches:", ", ".join(reached))2930# The catalog check, run on the catalog the pipeline wrote to.31for problem in metadata.problems(platform, metadata.load()):32    print("the catalog check finds:", problem)

events: START of marketing_audience, COMPLETE of marketing_audience
the run read: crm.learners, learning.courses, learning.enrollments
the run wrote: marketing.audience
marketing.audience.full_name comes from: crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email comes from: crm.learners.email (DIRECT IDENTITY)
its rows are shaped by, INDIRECT FILTER: crm.learners.marketing_consent, crm.learners.status
its rows are shaped by, INDIRECT JOIN: crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id
consenting, active learners in crm.learners: 1,861
rows in marketing.audience: 1,829
a change to crm.learners.email reaches: marketing.audience.email
the catalog check finds: marketing.audience: no metadata

The file holds two events, START of marketing_audience, COMPLETE of marketing_audience. The run read crm.learners, learning.courses, learning.enrollments and wrote marketing.audience. Column by column, the lineage says what the table's name doesn't: email comes from crm.learners.email (DIRECT IDENTITY), and full_name from crm.learners.full_name (DIRECT IDENTITY), both unchanged copies of personal columns.

Two kinds of INDIRECT input shape the rows. The filters, crm.learners.marketing_consent, crm.learners.status, mean that every row of the table says its learner consented. The joins, on crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id, decide which rows exist too: a learner with no enrollment has no favourite category, so the join drops them. crm.learners holds 1,861 consenting, active learners, and marketing.audience 1,829. Walking the graph from crm.learners.email reaches marketing.audience.email, so renaming the email column, or changing its format, reaches one column of this pipeline's table. With all four of the platform's pipelines sending lineage, chapter 4 finds that it reaches no other.

The catalog check from the section before, run on the same catalog, finds marketing.audience: no metadata. The pipeline created a table that nobody has described, and lineage turns that gap from "a table we know nothing about" into "a copy of two personal columns, chosen by consent". Until someone describes it, the platform's access policy treats every column it can't find a classification for as sensitive. So anyone who reads the copy through the platform's enforcement layer, which a later section introduces, gets no value from it, as chapter 6's who-sees-what test shows.

Working rule: Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run happens without them, and put the output's column lineage in the COMPLETE event, parsed from the job's SQL, with the inputs that join and filter rows as well as those that supply values. Read lineage back from the events, never from the SQL, so the graph shows what ran. Where a job isn't SQL that a parser can read, such as a Python script, write its lineage in the same code, and review the two together.

Enforce one access decision on a query

With the platform's tables cataloged, the next question is who may read what. Policy as code starts from OPA's premise: it "lets you decouple policy from that software service so that the people responsible for policy can read, write, analyze, version, distribute, and in general manage policy separate from the service itself" (OPA philosophy). A policy answers a request with a decision, and the platform's policy answers with a document rather than a yes or no: whether to allow the read, the row filters that keep only the rows the user may see, and the column masks that change how a column is shown. The filters and masks are data, never SQL. A decision does nothing on its own, though. Something between the user and the engine has to apply it to every table a query reads.

On the platform, support agents each serve one region, americas, emea or apac, and need learners' contact details to answer tickets. The policy, governance/policies/access.rego, denies every request by default, allows each group, the role a user holds in governance/users.toml, the tables its work needs, and reads the agent's region from the same file. This is the rule that keeps an agent to their region:

row_filters contains {"column": "support_region", "equals": user.region} if {
	"support" in groups
	table in {"crm.learners", "support.tickets"}
}

lib.policy.decide evaluates the policy with regopy, and lib.enforce holds the enforcement layer: its Enforcer asks the policy about every table a query reads, replaces each with a subquery that applies the decision, and runs the rewrite on DuckDB. lib.enforce.answer is the one function a service calls, and returns rows, never the engine's connection. agent-a05, a support agent in EMEA, asks for three columns of crm.learners, then counts learners by region, and reads their emails, which the example compares with the platform's data as lib.platform.load reads it from the generated files:

1"""One access decision from the platform's Rego policy, and the same decision enforced on one query."""2import json34from lib import policy5from lib.catalog import open_catalog6from lib.enforce import Enforcer, answer7from lib.platform import load89# agent-a05, a support agent in EMEA, asks to read three columns of crm.learners.10request = policy.request("agent-a05", "crm.learners", ["learner_id", "email", "support_region"])11decision = policy.decide(request)12print("allow:", decision["allow"])13print("row filters:", json.dumps(decision["row_filters"]))14print("column masks:", json.dumps(decision["column_masks"]))15print("reasons:", "; ".join(decision["reasons"]))1617# The enforcement layer asks the policy about every table a query reads, then rewrites the query to apply the answer.18layer = Enforcer(open_catalog(["crm.learners"]))19query = "select support_region, count(*) as learners from crm.learners group by support_region order by support_region"20for region, learners in answer(layer, query, "agent-a05"):21    print(f"learners agent-a05 sees in {region}:", f"{learners:,}")2223# Compare the emails the agent gets with the platform's data, without printing any.24stored = load("crm.learners")25emails = dict(zip(stored.column("learner_id").to_pylist(), stored.column("email").to_pylist()))26seen = answer(layer, "select learner_id, email from crm.learners", "agent-a05")27print("emails agent-a05 sees, as stored:", f"{sum(email == emails[learner] for learner, email in seen):,} of {len(seen):,}")28print("learners in the platform's data:", f"{stored.num_rows:,}")

allow: True
row filters: [{"column": "support_region", "equals": "emea"}]
column masks: {}
reasons: support may read crm.learners
learners agent-a05 sees in emea: 1,581
emails agent-a05 sees, as stored: 1,581 of 1,581
learners in the platform's data: 5,000

The decision allows the read, allow: True, with one row filter, [{"column": "support_region", "equals": "emea"}], and no column masks, {}, since support is the group that needs contact details. Its reason says which group allowed it: support may read crm.learners. The agent's query has no filter on region, yet it returns 1,581 learners, all in EMEA, of the 5,000 the platform holds. Their emails come back as stored, 1,581 of 1,581, because the decision masks nothing. The layer applied the region inside the query, so the agent's SQL didn't have to, and couldn't avoid it.

regopy is the Python binding of rego-cpp, Microsoft's C++ implementation of Rego, which supports "v1.18.1 of Rego as defined by OPA" (rego-cpp README at v1.5.2). regopy differs from OPA in ways chapter 5 lists, so OPA stays the reference for anything in production. Chapter 6 shows why the layer replaces whole tables rather than masking columns where a query names them.

Working rule: Write access rules as one policy in the repository that denies by default and returns its decision as data: allow, row filters and column masks. Apply that decision in one layer that rewrites every query before the engine runs it, and give users that layer's answers, never the engine's connection. When an engine can ask the policy itself, as Trino can ask OPA, let the engine be that layer.

Fail the pull request that adds an unclassified column

The catalog check proves the catalog complete on the day it runs, and the platform changes every week. A check that runs when someone remembers it is a report, and reports go unread. Run on every pull request, where its exit code can fail the pull request, the same check becomes a gate, and its findings appear where engineers already work. The authors of Tricorder, Google's platform for code analysis, wrote: "We have repeatedly found that when developers have to navigate to a dashboard or run a standalone command line tool, analysis usage drops off" (Sadowski et al., 2015). That is one company's experience with code analysis, not data, but governance checks meet the same people. A gate also has to judge what the pull request changes, not everything on the platform: a finding already on the main branch is reported with the team that owns it, and fails no pull request that leaves it as it is.

The book's module for this is lib.ci, the platform's governance job. On the platform, a schema change is a migration: an alter table statement in a file in the branch's migrations/ folder. The examples hold each side of a pull request in a folder, a checkout: one of the main branch and one of the pull request's branch. For each side, lib.ci.run builds a scratch platform, a fresh catalog made the way the platform runs: the source tables, the side's migrations, its metadata written into the catalog, and its pipelines. It runs the checks it's given on the branch's platform, runs them on main's alone to learn what is already there, and compares the findings by ID.

This section runs one check, coverage. It is the catalog check, lib.metadata.problems, run on the branch's platform, and for each column nobody classified it adds what a rule on column names suggests, from the rules chapter 3 writes. The CRM team's pull request adds a column for learners' work email, in one migration, and forgets the column's metadata:

1"""The CRM team's pull request adds a column to crm.learners and no metadata for it. One check runs on it: coverage."""2from lib import ci34main = ci.checkout()                      # the main branch: the governance files and pipelines, and no migrations yet5branch = ci.checkout(main)                # the pull request's branch, starting from main6(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")7print("files the pull request changes:", ", ".join(ci.changed(main, branch)))89# Build a scratch platform from each side, run the coverage check on both, and compare their findings by ID.10report = ci.run(main, branch, checks=["coverage"])11for line in ci.lines(report):12    if not line.startswith("change "):    # schema changes are chapter 10's report13        print(line)14for line in ci.totals(report):15    if line.startswith(("findings the pull request introduces", "findings already on main", "exit code")):16        print(line)

files the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
job: failed
findings the pull request introduces: 1
findings already on main: 2
exit code: 1

The pull request changes one file, migrations/0001-add-work-email.sql. The coverage check reports failed, findings 3, new 1, on main 2, waived 0, 12 tables checked: the tables it checked are the platform's eight source tables and the four its pipelines write. Its new finding is the pull request's own: crm.learners.work_email is not classified; its name suggests personal. The name rule matches email, so the finding tells the author what to add, and its ID, coverage/crm.learners.work_email, stays the same from run to run, so it can be discussed and counted.

The other two findings were there before the pull request: ai.review_corpus, and marketing.audience, the table the lineage section found undescribed, reported as no metadata; on main, owner team-platform. Nobody owns either table, so the job names the platform team, which the job's settings make answerable for tables without an owner. Of the findings, 1 is the pull request's own and 2 are already on main, and the job exits with code 1. The pull request fails until the column has a classification, and the two tables on main fail no pull request that leaves them as they are. Chapter 10's job runs the checks of chapters 2 to 7 on the same pull request, 11 of them.

Working rule: Run every governance check as one job on every pull request, required by branch protection, and fail it by exit code on any finding the pull request introduces. Report the findings already on the main branch with their owners, without failing the pull request for them. Put every finding in the pull request itself, under an ID that stays the same from run to run, with what to do about it. Run the same checks on a schedule against the production catalog too, for what changes outside pull requests.

What goes wrong in production

Governance fails without an error in two ways that the pieces above answer: a spreadsheet nobody updates, and a policy that another tool bypasses. The first is why the metadata is code, and the second why enforcement sits in one place.

A governance spreadsheet nobody updates

The platform's governance started with a spreadsheet of tables and columns, with a classification beside each, and a spreadsheet describes the schema on the day it was written. Schemas don't hold still. A study of ten open-source database applications, with more than 160,000 revisions between them, lists first among its major findings that "Database schemas evolve frequently during the application lifecycle" (Qiu, Li and Su, 2013). The study is of application databases, not warehouses, but every change that adds a column to a table adds one its spreadsheet doesn't list.

Take the platform's spreadsheet, an inventory of crm.learners exported as CSV from the catalog and the metadata, one row per column. Then the CRM team's migration from the pull request above adds work_email. Compare what the spreadsheet says with what the check against the catalog finds:

1"""A governance inventory kept as a spreadsheet, written before a schema change, against the check on the live schema."""2import csv3import io45from lib import ci, metadata6from lib.catalog import open_catalog78platform = open_catalog(["crm.learners"])9described = metadata.load()1011# The inventory: one row per column of crm.learners, exported on the day someone was asked for one.12inventory = io.StringIO()13writer = csv.writer(inventory)14writer.writerow(["table", "column", "classification"])15for field in platform.load_table("crm.learners").schema().fields:16    writer.writerow(["crm.learners", field.name, described["crm.learners"].columns[field.field_id].classification])1718# Later, the CRM team's migration adds a column for learners' work email.19ci.migrate(platform, "alter table crm.learners add column work_email varchar;")2021rows = list(csv.DictReader(io.StringIO(inventory.getvalue())))22print("columns the spreadsheet lists:", len(rows))23print("of them without a classification, by the spreadsheet:", sum(not row["classification"] for row in rows))24live = [field.name for field in platform.load_table("crm.learners").schema().fields]25print("columns crm.learners has:", len(live))26print("columns the spreadsheet doesn't list:", ", ".join(sorted(set(live) - {row["column"] for row in rows})))27for problem in metadata.problems(platform, described):28    print("the check against the catalog finds:", problem)

columns the spreadsheet lists: 15
of them without a classification, by the spreadsheet: 0
columns crm.learners has: 16
columns the spreadsheet doesn't list: work_email
the check against the catalog finds: crm.learners.work_email: not classified

By its own account, the spreadsheet is complete: it lists 15 columns, and 0 lack a classification. The table has 16, and the one it doesn't list is work_email. Nothing in the spreadsheet can show that, because nothing compares it with the catalog. lib.metadata.problems, the catalog check, walks the catalog and finds crm.learners.work_email: not classified: the same finding that fails the CRM team's pull request. As a rule of thumb, a copy of the metadata kept by hand, whether a spreadsheet, a wiki page or a data dictionary, goes stale on its first schema change, and nothing says so.

Working rule: Never keep governance metadata in a document that nothing checks. Keep it in a reviewed file that a test compares with the live catalog, on every pull request and on a schedule. If people need a spreadsheet, generate it from the file, and say on it when it was generated.

A policy one tool enforces and another bypasses

A policy protects only the reads that pass through whatever enforces it. Trino applies row filters only when it's told where to fetch them: its opa.policy.row-filters-uri setting is "The optional URI for fetching row filters - if not set no row filtering is applied" (Trino 484, OPA access control). Iceberg's REST catalog specification, at 1.12.0, lets a catalog return read restrictions with a table, and says that "A reader that supports read restrictions must fail any read against the loaded table that cannot apply a returned restriction in full" (REST catalog spec, 1.12.0). The spec gives a reader no way to say that it supports them, as it does for delegated access to storage, with the X-Iceberg-Access-Delegation header: an "Optional signal to the server that the client supports delegated access". So a catalog can't tell which readers will apply its restrictions. PyIceberg 0.12.0's REST client reads none (source). We recommend that a catalog that returns read restrictions give storage access only to engines that apply them.

On the platform, the enforcement layer keeps agent-a05 to EMEA. Read the same table around it twice: as a notebook would, with DuckDB on the same catalog through lib.warehouse.connect, which the layer itself uses; and as any reader holding the catalog's access would, with PyIceberg's own scan:

1"""The same table, read three ways: through the enforcement layer, and around it with DuckDB and with PyIceberg."""2from collections import Counter34from lib import warehouse5from lib.catalog import open_catalog6from lib.enforce import Denied, Enforcer, answer78platform = open_catalog(["crm.learners"])9query = "select support_region, count(*) from crm.learners group by support_region order by support_region"101112def show(rows):13    return ", ".join(f"{region} {learners:,}" for region, learners in rows)141516layer = Enforcer(platform)17print("through the enforcement layer, as agent-a05:", show(answer(layer, query, "agent-a05")))1819# A notebook with DuckDB on the same catalog: the same SQL, and no policy in the way.20print("DuckDB on the same catalog:", show(warehouse.connect(platform).sql(query).fetchall()))2122# Any reader that holds the catalog's access, here PyIceberg's own scan: no SQL to rewrite at all.23scan = platform.load_table("crm.learners").scan(selected_fields=("support_region", "email")).to_arrow()24print("PyIceberg's scan of the table:", show(sorted(Counter(scan.column("support_region").to_pylist()).items())))25print("emails PyIceberg's scan returns:", f"{scan.num_rows - scan.column('email').null_count:,}")2627# lib.warehouse builds each table's view on a scan it registers under a name of its own. Through the layer, it's denied.28try:29    answer(layer, "select count(*) from scan_crm_learners", "agent-a05")30except Denied as reason:31    print("the layer, asked for the raw scan:", reason)

through the enforcement layer, as agent-a05: emea 1,581
DuckDB on the same catalog: americas 2,276, apac 1,143, emea 1,581
PyIceberg's scan of the table: americas 2,276, apac 1,143, emea 1,581
emails PyIceberg's scan returns: 5,000
the layer, asked for the raw scan: not a table in the catalog: scan_crm_learners

Through the layer, the agent's query returns emea 1,581. The same SQL on DuckDB with no layer in front returns americas 2,276, apac 1,143, emea 1,581, and PyIceberg's scan, which has no SQL to rewrite, returns americas 2,276, apac 1,143, emea 1,581, with 5,000 emails. The policy didn't fail: it was never asked. The same happens in production whenever a second engine, a Spark job or a script holding storage credentials reads the files that the governed engine reads.

lib.warehouse builds each table's view on a scan that it registers under a name of its own, scan_crm_learners for this table, so the layer's connection holds the raw scan too. The last line shows what the layer answers when a query names it: not a table in the catalog: scan_crm_learners. That's the design chapter 6 follows: enforce in one place, and deny everything else that reads data, whatever isn't a catalog table. In production, that place is the engine every tool queries through, as chapter 6 argues, and since storage permissions are part of enforcement, only that engine and the jobs that write the tables should be able to read the files.

Working rule: Choose one place that enforces the policy, the engine every tool queries through, and give storage credentials only to it and to the jobs that write the tables, never to people or notebooks. Deny every query in that place that reads data other than through a catalog table. Before you count a table as protected, list everything that can read its files, and treat any reader that doesn't ask the policy as a bypass, unless it's a job whose outputs are cataloged and classified like the tables it reads.

What this chapter decided

• Governance is four kinds of code in one repository: a metadata file, the code that sends each job's lineage events, an access policy in Rego, and checks that run on every pull request. Only one of them is a policy engine.
• Every table has an owning team, never a person, and every column one of four classifications: public, internal, personal or sensitive. They live in governance/metadata.toml, keyed by Iceberg field ID, and the catalog is tested against the file by walking the catalog.
• Every pipeline run sends OpenLineage events, with column lineage parsed from its SQL, including the columns that join and filter rows.
• Access decisions are data: allow, row filters and column masks, from one policy that denies by default. An enforcement layer applies them to every query, and denies every read that isn't of a catalog table.
• Missing metadata fails closed: the access policy treats a column nobody has classified as sensitive, and a table nobody has described stays a finding in every report until someone describes it.
• The governance job runs on every pull request, fails on the findings the pull request introduces, and reports the findings already on main with their owners.
• The examples run in one process, offline, through the standards the production services share. PyIceberg's SQL catalog stands in for a catalog server, DuckDB for Spark, DuckDB behind the enforcement layer for Trino, regopy for OPA's server, a file of events for a lineage server, and two folders for the CI service.