Rolling Edition

Data Governance as Code

Teaching kit

From Edition 2026.11.15, the book's current edition. It changes when the book does. Use and adapt it freely, with credit to the book.

On this page
  1. Syllabus
  2. Objectives and outlines
  3. Quiz bank

Syllabus

Reading times are for the chapter's text. Students also run its examples, in the Lab or on their own machine.

A chapter a week: 10 weeks

WeekChaptersReadingExamples
11. Governance end to end23 min6
22. A catalog you can test29 min11
33. Finding personal data33 min11
44. Lineage from pipelines25 min11
55. Access policies as code27 min13
66. Enforcing policies on queries32 min16
77. Retention, consent and erasure32 min11
88. Audit and evidence31 min9
99. Governing data for AI32 min10
1010. Governance in CI33 min10

In 5 weeks

WeekChaptersReadingExamples
11. Governance end to end
2. A catalog you can test		52 min17
23. Finding personal data
4. Lineage from pipelines		58 min22
35. Access policies as code
6. Enforcing policies on queries		59 min29
47. Retention, consent and erasure
8. Audit and evidence		63 min20
59. Governing data for AI
10. Governance in CI		65 min20

Objectives and outlines

Each chapter's objectives are its first sentence. Each outline follows the chapter's sections, with the terms it defines, the predictions it asks for before an example runs, and the rule each section ends on.

1. Governance end to end

About 23 minutes to read, 6 examples.

By the end, students can govern one table of a data platform with code, end to end.

Lecture outline: 8 sections
  1. Catalog a table with its owner and classifications
    • Run catalog-the-learners.py
    • Working rule Keep owners, domains, descriptions and classifications in one file in the repository, reviewed like the pipelines, with a team as every table's owner and one of a closed list of classifications on every column. Write the descriptions into the catalog, so every reader of a table sees them, and test the catalog against the file by walking the catalog, never the file. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it.
  2. Capture one pipeline's column lineage
    • Run audience-pipeline-lineage.py
    • Working rule Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run happens without them, and put the output's column lineage in the COMPLETE event, parsed from the job's SQL, with the inputs that join and filter rows as well as those that supply values. Read lineage back from the events, never from the SQL, so the graph shows what ran. Where a job isn't SQL that a parser can read, such as a Python script, write its lineage in the same code, and review the two together.
  3. Enforce one access decision on a query
    • Run support-region-enforced.py
    • Working rule Write access rules as one policy in the repository that denies by default and returns its decision as data: allow, row filters and column masks. Apply that decision in one layer that rewrites every query before the engine runs it, and give users that layer's answers, never the engine's connection. When an engine can ask the policy itself, as Trino can ask OPA, let the engine be that layer.
  4. Fail the pull request that adds an unclassified column
    • Run unclassified-column-fails.py
    • Working rule Run every governance check as one job on every pull request, required by branch protection, and fail it by exit code on any finding the pull request introduces. Report the findings already on the main branch with their owners, without failing the pull request for them. Put every finding in the pull request itself, under an ID that stays the same from run to run, with what to do about it. Run the same checks on a schedule against the production catalog too, for what changes outside pull requests.
  5. What goes wrong in production
  6. A governance spreadsheet nobody updates
    • Run spreadsheet-misses-new-column.py
    • Working rule Never keep governance metadata in a document that nothing checks. Keep it in a reviewed file that a test compares with the live catalog, on every pull request and on a schedule. If people need a spreadsheet, generate it from the file, and say on it when it was generated.
  7. A policy one tool enforces and another bypasses
    • Run read-around-the-layer.py
    • Working rule Choose one place that enforces the policy, the engine every tool queries through, and give storage credentials only to it and to the jobs that write the tables, never to people or notebooks. Deny every query in that place that reads data other than through a catalog table. Before you count a table as protected, list everything that can read its files, and treat any reader that doesn't ask the policy as a bypass, unless it's a job whose outputs are cataloged and classified like the tables it reads.
  8. What this chapter decided

2. A catalog you can test

About 29 minutes to read, 11 examples.

By the end, students can keep a catalog's metadata as code.

Lecture outline: 12 sections
  1. Keep the metadata in a reviewed file
    • Key terms metadata
    • Run metadata-model.py
    • Working rule Keep owners, domains, descriptions and classifications in a file in the repository, reviewed like the pipelines. Use a closed list of classifications, ordered by how much protection each needs, and one owning team per table. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it; two sources that both accept edits drift apart.
  2. Key column metadata by field ID
    • Run rename-orphans-name-keys.py
    • Working rule Key column metadata by field ID, with the name beside it for people. Treat a name that no longer matches as a reason to review the column's description, not as a new column. Where a system knows columns only by name, generate its name-keyed view from the ID-keyed file, never maintain it by hand.
  3. Write descriptions into the table's schema
  4. A description edit is a schema change
    • Run doc-edit-schema-id.py
    • Working rule Write column descriptions into Iceberg's doc, so every reader of the table gets them, and keep classifications in the reviewed file. Put no more than an owner and a domain in table properties, and say in your code why. To decide whether a schema change needs a review, compare field IDs, names and types, never schema IDs.
  5. Test the catalog against the metadata
    • Run metadata-gaps.py
    • Working rule Walk the catalog, not the metadata file, so a table nobody described shows up. Fail the test on any table without an owner and any column without one of the four classifications, and never default a missing classification to anything but the most protected value. A pipeline that creates a table ships with that table's metadata, in the same pull request.
  6. Catch schema changes before the metadata goes stale
    • Run schema-changes-checked.py
    • Working rule Run both checks on every schema change: problems for columns nobody has classified or renamed ones nobody has re-read, and dropped for entries that describe nothing. Remove a dropped column's entry in the same pull request that drops it. Dropping a personal column doesn't remove its values, which time travel reads until the snapshots that hold them expire and their files are deleted. Never drop and re-add a column to rename it: rename it in place, so it keeps its field ID and its data.
  7. Classify pseudonymous IDs for joining
    • Run pseudonymous-learner-id.py
    • Working rule Classify a pseudonymous ID by what its own table reveals, so the joins your analysts need keep working, and say in its description what it resolves through. Protect the table that resolves it, and include the ID in every erasure. Who can resolve the ID is an access decision: list the users who may read both the ID and the resolving table's personal columns unmasked, and grant that pair only to those whose job is linking rows to people.
  8. Draft metadata with a local model
  9. Examples in a prompt can give the answer away
    • Run suggestions-with-platform-examples.py
    • Working rule Draft metadata with a model only from inputs you're allowed to send it, preferably to a model on your own machines. Record each answer with the model, its revision, the prompt and the settings, so a reviewer can see what produced each draft, and parse and validate every answer before anything reads it. Take the prompt's examples from outside the tables you measure it on. Report agreement with the reviewed metadata per classification, never as one number.
  10. Measure where the model's drafts are wrong
    • Run suggestion-mistakes.py
    • Run suggestion-descriptions.py
    • Working rule Let a model raise a column's classification, never lower it: only a reviewer lowers one, in the pull request, with the reason in the column's description. Treat the model's descriptions as first drafts for someone who knows the column to rewrite. Before a model drafts metadata for tables nobody has reviewed, measure it against your reviewed metadata, per classification and per direction of error, and measure again whenever the model or the prompt changes.
  11. What goes wrong in production
  12. What this chapter decided

3. Finding personal data

About 33 minutes to read, 11 examples.

By the end, students can find personal data on a data platform automatically.

Lecture outline: 13 sections
  1. Classify columns from the schema
    • Run column-name-rules.py
    • Working rule Run name rules over every column first, because they're free, and let them raise a classification, never lower one. Keep the rules short, in code, and reviewed like any other rule. Treat any free-text column, whatever it's called, as one that needs its values sampled, and classify columns of other types by name alone.
  2. Run Presidio offline
    • Run presidio-offline-engine.py
    • Working rule Configure Presidio's engine explicitly: NoOpNlpEngine when you run patterns only, or a pinned model you install yourself when you need its names. Never call AnalyzerEngine() bare in a pipeline, and pin anything a recognizer would download, such as the Public Suffix List. Read each score as a constant set for its pattern, never as a probability.
  3. Sample a column's values the same way every run
    • Run seeded-column-sample.py
    • Working rule Sample non-null values, with a fixed seed, on one thread, and keep the seed with the result. Size the sample from the rarest personal data you must catch, with n = ln(0.01) / ln(1 − p) for 99%. The platform samples 200 values per text column, set in governance/pii.toml, which catches personal data held by 1 in 43 values or more 99% of the time. For anything rarer, scan the whole column, or classify it by rule.
  4. Classify columns from their values
    • Run classify-sampled-columns.py
    • Working rule Propose the strictest classification that the name or any sampled value supports, with internal as the floor, and show the evidence with the proposal. A proposal stricter than the reviewed metadata is a finding for the table's owner, who raises the classification or records why not; never let a detector lower one. OpenMetadata skips a column "already tagged as PII"; we recommend sampling every text column on every run instead, because the reviewed tag is the thing being checked.
  5. Measure a detector against labelled spans
    • Run measure-pattern-recognizers.py
    • Working rule Measure every detector against labelled spans before it masks anything: recall per entity type by exact and by overlap matching, both with a label of the right type, coverage whatever the label, precision per label, and the share of texts with nothing personal left. Weigh recall first. Before you trust a detector on your text, label a few hundred real records by hand and measure on those too.
  6. Tune the recognizers against the truth
    • Run tune-pattern-recognizers.py
    • Working rule Never run with Presidio's default threshold of 0, or with one threshold for every recognizer. Keep only the recognizers for identifiers your text can hold, set each one's threshold from its own scores, and read phone numbers in the regions your users live in, derived from your data. Put all of it in code, and measure again whenever it changes.
  7. See where patterns fail
    • Run pattern-hard-cases.py
    • Working rule Read a detector's misses by case, not by total, and keep a list of the cases your text holds. Use patterns for what has a checkable shape, such as emails, IBANs, card numbers, phone numbers and dates, and never for names or addresses. Keep the checksums and numbering plans, because they're what makes the patterns precise. Check the card pattern against the card numbers your payment provider issues, and the date patterns against the ways your users write dates.
  8. Compare a PII model
  9. Read the model's misses
    • Run privacy-filter-hard-cases.py
    • Working rule Mask free text with a PII model and the tuned patterns together, whatever either finds: the model for names and addresses, the patterns' checksums for card numbers and IBANs. Record the model's output with its revision, weights and operating point pinned, keep its own labels, and map them in code. Measure it on your own text, by hard case, before use and after every model change; if it misses too much, try an operating point that leans toward recall and record again. Treat masked text as reduced, never as anonymous.
  10. Count what a wrong label costs
    • Run privacy-filter-labels.py
    • Working rule Report coverage, whatever the label, beside recall with the right label: masking needs the first, classification the second. Read which labels a detector gives each type of span, and keep a recognizer with a checksum for every identifier whose label decides a classification, such as card numbers.
  11. Classify again when the schema changes
    • Run reclassify-new-columns.py
    • Working rule Classify every field ID the metadata doesn't have, in the pull request that adds it, and don't let it merge without a reviewed classification. Key the trigger on field IDs, never on schema IDs. Sample every text column again on a schedule too, because values change without a schema change.
  12. What goes wrong in production
  13. What this chapter decided

4. Lineage from pipelines

About 25 minutes to read, 11 examples.

By the end, students can capture lineage from your pipelines as OpenLineage run events, with column lineage parsed from their SQL, including the columns that only join, filter and group.

Lecture outline: 11 sections
  1. Send a run event from every pipeline
    • Key terms job, run, dataset, facet
    • Run pipeline-run-events.py
    • Working rule Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run can happen without them, and name datasets by catalog identifiers under a namespace you choose, never a path. In tests, use the file transport with a fixed event time and static run IDs, and compare what events say, never their bytes. In production, send through a transport to your lineage server, and derive each run ID from the scheduled time, the job and the attempt, so a resent event never creates a duplicate run.
  2. Trace each output column to its sources
    • Run sqlglot-misses-row-inputs.py
    • Working rule Use sqlglot's lineage for the columns each output's value is computed from, and always pass the catalog's schema, generated from the table metadata, never typed. Don't read its answer as everything a column depends on: it misses the columns that join, filter and group, and it doesn't say which of its inputs are values.
  3. Add the inputs that decide which rows exist
    • Run indirect-column-lineage.py
    • Working rule Record INDIRECT inputs with every job's column lineage: the join keys, filters and grouping columns of every SELECT, CTEs included, in the facet's dataset list, and type each value input with OpenLineage's subtypes. Expect more edges, many of them join keys. Keep them anyway: a filter on a consent column decides who may be emailed, and it reaches the table only as an INDIRECT input.
  4. Fail on statements the parser can't read
    • Run insert-and-merge-lineage.py
    • Run misspelt-column-lineage.py
    • Working rule Treat a statement your parser can't read correctly as an error, never as empty or partial lineage. Refuse every kind of statement it gets wrong, such as MERGE, before you ask it, and fail on empty lineage, a source the parser can't place and a column the schema doesn't have. Match an INSERT's columns by position against the table's columns in the catalog, never the SELECT's names, and fail when the counts differ. For a MERGE, write a SELECT of the columns it writes, and record the job's lineage from that, as an INSERT ... SELECT into the MERGE's table, until your parser handles MERGE itself.
  5. Find what a column change breaks
    • Ask first Before you run it, predict how many columns the consent change can reach:
    • Run column-impact.py
    • Working rule Run impact analysis on the graph built from events, INDIRECT edges included, and report two lists: columns whose values can change, and tables whose rows can change. Treat a change to a column used only in filters and joins as a change to every column of the tables it filters, and send the report to the owners of each table it reaches.
  6. Propose classifications from lineage
    • Run propagate-classifications.py
    • Run personal-filter-propagates.py
    • Working rule Propose each derived column's classification as the stricter of its inputs', counting INDIRECT inputs, and send the proposals to the table's owner for review before they become metadata. Never let lineage lower a classification someone declared, and never treat masking as declassification. Count DIRECT inputs only if your pipelines often join or filter on personal columns, such as matching on email, and every output would otherwise come out personal; then review INDIRECT personal inputs by hand.
  7. What goes wrong in production
  8. Events that never arrive
    • Run lineage-disabled-silently.py
    • Working rule After every scheduled run, compare the jobs that ran with the jobs whose COMPLETE events arrived, and alert on any difference. Don't rely on the emitter to report its own silence.
  9. A job nobody instrumented
    • Run uninstrumented-job.py
    • Working rule Check the catalog against the lineage on a schedule, and treat a table no job's lineage accounts for as an incident: find the job, and make it send events.
  10. Personal data in a table classified internal
    • Run personal-copy-tagged-internal.py
    • Working rule Run lib.lineage.understated, or your own check like it, on every lineage update, and fail when a declared classification is lower than lineage proposes. Classify by what a table holds, never by who reads it, and fix such a finding by raising the classification, not by dropping the edge.
  11. What this chapter decided

5. Access policies as code

About 27 minutes to read, 13 examples.

By the end, students can write a data platform's access rules as one policy that denies by default and returns row filters and column masks as data.

Lecture outline: 11 sections
  1. Shape the request like Trino's
    • Run trino-shaped-request.py
    • Working rule Shape the policy's input like the requests of the engine that will enforce it, field for field, and add only what that engine can't send, at the top level where a reviewer sees it. If your engine isn't Trino, start from its request format instead, and keep the shape in one function, as lib.policy.request does, so tests and callers can't drift apart.
  2. Generate the data the policy reads
    • Run generated-policy-data.py
    • Working rule Generate the policy's data from the file or table that owns each fact, every time you load the policy, and never copy a fact into the policy or into a second file. When a fact has no owner yet, such as a support agent's region, give it one file, reviewed like code, as the platform does with users.toml.
  3. Deny by default, then allow by role
    • Run allow-by-role.py
    • Working rule Start every policy with default allow := false, and allow only through rules that name the role and the resource. Keep the map from roles to tables in one place, where a reviewer can read all of it in one sitting, and give every decision a reason. When a request is denied that should have been allowed, add a rule that allows it; never weaken the default.
  4. Narrow by attribute: region, consent and purpose
    • Key terms region, Consent, purpose
    • Run attributes-as-row-filters.py
    • Working rule Model each condition as an attribute of the principal, the data or the request, read by one rule per role, rather than as another role. When an attribute comes from the requester, as a purpose does, let it unlock access only under a narrower condition, so that leaving it out, or getting it wrong, falls back to what the role allows anyway.
  5. Return the decision as one document
    • Run one-decision-document.py
    • Working rule Return one decision document per request, holding the allow, the row filters, the column masks and the reasons, and read it as a whole. Keep filters and masks as names and values that your enforcement code translates, never as SQL that the policy writes. If an engine needs SQL from the policy, as Trino does for its row filters and column masks (Trino 484, OPA access control), generate it in one tested function between the two.
  6. Unit-test the policy in Rego
    • Run policy-unit-tests.py
    • Working rule Write a test for every rule that grants or narrows access, and one for every way a request can be denied, and compare whole filters and masks, not single keys. Run the tests on every change to the policy and to the data the policy reads.
  7. Check who sees what from a decision table
    • Run who-sees-what.py
    • Working rule Keep a who-sees-what table in the repository with the policy, run it with the unit tests, and show its printed form to the data's owners, in terms they can check. When an owner disagrees with a row, change the row first, and let the failing row drive the change to the policy.
  8. Know where regopy differs from OPA
    • Run regopy-silent-differences.py
    • Run regopy-failed-outputs.py
    • Run regopy-sprintf-quotes.py
    • Working rule Treat regopy as a fast local evaluator and OPA as the reference. Write Rego v1 only, bind every decision to a variable, mock input one path at a time with literal values, and check that an output succeeded before reading it. Before a policy guards production data, check and test it with OPA itself, with opa check --strict and opa test.
  9. Write the same rules in Cedar
    • Run cedar-same-rules.py
    • Working rule Choose Cedar when every question is "may this principal do this to this resource?", as in an application's permission checks, and you want a schema to check every policy before it runs. Choose Rego when the answer has to carry data, as a warehouse's row filters and column masks do, or when the engine already asks OPA, as Trino does. For the platform's warehouse, we recommend Rego.
  10. What goes wrong in production
  11. What this chapter decided

6. Enforcing policies on queries

About 32 minutes to read, 16 examples.

By the end, students can enforce access decisions on every query the platform runs.

Lecture outline: 13 sections
  1. Find every table and column a query reads
    • Run tables-a-query-reads.py
    • Run unknown-tables-denied.py
    • Working rule Ask the policy about every table and column a query names, with * expanded against the catalog's own schema, and deny anything that schema doesn't hold: unknown tables, files, table functions and statements other than a single query. A denial costs a user one question to the platform team; a pass-through costs whatever the unchecked table holds.
  2. Allow only catalog tables, CTEs and subqueries
    • Run catalog-sources-only.py
    • Working rule Check every source a query reads against an allowlist of catalog tables, CTEs in scope and subqueries, and deny the rest, along with every function the engine marks as having side effects, read from its own list. Keep the cases as a regression test for every change to the layer and every engine upgrade, and add one whenever a new kind of source or function turns up.
  3. Masking column references leaks
    • Ask first Before you run it, predict how many of the five the column rewrite stops:
    • Run select-list-masks-leak.py
    • Working rule Don't enforce masks by rewriting the column references in a query. However many reference shapes you cover, the engine has more ways to read a column than the rewrite has to recognize one. Mask where the data comes from instead, so every read sees the masked value, which is what the next section does.
  4. Replace each table with a filtered, masked subquery
    • Run table-rewrite.py
    • Run table-rewrite-holds.py
    • Working rule Replace every protected table with a subquery generated from the decision, under the query's own alias, and leave the user's query otherwise untouched. Give columns the query doesn't read a typed NULL. If your users need whole-row reads to show values, ask the policy about every column of each table instead: the masks then cover all of them, at the cost of requests, and audit records, that name columns the query never used.
  5. Write each mask as SQL in one place
    • Key terms redact, pseudonymize
    • Run masks-as-sql.py
    • Run pseudonym-key.py
    • Run redact-contacts-recall.py
    • Working rule Keep every mask's SQL in one table in the enforcement layer, and let the policy name masks, never write them. Key every pseudonym with a secret only the layer holds, passed as a parameter. Measure each text mask against labelled data before a policy relies on it, and name masks for what they remove, so nobody reads redact_contacts as "anonymized".
  6. Build row filters from the decision's data
    • Run row-filters-from-data.py
    • Working rule Build every row filter from the decision's values with a SQL builder, and deny any filter shape or column the layer doesn't recognize. Treat an unknown consent as no consent. When a filter needs a table the user may not read, read it with the layer's own access inside the subquery, and never by widening the user's access.
  7. Gate every column on the row filters
    • Run row-filter-order.py
    • Working rule Never rely on the engine's evaluation order to keep a user's conditions off rows a filter drops. Make every value a user's query can read depend on the row filters, and test each filter shape on every engine upgrade. Where an engine has a barrier of its own, such as PostgreSQL's security_barrier views, use it as well.
  8. Test who sees what by running queries
    • Run who-sees-what-in-queries.py
    • Run group-change-caught.py
    • Working rule Run the who-sees-what cases as queries through the enforcement layer on every change to the policies, the users file, the metadata or the layer itself, and fail the build when any case disagrees. Fix a failing case by changing the policy or the users file, and change a case only when the people who own the data agree that what the user should get has changed.
  9. Hold back small groups in reports
    • Run small-groups-difference.py
    • Working rule Hold back every reported group below the minimum in governance/enforcement.toml, and treat the minimum as protection against looking one person up, never against differencing. Don't break a report down by an attribute its reader may not read, sent to a reader who can see the rows behind it: the instructor's report by country is a lookup table for countries. Drop the breakdown, or report it only after it has been made noisy.
  10. Hand the same policies to an engine that enforces them
    • Run trino-requests.py
    • Working rule Put enforcement in the engine every tool queries through, and keep one policy for it and for any rewrite like this chapter's. With Trino, answer every operation the plugin asks about, set both the row-filter and the column-mask URIs, since without them it applies no filters or masks, and return row filters only from a decision that allows the read. Build your tests from the request shapes in the plugin's source rather than from its documentation's example policies.
  11. Query on someone's behalf
    • Run assistant-on-behalf.py
    • Working rule Run an assistant's queries with the identity of the person it answers, taken from their authenticated session, and let the enforcement layer apply that person's decision. Use the assistant's own identity only for work it does for no one in particular, such as a nightly summary of consented reviews for the course pages, and record both identities in the audit log. Whoever builds the assistant's training set queries as themselves, with the purpose ai-training, as chapter 9 does.
  12. What goes wrong in production
  13. What this chapter decided

About 32 minutes to read, 11 examples.

By the end, students can keep retention rules as reviewed config, make a withdrawal of consent change the next query, and erase one person from every table their data reached.

Lecture outline: 13 sections
  1. Keep retention rules in a reviewed file
    • Run retention-rules-due.py
    • Working rule Keep each table's retention period, the column it counts from and the state a row must be in, in a reviewed file, and apply it against a day the scheduler passes in, never the clock. Count from the date the rule means. When the table doesn't record that date, as the platform's tickets don't record when they closed, ask the table's owner to add the column rather than let the rule count from another date for good, and say in the rule's file which way it errs until then.
  2. Delete expired rows with row deletes
    • Run apply-retention.py
    • Working rule Apply retention as Iceberg row deletes, one rule at a time, and record the rule and the day in each commit's snapshot summary. Make the job idempotent for a given day. Partition large tables by the column a rule counts from, so its deletes drop whole files instead of rewriting them; in PyIceberg, install its pyiceberg-core extra to write them. Count the job done only once the old snapshots are expired and their files removed.
  3. Read consent where the query runs
    • Run consent-withdrawal.py
    • Working rule Keep consent where the query runs: in one table that the system the learner uses keeps current, read by the policy as a row filter, so a withdrawal changes the next query and deletes nothing. When the platform's lawyers decide that a withdrawal is also a ground for erasure, send it through the erasure path as well. Rebuild every table built with a consent filter whenever consent changes, or give its readers the same filter. Record each consent as a state and the time it changed, and derive the true-or-false attribute the policy reads from it, counting unknown as not given.
  4. Plan an erasure from lineage
    • Run erasure-plan.py
    • Working rule Plan every erasure from the catalog and the lineage graph, not from a list someone maintains: every table with the person's ID column, then every table lineage says their rows reach. Delete from a derived table by a column that lineage shows is an unchanged copy of an ID, and rebuild the derived tables the person's rows only shaped. Keep exceptions in config, written by your legal team with their reasons, and let the plan read and show them; never let the code decide one. Run chapter 4's check for tables no job's lineage accounts for first, because the plan can't reach a copy the lineage doesn't show.
  5. Erase the rows the plan lists
    • Run erase-learner.py
    • Working rule Carry out an erasure as row deletes in the plan's order, and record only a request ID in each commit, never a value of the person's. Treat the delete as the first of three steps: the snapshots before it still hold the person, and anyone allowed to time travel can read them until they're expired.
  6. Expire snapshots by the table's own settings
    • Run expiry-ignores-table-settings.py
    • Run erasure-refuses-to-start.py
    • Working rule Expire snapshots through one function of your own that reads gc.enabled and the history.expire settings before it expires anything, and refuses when garbage collection is off, because PyIceberg's expiry reads neither; check again at every PyIceberg release. Before an erasure deletes anything, check every table in its plan the same way, and refuse the whole erasure when one would keep the person. For an erasure, expire every snapshot older than the delete. For retention, expire by age, as history.expire.max-snapshot-age-ms says, measured on the clock, since snapshots carry real commit times.
  7. Remove the files no snapshot references
    • Run sweep-unreferenced-files.py
    • Working rule Remove unreferenced files after every expiry, with a sweep of your own built on the inspect tables, since PyIceberg has none, and keep a minimum age of days in production so that a write in progress keeps its files. Use an age of zero only where a single process writes. Don't count a person erased because no data file holds them: the manifests may still hold their values.
  8. Keep personal values out of manifests
    • Run statistics-off-for-personal-columns.py
    • Working rule Generate per-column metrics properties from your classifications, none or counts for every personal, sensitive or unclassified column, and apply them when a table is created or, for an existing table, with one rewrite followed by expiry and a sweep. Keep statistics on the ID columns you filter and erase by. Generate the properties again after every schema change.
  9. Prove the erasure by reading every file
    • Run verify-erasure.py
    • Working rule End every erasure with a scan of every file the platform stores, for the person's IDs and identifying values, read with each format's own reader, and keep its result, counts and IDs only, as the evidence. Search for as much of each value as every bound keeps unchanged, which is one character less than the truncation length. Treat any file that still holds the person as a failed erasure, not a warning. Search the stores outside the warehouse that you control, and list in the record the ones you can't search.
  10. What goes wrong in production
  11. An erased learner comes back
    • Run erased-learner-returns.py
    • Working rule Erase in the source system first, or in the same request, and keep a register of keyed hashes of erased IDs that every load filters against and every run checks, with the key held as a secret apart from the data. Treat a row with an erased ID as an incident, as you would a leak.
  12. Other failures
  13. What this chapter decided

8. Audit and evidence

About 31 minutes to read, 9 examples.

By the end, students can keep an audit log of every access decision and query, with no values in it, chained so that an edit shows.

Lecture outline: 9 sections
  1. Record every decision where it's made
    • Run decision-record.py
    • Working rule Write the audit record in the enforcement layer, where the principal, the query and the decision meet, and write it before the query runs, so that a failed write stops the query. Record every identity involved, the person and any assistant acting for them, and every table's decision with its filters, masks and reasons. When an engine enforces the policy itself, as the SQL engine Trino does through OPA, turn on OPA's decision log instead, and make sure its input names every identity involved: Trino's opa.identity.extra-credentials-keys, for example, forwards the credentials it lists "to OPA as extraCredentials in the identity context" (Trino 484, OPA access control).
  2. Keep values out of the record
    • Run query-without-literals.py
    • Run query-shape-hard-cases.py
    • Working rule Record a query's shape, never its values: replace every literal, in every syntax the engine accepts, and every name the catalog doesn't hold, drop comments, take the same values out of denial reasons, and keep the executed SQL only as a keyed digest whose key the log's readers never get. Never use a plain hash of text that may hold personal data. When you need to know who read one person's data, keep that in a separate log that retention and erasure reach, not in the tamper-evident one.
  3. Replay an old decision under its own policy
    • Run explain-old-decision.py
    • Working rule Fingerprint every decision with a hash of everything it depended on besides the request: the policy's modules and its data document, together. Archive each revision for as long as you keep the log, and explain an old decision by replaying it under its own revision, never under the current one. If a policy reads the clock or calls another service, log those results too, as OPA's nd_builtin_cache does.
  4. Chain the records so an edit shows
    • Ask first Before you run it, predict which of the five the chain finds on its own, and which against the noon head.
    • Run chain-catches-edit.py
    • Working rule Chain every record to the one before with SHA-256 and a sequence number, and count on the chain to protect only the records up to the last head written down outside the layer's control. Write the head down often, to an append-only store or to another system under other credentials, as often as the number of unprotected records you can accept; if that number is zero, send each record off the machine as it's written. Check the chain against the last head on a schedule. If you need what AU-9(3) describes, have each head signed by a system whose key the layer never holds. Never chain a record that might one day have to be erased for one person; let whole periods expire instead.
  5. Cross-check the log against DuckDB's query log
    • Run query-log-cross-check.py
    • Working rule Turn on the engine's own query log wherever you enforce policies, fetch every result inside the layer, and check the log on a schedule against the records written since the last check, in both directions: every query the engine ran must match a record, and every allowed record must match a query. Then empty it, since it keeps the queries' values. Treat a mismatch as an incident to explain, not noise to filter. Count on the check to catch mistakes, not code that holds the connection and means to hide: against that, only a log written outside the process helps. Never use the engine's log as the audit log: it has no user and no decision.
  6. Generate the access review from the log
    • Run access-review.py
    • Working rule Generate access reviews from the policy and the audit log, never from a list of role memberships: what each person may read against what they read, their denials, and memberships the policy no longer knows. Run them over a period long enough for every job's rhythm, such as a quarter, and send each finding to the owner of the table or group it concerns. Remove a grant nobody used in the period unless its owner says why it's needed.
  7. Produce evidence as data
    • Run evidence-report.py
    • Working rule Generate evidence from records with code, on a schedule and on every policy change, and stamp each section with the policy revision, the digests of its inputs and what it takes to compute it again, saying which parts nobody can. Keep the reports where the log's writer can't change them, include failing findings rather than filtering them out, and let each report check the chain against the head the last one wrote down. Never send a screenshot where a report can be generated; chapter 10 runs checks like these on every pull request.
  8. What goes wrong in production
  9. What this chapter decided

9. Governing data for AI

About 32 minutes to read, 10 examples.

By the end, students can build the data an AI assistant learns from under the same rules as every other read of your platform.

Lecture outline: 12 sections
  1. Admit only reviews whose authors consented
    • Run corpus-consent-filter.py
    • Working rule Decide consent per record, when the corpus is built, from the answer your legal team says covers the use, and admit only records whose answer is yes: never filter out the refusals and keep the rest. Build the corpus through the enforcement layer with its purpose stated, and have the build check that the layer filters on that answer. Keep the legal team's answers, with their reasons, in a reviewed file the code reads, and carry licences per record when a source mixes them.
  2. Replace personal data in the text, then measure what's left
    • Run corpus-personal-data-left.py
    • Run privacy-filter-on-the-corpus.py
    • Working rule Detect personal data in corpus text after the enforcement layer's masks, add the identifiers your platform holds about the people the text is about, and measure what's left against labelled data, by value, after every change to the detector. Prefer a short list of the names your text is about to every name you hold, since a long list matches ordinary words. Have a job that may read the names build the list, and record which list each corpus used. Measure a named-entity model the same way before you rely on it for names no list holds. Keep the list out of the corpus, and call the result pseudonymized.
  3. Keep pseudonyms consistent with a keyed hash
    • Run keyed-pseudonyms.py
    • Working rule Make pseudonyms with HMAC under a key only the job holds, and give each corpus or audience whose pseudonyms must not join a key of its own. Never use Presidio's hash operator without a salt for pseudonyms, since at its pinned version each hash is random. Pseudonymize a person's ID rather than a word of their name when two people can share one, and use a plain placeholder when nothing downstream needs to tell people apart.
  4. Find near-duplicates with MinHash
    • Run corpus-duplicates.py
    • Run names-and-near-duplicates.py
    • Working rule Deduplicate training text in two steps, exact copies by a hash of the normalized text and then near-duplicates by MinHash, each confirmed by its exact similarity, and report how much each step removed. Pseudonymize first, so that no name reaches the comparison or the sketches and the comparison can tell which pairs differ only in a name. Keep those pairs apart when the text's meaning depends on whom it names, and compare on placeholders when it doesn't. Choose the threshold by reading the pairs close above and below it in your own text.
  5. Tune LSH for recall, then confirm every pair
    • Run lsh-misses-near-duplicates.py
    • Working rule Tune LSH below your similarity threshold, so that it misses few pairs, and confirm every candidate by its exact similarity. Measure LSH's misses against an exhaustive comparison on a sample of your corpus whenever its settings, its library or the text change. Pin the MinHash library, write its hashing scheme into your settings, and never compare counts or stored sketches made under different schemes.
  6. Carry provenance on every record
    • Run corpus-provenance.py
    • Working rule Give every record its source table, its row's ID, and the job and run that wrote it, and record the job's lineage from the query as it ran, filters and masks included. Check old builds against today's consent through that provenance. Filter every retrieval by the reader's own decision on each record's source row before a model sees it.
  7. Generate the dataset card from the run
    • Run corpus-dataset-card.py
    • Working rule Generate the card in the run that builds the data, with every count, hash, source and version taken from that run, and write the format's version out. Check the card with your own code for every property the specification requires, since the library's validator doesn't report them all, and don't publish or train on a corpus whose card fails.
  8. Quote what the EU AI Act says about training data
    • Working rule Quote the Act as amended, from the Official Journal, with a link and the date you read it, because the Omnibus moved both text and dates. Keep the law out of your pipeline's logic: generate the card's facts from the run, and let the people who answer legal questions decide which passages apply.
  9. What goes wrong in production
  10. A learner withdraws after the build
    • Run consent-withdrawn-after-build.py
    • Working rule Record which build, by run ID, every model and index was made from. Before every training run, check the build's records against today's consent and the file's hash against its card, and refuse a build that fails either. When a learner withdraws, find the builds that hold their records and the models made from them, and let your legal team's answer in config decide what happens to those.
  11. Other failures
  12. What this chapter decided

10. Governance in CI

About 33 minutes to read, 10 examples.

By the end, students can run the governance checks of chapters 2 to 7 as one job on each pull request, judging only what the pull request changes.

Lecture outline: 14 sections
  1. Run every check as one job
    • Run governance-ci-report.py
    • Working rule Run every check as one job on every pull request, each reporting all its findings under stable IDs, and a crashed check as a failing finding. Fail the job by exit code on any finding the pull request introduces that is neither waived nor approved. Report every finding already on main, counted and named with its owner, without failing a pull request that leaves it as it is. Keep the list of checks in a reviewed file.
  2. Gate the pull request, not the commit
    • Working rule Use hooks for fast checks of single files on the author's machine, and make the CI job on the pull request the gate, required by branch protection on the main branch. Put every finding in the pull request itself, with its ID and what to do about it, never only on a dashboard. Run its checks on a schedule against the production catalog too, for the tables that change outside any pull request.
  3. Report what a schema change reaches
    • Run schema-change-impact.py
    • Working rule Diff schemas by field ID, never by name or schema ID, and report every rename, type change and removal three ways: whether Iceberg readers are affected, which pipelines' SQL names the column, and what it reaches through lineage, INDIRECT edges included. Send the report to the owners of every table it reaches. Merge a rename that breaks a pipeline in the same pull request as the pipeline's fix, never before it.
  4. Ask the owner before lowering a classification
    • Run classification-downgrade.py
    • Working rule Treat a lowered classification, a changed owner and a move to another domain as changes that need the approval of the table's owner on the main branch, and fail the job until it's given; never rely on a breaking-change checker for them. Ask for the reason in the column's description in the same pull request. A raised classification needs no such approval, because it can only hide more.
  5. Deprecate a column before removing it
    • Run deprecate-then-remove.py
    • Working rule Remove a column in two pull requests: first mark it deprecated, with its replacement in the description, and merge that; then drop it from the table and its entry from the metadata. Fail any removal of a column the main branch hadn't deprecated. Agree with the column's readers how long the deprecation lasts, since the job checks only that it came first.
  6. Route each domain's reviews by path
    • Run domain-contracts.py
    • Working rule Keep each domain's metadata under a path of its own, as one ODCS contract per table with one domain and one owning team, generated from the metadata and checked by the job, with a CODEOWNERS line for that team. Give the platform team the CODEOWNERS file, the job's settings, its global rules, the waivers and the migrations. Require code-owner review, and count approvals per team in the job when one isn't enough.
  7. Layer domain rules on the global rules
    • Run global-and-domain-rules.py
    • Working rule Write the rules every domain must follow as one global package, kept by the platform team, and let each domain add a package of its own, owned by its team, that runs on that domain's contracts only. Return violations as data with a stable subject, never as a bare pass or fail. Let domain rules add to the global rules, never replace them, and treat a table's move to another domain as an ownership change, so no team escapes its domain's rules by changing one word.
  8. Count waived findings per check
    • Run waived-findings.py
    • Working rule Let a waiver name exactly one finding a pull request introduces, with an owning team, a reason and an end date at most a few months away, and list the checks no waiver may silence, such as a missing classification or owner. Report, for each check, how many findings nobody acted on, waived or left on main, and review any check where that's most of them: either the check is wrong or the platform is, and the count hides which.
  9. What goes wrong in production
  10. A check that reads every row
    • Run slow-check-every-row.py
    • Working rule Size pull-request checks by the change, not by the data: metadata, schemas, policies and the names of the columns a pull request adds. Run checks that read values on a schedule, with a fixed sample, as chapter 3 does, and send their findings to the owning team. Document where the job's checks stop, such as at a column added empty.
  11. A check that flags every ID
    • Run noisy-check-waivers.py
    • Working rule Before adding a check, run it on the main branch and read every finding; if most would be waived, change the rule rather than waiving its findings. Remove a check whose findings stay mostly waived, and treat a high share on a correct check as work to schedule, with an owner and a date.
  12. A broken file becomes a finding
    • Run broken-files-become-findings.py
    • Working rule Turn a check that crashes into one finding that fails the job, with the error's first line and the file it names, run the other checks anyway, and never let a waiver silence it.
  13. Other failures
  14. What this chapter decided

Quiz bank

Questions on every chapter, with answer keys, for verified teachers with the Educator plan. Each answer is tied to its section. Ask for the answer keys