Data Governance as Code
Written with AI. Verified by people. How we make books
Anyone can check this book or correct it. Report an error · How to fix one
Governing a data platform with code that runs in CI: a catalog where every table has an owner and every column a classification, personal data found automatically, lineage from pipelines, access policies as tested Rego enforced as row filters and column masks, retention, consent and erasure that remove data, an audit trail, and the same rules for data that trains or grounds AI.
Who it's for For data, analytics and platform engineers who already build pipelines and warehouses and have been asked to make them governed: comfortable with SQL, Python and Parquet, and looking for governance that is tested and reviewed like the rest of their code.
Edition history1 edition since October 2026Full changelog
2026.10.1
2026.10.1 · Oct 23, 2026First edition108 examples
Contents
-
1Governance end to end
- Catalog a table with its owner and classifications
- Capture one pipeline's column lineage
- Enforce one access decision on a query
- Fail the pull request that adds an unclassified column
- What goes wrong in production
- A governance spreadsheet nobody updates
- A policy one tool enforces and another bypasses
- What this chapter decided
-
2A catalog you can test
- Keep the metadata in a reviewed file
- Key column metadata by field ID
- Write descriptions into the table's schema
- A description edit is a schema change
- Test the catalog against the metadata
- Catch schema changes before the metadata goes stale
- Classify pseudonymous IDs for joining
- Draft metadata with a local model
- Examples in a prompt can give the answer away
- Measure where the model's drafts are wrong
- What goes wrong in production
- What this chapter decided
-
3Finding personal data
- Classify columns from the schema
- Run Presidio offline
- Sample a column's values the same way every run
- Classify columns from their values
- Measure a detector against labelled spans
- Tune the recognizers against the truth
- See where patterns fail
- Compare a PII model
- Read the model's misses
- Count what a wrong label costs
- Classify again when the schema changes
- What goes wrong in production
- What this chapter decided
-
4Lineage from pipelines
- Send a run event from every pipeline
- Trace each output column to its sources
- Add the inputs that decide which rows exist
- Fail on statements the parser can't read
- Find what a column change breaks
- Propose classifications from lineage
- What goes wrong in production
- Events that never arrive
- A job nobody instrumented
- Personal data in a table classified internal
- What this chapter decided
-
5Access policies as code
- Shape the request like Trino's
- Generate the data the policy reads
- Deny by default, then allow by role
- Narrow by attribute: region, consent and purpose
- Return the decision as one document
- Unit-test the policy in Rego
- Check who sees what from a decision table
- Know where regopy differs from OPA
- Write the same rules in Cedar
- What goes wrong in production
- What this chapter decided
-
6Enforcing policies on queries
- Find every table and column a query reads
- Allow only catalog tables, CTEs and subqueries
- Masking column references leaks
- Replace each table with a filtered, masked subquery
- Write each mask as SQL in one place
- Build row filters from the decision's data
- Gate every column on the row filters
- Test who sees what by running queries
- Hold back small groups in reports
- Hand the same policies to an engine that enforces them
- Query on someone's behalf
- What goes wrong in production
- What this chapter decided
-
7Retention, consent and erasure
- Keep retention rules in a reviewed file
- Delete expired rows with row deletes
- Read consent where the query runs
- Plan an erasure from lineage
- Erase the rows the plan lists
- Expire snapshots by the table's own settings
- Remove the files no snapshot references
- Keep personal values out of manifests
- Prove the erasure by reading every file
- What goes wrong in production
- An erased learner comes back
- Other failures
- What this chapter decided
-
8Audit and evidence
- Record every decision where it's made
- Keep values out of the record
- Replay an old decision under its own policy
- Chain the records so an edit shows
- Cross-check the log against DuckDB's query log
- Generate the access review from the log
- Produce evidence as data
- What goes wrong in production
- What this chapter decided
-
9Governing data for AI
- Admit only reviews whose authors consented
- Replace personal data in the text, then measure what's left
- Keep pseudonyms consistent with a keyed hash
- Find near-duplicates with MinHash
- Tune LSH for recall, then confirm every pair
- Carry provenance on every record
- Generate the dataset card from the run
- Quote what the EU AI Act says about training data
- What goes wrong in production
- A learner withdraws after the build
- Other failures
- What this chapter decided
-
10Governance in CI
- Run every check as one job
- Gate the pull request, not the commit
- Report what a schema change reaches
- Ask the owner before lowering a classification
- Deprecate a column before removing it
- Route each domain's reviews by path
- Layer domain rules on the global rules
- Count waived findings per check
- What goes wrong in production
- A check that reads every row
- A check that flags every ID
- A broken file becomes a finding
- Other failures
- What this chapter decided