Governance end to end
읽는 데 약 58분분 걸린다, 8절에 6개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.31판을 번역한 것이다. 원문 읽기
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
By the end of this chapter you can govern one table of a data platform with code, end to end. You'll catalog it with an owning team and a classification for every column, capture the lineage of a pipeline that copies it, decide who may read which of its rows and enforce that decision on a query, and see a pull request fail because it added a column nobody classified. Each piece is small and runs in seconds, and five later chapters each make one of them solid.
This book is for data, analytics and platform engineers who already build pipelines and warehouses and have been asked to make them governed. You know SQL, Python and Parquet or a lakehouse table, and you don't need governance theory: you need governance that runs, is tested, and is reviewed in pull requests like the rest of your code. By the end of the book, you can govern a data platform with code that runs in CI:
- a catalog where every table has an owner and every column a classification;
- personal data found automatically, with what the detector misses measured;
- lineage captured from pipelines;
- access policies as tested code, enforced as row filters and column masks;
- retention and erasure that remove data;
- an audit trail;
- checks that fail a pull request;
- the same rules applied to data that trains or grounds AI.
This book follows one case throughout: an online learning platform that sells professional courses to adult learners, which we call the platform. Its warehouse is a catalog of Iceberg tables, each under the namespace of the team that runs it: learners and instructors under crm, courses, enrollments, learning events and course reviews under learning, payments under billing, and support tickets under support. SQL pipelines build reporting tables from them. Support agents, instructors, marketing, finance, data scientists and the platform's course assistant, its AI, each need something different from those tables, and that's what makes the platform worth governing.
The platform's data, which every example in this book reads, is generated. A script makes it once from fixed seeds with Faker, a library for fake data, and the files are committed with a data card that says how they were made. No real person is in the data. Its email addresses are on domains reserved for examples, so they can't reach anyone, and its IP addresses are in ranges reserved for documentation, so they point at no one. Its phone numbers, card numbers and IBANs are random in a valid format, so one may match a number or an account in service: never call, message, charge or pay them. The data card also lists what is planted for the examples to find, such as full card numbers in a legacy column and personal data in free text.
Every example runs offline, in one Python 3.12.15 process, on pinned libraries: PyIceberg 0.12.0 for the Iceberg catalog and its tables, DuckDB 2.0.0 for queries, sqlglot 30.33.0 to parse SQL, openlineage-python 1.55.0 for lineage, and regopy 1.5.2 for policies written in Rego, the policy language of Open Policy Agent (OPA). Six services that a production platform runs don't run here, and each has a stand-in:
- Trino, the SQL engine tools query through: DuckDB behind the book's enforcement layer, which rewrites each query before running it.
- OPA's server, which Trino asks for access decisions (Trino 484, OPA access control): regopy, evaluating the same Rego inside the Python process.
- Spark, or whatever engine runs the pipelines at scale: DuckDB, running each pipeline's SQL file.
- A catalog server or a cloud catalog, serving table metadata through an API such as the Iceberg REST catalog's (REST catalog spec, 1.12.0): PyIceberg's SQL catalog, backed by a SQLite file that records where each table's metadata is, with the metadata beside the data on local disk.
- A lineage server: a file of OpenLineage events, which the book's code reads back into a graph.
- The CI service and git: two folders, one for the main branch and one for a pull request's branch.
What you learn here is the open standards those services work with: Iceberg's table metadata, OpenLineage and Rego. What you build maps onto them.
The platform's governance starts with a spreadsheet. Someone listed the columns of crm.learners, which holds every learner's name, email, phone number and address, with a classification beside each, and that's all there is. No team is named as the table's owner. A nightly pipeline copies learners' names and emails into a table for marketing, and nothing records that it does. Support agents each serve one region, but a query on the warehouse returns every region's learners. And the CRM team is about to add a column. This chapter answers each of those with the code that the later chapters build, at the smallest scale that works: one table, one pipeline, one rule and one check. Only one of the four is a policy engine; the others are metadata, lineage events and a test, kept in the same repository and reviewed the same way.
Chapter 2 keeps the catalog in step with schema changes and measures a local model's drafted metadata. Chapter 4 uses lineage for impact analysis and to propose classifications. Chapter 5 writes the whole policy, with its tests, and chapter 6 enforces it on every query. Chapter 10 runs the checks of chapters 2 to 7 in CI, with owners' approvals and waivers.
Four chapters add what this one leaves out. Chapter 3 finds the personal data the metadata misses. Chapter 7 covers retention, consent and erasure. Chapter 8 keeps an audit trail, and chapter 9 applies the same rules to data that trains or grounds AI.
The shared modules, files in examples/lib/ that hold the code many examples need, are yours to open and read, and each section names the ones it uses.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Catalog a table with its owner and classifications
The platform's first problem is its spreadsheet, a list of columns that nothing checks. A catalog lists a platform's tables and their schemas, and governance needs two facts the data doesn't carry: who answers for each table, and how each column must be protected. Kept in a spreadsheet or a catalog server's web page, those facts change without review, and no test can say whether they're complete. Kept in a file in the repository, beside the pipelines, every change is a diff someone approves, and a test can compare the file with the catalog.
The platform's file is governance/metadata.toml. It gives each table an owner, a domain, the part of the business the table belongs to, and a description, and each column a description and a classification. Catalog tools mean different things by an owner, as chapter 2 compares, and we use one meaning throughout: the team accountable for the table, never a person, so the answer survives someone leaving. A classification is one of four values, from least to most protected:
public: may be shown to anyone outside the platform;internal: for the platform's staff, and doesn't identify a person by itself;personal: identifies a person, or tells something about one;sensitive: data whose leak would directly harm someone, such as a card number.
Columns are keyed by their Iceberg field ID, which survives a rename, with the name beside it for people:
[tables."crm.learners"]
owner = "team-crm"
domain = "crm"
description = "Every learner who has signed up, with their contact details and consents. One row per learner."
[tables."crm.learners".columns]
3 = { name = "email", classification = "personal", description = "The email the learner signs in with." }
Three of the book's shared modules do the work. lib.catalog.open_catalog builds a fresh Iceberg catalog of the platform's tables in a temporary folder, which it removes when the example exits. lib.metadata loads the file, writes it into the catalog with apply, and checks the catalog against it with problems, which reports any table with no metadata or no owner, and any column with no classification.
Loading the file and cataloging crm.learners with it takes a few lines, and the check runs on what it builds:
1"""Apply governance/metadata.toml to crm.learners: write its owner, domain and descriptions into the catalog, then run the2check on the catalog. The point rests on the last line, the problems the check finds once the file is applied."""3from collections import Counter45from lib import metadata6from lib.catalog import open_catalog78platform = open_catalog(["crm.learners"]) # a fresh Iceberg catalog holding the learners table9described = metadata.load()10learners = described["crm.learners"]11print("owner:", learners.owner)12print("domain:", learners.domain)13counts = Counter(column.classification for column in learners.columns.values())14print("columns classified personal:", counts["personal"])15print("personal columns:", ", ".join(c.name for c in learners.columns.values() if c.classification == "personal"))16print("columns classified internal:", counts["internal"])1718# Write the descriptions into the table's schema, where readers see them, and the owner and domain into its properties.19metadata.apply(platform, described)20table = platform.load_table("crm.learners")21print("table properties:", ", ".join(f"{key}={value}" for key, value in sorted(table.properties.items())))22print("email's description, in the Iceberg schema:", table.schema().find_field("email").doc)2324# The check: every table in the catalog has an owner, and every column one of the four classifications.25print("problems the check finds:", len(metadata.problems(platform, described)))
python catalog-the-learners.pyowner: team-crm
domain: crm
columns classified personal: 7
personal columns: full_name, email, phone, city, street_address, date_of_birth, notes
columns classified internal: 8
table properties: domain=crm, owner=team-crm
email's description, in the Iceberg schema: The email the learner signs in with.
problems the check finds: 0
The file gives crm.learners to team-crm, in the domain crm, and classifies 7 of its columns as personal: full_name, email, phone, city, street_address, date_of_birth, notes. notes is among them although its name says nothing, because support agents type whatever they like into it. The remaining columns are internal: the learner's ID, two dates, their country, their two consent flags, their account's status and their support region.
Applying the file also puts the owner and the descriptions where every reader sees them. The table's properties read domain=crm, owner=team-crm, and email's description sits in the Iceberg schema, where any reader of the table finds it: "The email the learner signs in with." The classifications stay in the file, because that is where the access policy and the checks read them. After apply, the check finds 0 problems.
A spreadsheet can't give you that zero. The check walks the catalog, so a column the file doesn't describe would show up as a problem, as work_email does in the section on spreadsheets.
Working rule: Keep owners, domains, descriptions and classifications in one reviewed file in the repository, with a team as every table's owner and one of a closed list of classifications on every column. Write the descriptions into the catalog, so every reader of a table sees them, and test the catalog against the file by walking the catalog, never the file. If edits happen in a catalog's web page, pick one side as the source, and generate the other from it (chapter 2).
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Capture one pipeline's column lineage
The catalog says what a table holds. Lineage says where its data came from and where it went, and governance needs both: a table built from crm.learners may or may not copy its emails. OpenLineage, an open specification for lineage, describes a job, such as a pipeline, its runs, and the datasets each run reads and writes. A run reports itself in events: a START when it begins, then one that ends it, such as COMPLETE or FAIL. Events carry facets, OpenLineage's name for blocks of metadata attached to a run, a job or a dataset.
The column lineage facet, on the output of a COMPLETE event, lists for each output column the input columns it came from, each marked DIRECT or INDIRECT. DIRECT means the output is derived from the input. INDIRECT means the input shapes the output without being copied: the output column value is "impacted by the value of inputField column, but it's not derived from it" (column lineage facet). The columns of a join or a filter condition are INDIRECT inputs, since they decide which rows exist, and so are those of a grouping, sort, window or condition.
The client library computes no lineage of its own, as chapter 4 finds in its source: it sends what you give it. So the book's module lib.lineage does the work. It parses each pipeline's SQL with sqlglot and adds the INDIRECT inputs that sqlglot's own lineage leaves out. It also wraps lib.pipelines.run, which runs one of the platform's SQL pipelines on DuckDB and writes its output to the catalog: START before the run, COMPLETE with the lineage after it. The events go through OpenLineage's file transport to one file, with a fixed time and a run ID derived from that time and the job's name, so the example writes the same bytes every time.
The platform's marketing_audience pipeline is one create table ... as select ... statement that writes marketing.audience, the list of learners marketing may email. It selects each learner's ID, name, email and country from crm.learners, joins each learner to their favourite course category, ranked from their enrollments, and keeps the learners for whom l.marketing_consent and l.status = 'active' holds.
Run the pipeline with events on, read its lineage back from the events alone, and count the rows through lib.warehouse.connect, which gives DuckDB a view of each catalog table:
1"""Run the marketing_audience pipeline with OpenLineage events, then read its column lineage back from the events alone.2The point rests on the two INDIRECT lines: the filter and the joins decide which rows the audience table holds."""34from lib import lineage, metadata, warehouse5from lib.catalog import open_catalog6from lib.platform import scratch78platform = open_catalog(["crm.learners", "learning.courses", "learning.enrollments"])9events_file = scratch() / "events.jsonl" # the file that stands in for a lineage server: one event per line10lineage.run(platform, "marketing_audience", lineage.client(events_file))11events = lineage.read_events(events_file)1213print("events:", ", ".join(f"{e['eventType']} of {e['job']['name']}" for e in events))14print("the run read:", ", ".join(dataset["name"] for dataset in events[0]["inputs"])) # the START event lists the inputs15print("the run wrote:", events[-1]["outputs"][0]["name"]) # the last event, COMPLETE, names the output1617graph = lineage.graph(events) # built from the COMPLETE events' column lineage, not from the SQL18for column in ("full_name", "email"):19 inputs = [e for e in graph.edges if e.target == f"marketing.audience.{column}"]20 print(f"marketing.audience.{column} comes from:", ", ".join(f"{e.source} ({e.type} {e.subtype})" for e in inputs))21# The INDIRECT inputs of each kind: the inputs that decide which rows exist, not the ones copied into them.22for subtype in ("FILTER", "JOIN"):23 shaping = [e.source for e in graph.edges if e.target == "marketing.audience" and e.subtype == subtype]24 print(f"its rows are shaped by, INDIRECT {subtype}:", ", ".join(shaping))25duckdb = warehouse.connect(platform) # DuckDB on the catalog's tables, to count what the join dropped26# The learners the filter keeps, counted on crm.learners before any join drops one.27consenting = "select count(*) from crm.learners where marketing_consent and status = 'active'"28print("consenting, active learners in crm.learners:", f"{duckdb.sql(consenting).fetchall()[0][0]:,}") # one row, one column29print("rows in marketing.audience:", f"{duckdb.sql('select count(*) from marketing.audience').fetchall()[0][0]:,}") # one row, one column30# Every column a change to email can reach, at any depth of the graph.31reached = lineage.columns_reached(graph, lineage.downstream(graph, "crm.learners.email"))32print("a change to crm.learners.email reaches:", ", ".join(reached))3334# The catalog check, run on the catalog the pipeline wrote to.35for problem in metadata.problems(platform, metadata.load()):36 print("the catalog check finds:", problem)
python audience-pipeline-lineage.pyevents: START of marketing_audience, COMPLETE of marketing_audience
the run read: crm.learners, learning.courses, learning.enrollments
the run wrote: marketing.audience
marketing.audience.full_name comes from: crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email comes from: crm.learners.email (DIRECT IDENTITY)
its rows are shaped by, INDIRECT FILTER: crm.learners.marketing_consent, crm.learners.status
its rows are shaped by, INDIRECT JOIN: crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id
consenting, active learners in crm.learners: 1,861
rows in marketing.audience: 1,829
a change to crm.learners.email reaches: marketing.audience.email
the catalog check finds: marketing.audience: no metadata
Read from the events alone, the lineage shows what the table's name leaves out. marketing.audience.email comes from crm.learners.email (DIRECT IDENTITY), and full_name from crm.learners.full_name (DIRECT IDENTITY), both unchanged copies of personal columns.
The filter and the joins decide which rows the table holds. The filter's inputs are crm.learners.marketing_consent, crm.learners.status, so every row says its learner consented. The joins' inputs are crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id. A learner with no enrollment has no favourite category, so the join to it drops them. crm.learners holds 1,861 consenting, active learners, and marketing.audience holds 1,829.
A change to crm.learners.email reaches only marketing.audience.email in this pipeline's table, so renaming the column, or changing its format, touches that column alone. With all four of the platform's pipelines sending lineage, chapter 4 finds that it reaches no other.
The catalog check, from the section before and run on the same catalog, finds marketing.audience: no metadata. The pipeline created a table nobody has described, and lineage changes what that gap means: from a table the platform knows nothing about to a copy of two personal columns, chosen by consent. Until someone describes it, the platform's access policy treats every column it can't find a classification for as sensitive. So anyone who reads the copy through the platform's enforcement layer, which a later section introduces, gets no value from it, as chapter 6's who-sees-what test shows.
Working rule: Emit lineage from the code that runs each job, never after it (chapter 4): a START event, then a COMPLETE or FAIL event, with the output's column lineage in the COMPLETE event, parsed from the job's SQL, and the inputs that join and filter rows as well as those that supply values. Read lineage back from the events, never from the SQL, so the graph shows what ran. Where a job isn't SQL that a parser can read, such as a Python script, write its lineage in the same code, and review the two together.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Enforce one access decision on a query
With the platform's tables cataloged, the next question is who may read what. Policy as code starts from OPA's premise that policy can be decoupled from the software it governs, so the people responsible for it can write, analyze, version and distribute it separately (OPA philosophy). A policy answers a request with a decision. The platform's policy answers with a document rather than a yes or no: whether to allow the read, the row filters that keep only the rows the user may see, and the column masks that change how a column is shown. The filters and masks are data, never SQL. A decision does nothing on its own. Something between the user and the engine has to apply it to every table a query reads.
Support agents on the platform each serve one region, americas, emea or apac, and need learners' contact details to answer tickets. The policy, governance/policies/access.rego, denies every request by default. It allows each group, the role a user holds in governance/users.toml, the tables its work needs, and reads the agent's region from the same file. This rule keeps an agent to their region:
row_filters contains {"column": "support_region", "equals": user.region} if {
"support" in groups
table in {"crm.learners", "support.tickets"}
}
lib.policy.decide evaluates the policy with regopy. lib.enforce holds the enforcement layer: its Enforcer asks the policy about every table a query reads, replaces each with a subquery that applies the decision, and runs the rewrite on DuckDB. lib.enforce.answer is the one function a service calls, and it returns rows, never the engine's connection.
agent-a05, a support agent in EMEA, asks for three columns of crm.learners and counts learners by region through the layer. Next, the emails are checked against the generated files that lib.platform.load reads:
1"""One access decision from the platform's Rego policy, and the same agent's query enforced in the layer. The point rests on2the emea line: the query names no region, yet only EMEA learners come back."""3import json45from lib import policy6from lib.catalog import open_catalog7from lib.enforce import Enforcer, answer8from lib.platform import load910# agent-a05, a support agent in EMEA, asks to read three columns of crm.learners.11request = policy.request("agent-a05", "crm.learners", ["learner_id", "email", "support_region"]) # the shape Trino's OPA plugin sends12decision = policy.decide(request)13assert decision["allow"] is True # support may read crm.learners; a policy that denies it fails this run14print("allow:", decision["allow"])15print("row filters:", json.dumps(decision["row_filters"]))16print("column masks:", json.dumps(decision["column_masks"]))17print("reasons:", "; ".join(decision["reasons"]))1819# The enforcement layer asks the policy about every table a query reads, then rewrites the query to apply the answer.20layer = Enforcer(open_catalog(["crm.learners"]))21query = "select support_region, count(*) as learners from crm.learners group by support_region order by support_region"22for region, learners in answer(layer, query, "agent-a05"): # the layer adds the agent's region to the query before the engine runs it23 print(f"learners agent-a05 sees in {region}:", f"{learners:,}")2425# Compare the emails the agent gets with the platform's data, without printing any.26stored = load("crm.learners") # the generated data, not the catalog's copy27emails = dict(zip(stored.column("learner_id").to_pylist(), stored.column("email").to_pylist()))28seen = answer(layer, "select learner_id, email from crm.learners", "agent-a05") # the agent's own query, with no region in it29print("emails agent-a05 sees, as stored:", f"{sum(email == emails[learner] for learner, email in seen):,} of {len(seen):,}")30print("learners in the platform's data:", f"{stored.num_rows:,}")
python support-region-enforced.pyallow: True
row filters: [{"column": "support_region", "equals": "emea"}]
column masks: {}
reasons: support may read crm.learners
learners agent-a05 sees in emea: 1,581
emails agent-a05 sees, as stored: 1,581 of 1,581
learners in the platform's data: 5,000
The layer applied the region inside the query, so the agent's SQL didn't have to, and couldn't avoid it. The decision allows the read, with the row filter [{"column": "support_region", "equals": "emea"}] on the agent's own region and the column masks {}, since support is the group that needs contact details. Its reason is support may read crm.learners. The agent's query has no filter on region, yet it returns 1,581 learners, all in EMEA, of the 5,000 the platform holds. Their emails come back as stored, 1,581 of 1,581, because the decision masks nothing.
regopy is the Python binding of rego-cpp, Microsoft's C++ implementation of Rego, which supports Rego v1.18.1 as OPA defines it (rego-cpp README at v1.5.2). regopy differs from OPA in ways chapter 5 lists, so OPA stays the reference for anything in production. Chapter 6 shows why the layer replaces whole tables rather than masking columns where a query names them.
Working rule: Write access rules as one policy in the repository that denies by default and returns its decision as data: allow, row filters and column masks. Apply that decision in one layer that rewrites every query before the engine runs it, and give users that layer's answers, never the engine's connection. When an engine can ask the policy itself, as Trino can ask OPA, let the engine be that layer.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Fail the pull request that adds an unclassified column
The catalog check proves the catalog complete on the day it runs, and the platform changes every week. A check that waits for someone to remember it gets run rarely and read less. Run on every pull request, where its exit code can fail it, the same check becomes a gate, and its findings appear where engineers already work. The authors of Tricorder, Google's platform for code analysis, found time and again that when developers have to open a dashboard or run a separate command line tool, their use of analysis drops off (Sadowski et al., 2015). That is one company's experience with code analysis, not data, but governance checks meet the same people. A gate also fails a pull request only for what it changes, so a finding already on main is reported with its owner and fails no pull request that leaves it as it is.
The book's module for this is lib.ci, the platform's governance job. A schema change on the platform is a migration, an alter table statement in a file in the branch's migrations/ folder. Each side of a pull request is a checkout in its own folder, one of main and one of the branch. For each side, lib.ci.run builds a scratch platform, a fresh catalog made the way the platform runs. It holds the source tables, the side's migrations, its metadata written into the catalog, and its pipelines. It runs the checks it's given on the branch's platform, runs them on main's alone to learn what is already there, and compares the findings by ID.
This section runs one check, coverage. The coverage check is the catalog check, lib.metadata.problems, run on the branch's platform, and for each column nobody classified it adds what a rule on column names suggests, from the rules chapter 3 writes.
The CRM team's pull request adds a column for learners' work email in one migration and forgets the column's metadata, so the coverage check has something to find:
1"""The CRM team's pull request adds a column to crm.learners and no metadata for it; the coverage check runs on its branch and2on main. The point rests on the exit code line: the pull request's own finding fails the job, and the findings on main don't."""3from lib import ci45main = ci.checkout() # the main branch: the governance files and pipelines, and no migrations yet6branch = ci.checkout(main) # the pull request's branch, starting from main7# The pull request's only change: a migration that adds the column, with no metadata for it.8(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")9print("files the pull request changes:", ", ".join(ci.changed(main, branch)))1011# Build a scratch platform from each side, run the coverage check on both, and compare their findings by ID.12report = ci.run(main, branch, checks=["coverage"])13for line in ci.lines(report):14 if not line.startswith("change "): # schema changes are chapter 10's report15 print(line)16for line in ci.totals(report):17 if line.startswith(("findings the pull request introduces", "findings already on main", "exit code")):18 print(line)
python unclassified-column-fails.pyfiles the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
job: failed
findings the pull request introduces: 1
findings already on main: 2
exit code: 1
The pull request fails on 1 finding of its own, the column nobody classified. The file it changes is migrations/0001-add-work-email.sql, and the coverage check reports failed, findings 3, new 1, on main 2, waived 0, 12 tables checked, which counts the platform's eight source tables and the four its pipelines write. The new finding is crm.learners.work_email, reported as not classified; its name suggests personal. The name rule matches email, which suggests the classification, and its ID, coverage/crm.learners.work_email, stays the same from run to run, so it can be discussed and counted.
The findings already on main were there before the pull request: coverage/ai.review_corpus, and coverage/marketing.audience, the table the lineage section found undescribed. Nobody owns either table, so the job names the platform team, the team answerable for tables without an owner. The job exits with code 1, so the pull request fails until the column has a classification, and the findings on main fail no pull request that leaves them as they are. Chapter 10's job runs the checks of chapters 2 to 7 on the same pull request, 11 of them.
Working rule: Run every governance check as one job on every pull request, required by branch protection, and fail it by exit code on any finding the pull request introduces. Report the findings already on the main branch with their owners, without failing the pull request for them. Put every finding in the pull request itself, under an ID that stays the same from run to run, with what to do about it. Run the same checks on a schedule against the production catalog too, for what changes outside pull requests.
운영 환경에서 무엇이 잘못되는가
거버넌스는 오류 없이 두 가지 방식으로 실패하며, 앞에서 다룬 구성 요소가 각각에 대한 답이 된다. 하나는 아무도 갱신하지 않는 스프레드시트이고, 다른 하나는 다른 도구가 우회하는 정책이다. 전자는 메타데이터를 코드로 다루는 이유이고, 후자는 시행 지점을 한곳에 두는 이유이다.
아무도 갱신하지 않는 거버넌스 스프레드시트
이 장이 출발한 곳인 스프레드시트는 테이블과 열을 나열하고 각각에 분류를 적어 둔다. 그런데 스프레드시트는 작성된 날의 스키마를 설명할 뿐이다. 스키마는 가만히 있지 않는다. 오픈 소스 데이터베이스 애플리케이션 열 개를 대상으로 하고 리비전을 합쳐 160,000개 넘게 가진 한 연구는, 주요 발견 가운데 첫째로 데이터베이스 스키마가 애플리케이션의 수명 동안 자주 진화한다는 점을 든다(Qiu, Li and Su, 2013). 이 연구가 다루는 것은 애플리케이션 데이터베이스이지 웨어하우스가 아니지만, 열을 추가하는 변경은 모두 스프레드시트에 없는 열을 하나씩 만들어 낸다.
플랫폼의 스프레드시트를 예로 들어 보자. 이 스프레드시트는 crm.learners의 목록을 CSV로 내보낸 것이며, 열 하나가 한 행이다. 그런 다음 위의 풀 리퀘스트에 있던 CRM 팀의 마이그레이션이 work_email을 추가한다. 마이그레이션 후의 카탈로그와 비교해서 스프레드시트가 놓친 것을 찾아본다.
1"""A governance inventory kept as a spreadsheet, exported before a schema change, then compared with the catalog after it.2The point rests on the two work_email lines: the spreadsheet doesn't list the column, and the check finds it unclassified."""3import csv4import io56from lib import ci, metadata7from lib.catalog import open_catalog89platform = open_catalog(["crm.learners"])10described = metadata.load()1112# The inventory: one row per column of crm.learners, exported on the day someone was asked for one.13inventory = io.StringIO()14writer = csv.writer(inventory)15writer.writerow(["table", "column", "classification"])16for field in platform.load_table("crm.learners").schema().fields:17 # Metadata is keyed by Iceberg field ID, not by name, so the lookup survives a rename.18 writer.writerow(["crm.learners", field.name, described["crm.learners"].columns[field.field_id].classification])1920# Later, the CRM team's migration adds a column for learners' work email, applied as an Iceberg schema change.21ci.migrate(platform, "alter table crm.learners add column work_email varchar;")2223rows = list(csv.DictReader(io.StringIO(inventory.getvalue())))24print("columns the spreadsheet lists:", len(rows))25print("of them without a classification, by the spreadsheet:", sum(not row["classification"] for row in rows)) # a blank classification cell: a column the spreadsheet never classified26live = [field.name for field in platform.load_table("crm.learners").schema().fields]27print("columns crm.learners has:", len(live))28print("columns the spreadsheet doesn't list:", ", ".join(sorted(set(live) - {row["column"] for row in rows})))29for problem in metadata.problems(platform, described):30 print("the check against the catalog finds:", problem)
python spreadsheet-misses-new-column.pycolumns the spreadsheet lists: 15
of them without a classification, by the spreadsheet: 0
columns crm.learners has: 16
columns the spreadsheet doesn't list: work_email
the check against the catalog finds: crm.learners.work_email: not classified
스프레드시트는 완전해 보이지만 열 하나를 놓치고 있다. 스프레드시트 스스로의 기록으로는 열 15개가 나열되어 있고, 그중 0개가 분류되어 있지 않다. 테이블의 열은 16개이며, 스프레드시트에 없는 열은 work_email이다. 스프레드시트를 카탈로그와 비교하는 것이 아무것도 없으므로, 스프레드시트 안에는 이를 보여 줄 수 있는 것이 없다. 카탈로그 검사인 lib.metadata.problems은 카탈로그를 훑으며 crm.learners.work_email: not classified을 찾아내는데, 이는 CRM 팀의 풀 리퀘스트를 실패시키는 것과 같은 발견이다. 우리의 경험으로는 스프레드시트, 위키 페이지, 데이터 사전 등 손으로 관리하는 메타데이터 사본은 첫 스키마 변경과 함께 낡아 버리고, 그 사실을 알려 주는 것은 아무것도 없다.
작업 규칙: 거버넌스 메타데이터를 아무것도 검사하지 않는 문서에 보관하지 않는다. 검토를 거친 파일에 보관하고, 테스트가 이를 모든 풀 리퀘스트마다, 그리고 일정에 따라 실제 카탈로그와 비교하게 한다. 사람들에게 스프레드시트가 필요하다면 그 파일에서 생성하고, 생성한 시점을 스프레드시트에 적는다.
한 도구는 집행하고 다른 도구는 우회하는 정책
정책은 그것을 집행하는 주체를 거치는 읽기만 보호한다. Trino는 행 필터를 어디서 가져올지 안내받을 때만 행 필터를 적용하므로, opa.policy.row-filters-uri 설정이 없으면 행 필터링을 전혀 적용하지 않는다(Trino 484, OPA access control). Iceberg의 REST 카탈로그 명세 1.12.0은 카탈로그가 테이블과 함께 읽기 제한을 반환할 수 있게 하며, "읽기 제한을 지원하는 리더는 반환된 제한을 온전히 적용할 수 없는 경우, 로드한 테이블에 대한 모든 읽기를 실패시켜야 한다"고 명시한다(REST catalog spec, 1.12.0). 이 명세는 위임된 접근을 위한 신호로 X-Iceberg-Access-Delegation 헤더를 리더에게 제공하지만, 읽기 제한을 위한 신호는 제공하지 않는다. 그래서 카탈로그는 어떤 리더가 자신의 제한을 적용할지 알 수 없다. PyIceberg 0.12.0의 REST 클라이언트는 읽기 제한을 전혀 읽지 않는다(source). 읽기 제한을 반환하는 카탈로그는 그 제한을 적용하는 엔진에만 스토리지 접근을 허용할 것을 권장한다.
집행 계층은 agent-a05이(가) EMEA 데이터만 보게 한다. 그러나 카탈로그 접근 권한이 있으면 누구나 테이블의 파일을 읽을 수 있으므로, 이 계층을 우회하는 경로가 있다. 계층을 거치지 않고 테이블을 두 가지 방식으로 읽어 본다. 하나는 계층 자체가 사용하는 lib.warehouse.connect을(를) 통해 노트북처럼 DuckDB로 읽는 방식이고, 다른 하나는 카탈로그 접근 권한을 가진 어떤 리더든 할 수 있듯이 PyIceberg 자체의 스캔으로 읽는 방식이다.
1"""The same table, read three ways: through the enforcement layer, and around it with DuckDB and with PyIceberg. The point2rests on the two lines around the layer, which return every region where the layer returns only EMEA's."""3from collections import Counter45from lib import warehouse6from lib.catalog import open_catalog7from lib.enforce import Denied, Enforcer, answer89platform = open_catalog(["crm.learners"])10query = "select support_region, count(*) from crm.learners group by support_region order by support_region"111213def show(rows):14 return ", ".join(f"{region} {learners:,}" for region, learners in rows)151617layer = Enforcer(platform)18print("through the enforcement layer, as agent-a05:", show(answer(layer, query, "agent-a05")))1920# A notebook with DuckDB on the same catalog: the same SQL, and no policy in the way.21print("DuckDB on the same catalog:", show(warehouse.connect(platform).sql(query).fetchall()))2223# Any reader that holds the catalog's access, here PyIceberg's own scan: no SQL to rewrite at all.24scan = platform.load_table("crm.learners").scan(selected_fields=("support_region", "email")).to_arrow()25print("PyIceberg's scan of the table:", show(sorted(Counter(scan.column("support_region").to_pylist()).items())))26print("emails PyIceberg's scan returns:", f"{scan.num_rows - scan.column('email').null_count:,}") # rows whose email isn't null2728# lib.warehouse builds each table's view on a scan it registers under a name of its own. Through the layer, it's denied.29try:30 answer(layer, "select count(*) from scan_crm_learners", "agent-a05")31except Denied as reason:32 print("the layer, asked for the raw scan:", reason)
python read-around-the-layer.pythrough the enforcement layer, as agent-a05: emea 1,581
DuckDB on the same catalog: americas 2,276, apac 1,143, emea 1,581
PyIceberg's scan of the table: americas 2,276, apac 1,143, emea 1,581
emails PyIceberg's scan returns: 5,000
the layer, asked for the raw scan: not a table in the catalog: scan_crm_learners
계층 밖의 리더는 정책에 묻지 않았으므로 모든 지역의 데이터를 얻는다. 계층을 거치면 에이전트의 쿼리는 emea 1,581을(를) 반환한다. 앞에 계층이 없는 DuckDB에서 같은 SQL을 실행하면 americas 2,276, apac 1,143, emea 1,581을(를) 반환한다. 다시 쓸 SQL이 없는 PyIceberg의 스캔은 americas 2,276, apac 1,143, emea 1,581을(를) 반환하며, 이메일 5,000개를 포함한다. 프로덕션에서도 Spark 작업이나 스토리지 자격 증명을 가진 스크립트처럼 두 번째 엔진이 통제되는 엔진과 같은 파일을 읽을 때마다 같은 우회가 일어난다.
lib.warehouse은(는) 각 테이블의 뷰를 자체 이름으로 등록한 스캔 위에 만드는데, 이 테이블의 경우 그 이름이 scan_crm_learners이다. 따라서 계층의 연결도 원본 스캔을 가지고 있다. 출력의 마지막 줄은 쿼리가 그 스캔을 이름으로 지정할 때 계층이 내놓는 응답을 보여 준다. 그 응답은 not a table in the catalog: scan_crm_learners이다. 이것이 6장이 따르는 설계이며, 한 곳에서 집행하고 카탈로그 테이블이 아닌 것을 비롯해 데이터를 읽는 나머지는 모두 거부한다. 프로덕션에서 그 한 곳은 6장이 주장하듯 도구가 쿼리하는 엔진이다. 또한 스토리지 권한은 집행의 일부이므로, 그 엔진과 테이블을 쓰는 작업만 파일을 읽을 수 있어야 한다.
작업 규칙: 정책을 집행할 곳을 하나, 즉 모든 도구가 거쳐 쿼리하는 엔진으로 정하고, 스토리지 자격 증명은 그 엔진과 테이블을 쓰는 작업에만 주며 사람이나 노트북에는 주지 않는다. 그곳에서 카탈로그 테이블을 거치지 않고 데이터를 읽는 쿼리는 모두 거부한다. 테이블을 보호된 것으로 세기 전에, 그 파일을 읽을 수 있는 모든 것을 나열하고, 정책에 묻지 않는 리더는 모두 우회로 취급한다. 다만 읽는 테이블과 똑같이 출력이 카탈로그에 등록되고 분류되는 작업은 예외이다.
이 장에서 결정한 것
- 거버넌스는 하나의 저장소에 담긴 네 종류의 코드이다. 메타데이터 파일, 각 작업의 리니지 이벤트를 보내는 코드, Rego로 작성한 접근 정책, 모든 풀 리퀘스트에서 실행되는 검사가 그것이다. 이 가운데 정책 엔진은 하나뿐이다.
- 모든 테이블에는 사람이 아닌 소유 팀이 있고, 모든 컬럼에는 네 가지 분류 중 하나가 붙는다. 분류는
public,internal,personal,sensitive이다. 이 정보는 Iceberg 필드 ID를 키로 하여governance/metadata.toml에 두며, 카탈로그를 순회하면서 카탈로그가 이 파일과 일치하는지 테스트한다. - 모든 파이프라인 실행은 OpenLineage 이벤트를 보낸다. 이때 컬럼 리니지는 SQL을 파싱해서 얻으며, 행을 조인하고 필터링하는 컬럼도 포함한다.
- 접근 결정은 데이터이다. 허용, 행 필터, 컬럼 마스크는 기본적으로 거부하는 하나의 정책에서 나온다. 강제 계층이 이를 모든 쿼리에 적용하고, 카탈로그 테이블에 대한 것이 아닌 모든 읽기는 거부한다.
- 메타데이터가 없으면 닫힌 상태로 실패한다. 접근 정책은 아무도 분류하지 않은 컬럼을
sensitive으로 취급하며, 아무도 설명하지 않은 테이블은 누군가 설명할 때까지 모든 보고서에 발견 사항으로 남는다. - 거버넌스 작업은 모든 풀 리퀘스트에서 실행된다. 풀 리퀘스트가 새로 만들어 내는 발견 사항에는 실패를 내고, main에 이미 있는 발견 사항은 소유자와 함께 보고한다.
- 모든 것은 하나의 프로세스에서 오프라인으로 실행되며, 이 장의 처음에 나열한 대역(stand-in)을 사용한다. 이런 방식의 거버넌스는 데이터가 바뀌는 지점마다 확인하는 작은 규칙들의 집합이므로, 메타데이터의 빈틈은 보고서에서 뒤늦게 발견되는 뜻밖의 문제가 아니라 실패한 검사로 드러난다. 이어지는 장들에서는 플랫폼 자체의 테이블을 대상으로 각 규칙을 차례로 테스트한다.
이 판의 이 장
2026.11.31판예제 통과 2026년 11월 19이 장에서 6개 예제개 중 6개가 통과한다
영어판 2026.11.31판을 번역한 것이다.