端到端的数据治理
阅读约需 56分钟 分钟,8节中含6个示例。
读完本章,你可以用代码对数据平台中的一张表做端到端的治理。你会为它编目,写明所属团队,并为每一列标注分类;捕获一条复制该表的管道的血缘;决定谁可以读取其中的哪些行,并在查询上强制执行这一决定;还会看到一个拉取请求因为新增了一列而失败,而这一列没有人做过分类。每一部分都很小,几秒内即可运行,后面的五章则各自把其中一部分做得扎实可靠。
本书写给已经构建管道和数仓、并被要求使其受到治理的数据工程师、分析工程师和平台工程师。你了解 SQL、Python,以及 Parquet 或湖仓表,不需要治理理论:你需要的是能够运行、经过测试、并像其他代码一样在拉取请求中评审的治理。读完本书,你就能用在 CI 中运行的代码来治理一个数据平台:
- 一个目录,其中每张表都有所有者,每一列都有分类;
- 自动发现个人数据,并度量检测器漏掉了什么;
- 从管道中捕获的血缘;
- 以经过测试的代码形式编写的访问策略,并以行过滤和列掩码的方式强制执行;
- 真正删除数据的保留与擦除;
- 审计轨迹;
- 能使拉取请求失败的检查;
- 同样的规则应用于训练 AI 或为 AI 提供依据的数据。
本书自始至终围绕一个案例展开:一个面向成年学习者销售职业课程的在线学习平台,我们称之为该平台。它的数仓是一个由 Iceberg 表构成的目录,每张表都位于运行它的团队的命名空间之下:学习者和讲师在 crm 下,课程、报名、学习事件和课程评价在 learning 下,支付在 billing 下,支持工单在 support 下。SQL 管道用它们构建报表表。客服人员、讲师、市场、财务、数据科学家,以及平台的课程助手(它的 AI),各自需要从这些表中得到不同的东西,这正是该平台值得治理的原因。
本书每个示例读取的平台数据都是生成的。一个脚本使用 Faker(一个生成虚假数据的库),依据固定的种子一次性生成数据,文件连同一张说明生成方式的数据卡一并提交。数据中没有任何真实的人。其中的电子邮件地址使用为示例保留的域名,因此不会送达任何人;其中的 IP 地址位于为文档保留的地址段,因此不指向任何人。其中的电话号码、银行卡号和 IBAN 是格式有效的随机值,因此可能恰好与在用的号码或账户相同:切勿拨打、发送消息、扣款或向其付款。数据卡还列出了为示例预先埋入、等待被发现的内容,例如旧版列中的完整银行卡号和自由文本中的个人数据。
每个示例都离线运行,在单个 Python 3.12.15 进程中,使用固定版本的库:用 PyIceberg 0.12.0 处理 Iceberg 目录及其表,用 DuckDB 2.0.0 执行查询,用 sqlglot 30.33.0 解析 SQL,用 openlineage-python 1.55.0 处理血缘,用 regopy 1.5.2 处理以 Rego(Open Policy Agent(OPA)的策略语言)编写的策略。生产平台所运行的六项服务在这里并不运行,每一项都有一个替身:
- Trino,工具通过它查询的 SQL 引擎:由本书的强制执行层之后的 DuckDB 代替,该层在运行前改写每一条查询。
- OPA 服务器,Trino 向它请求访问决定(Trino 484,OPA 访问控制):由 regopy 代替,在 Python 进程内评估同样的 Rego。
- Spark,或任何大规模运行管道的引擎:由 DuckDB 代替,运行每条管道的 SQL 文件。
- 目录服务器或云目录,通过 API 提供表元数据,例如 Iceberg REST 目录的 API(REST 目录规范,1.12.0):由 PyIceberg 的 SQL 目录代替,它以一个 SQLite 文件为后端,记录每张表的元数据所在位置,元数据与数据一起放在本地磁盘上。
- 血缘服务器:由一个 OpenLineage 事件文件代替,本书的代码将它读回为一张图。
- CI 服务和 git:两个文件夹,一个对应主分支,一个对应拉取请求的分支。
你在这里学到的,是这些服务所依据的开放标准:Iceberg 的表元数据、OpenLineage 和 Rego。你所构建的内容可以映射到它们上面。
该平台的治理从一张电子表格开始。有人列出了 crm.learners 的各列,这张表保存着每位学习者的姓名、电子邮件、电话号码和地址,每列旁边写有分类,仅此而已。没有任何团队被指定为这张表的所有者。一条夜间管道把学习者的姓名和电子邮件复制到一张供市场使用的表中,而没有任何记录说明这件事。每位客服人员只服务一个地区,但对数仓的一次查询会返回所有地区的学习者。而 CRM 团队即将新增一列。本章用后面各章所构建的代码,以尽可能小的规模逐一回应这些问题:一张表、一条管道、一条规则和一项检查。四者中只有一个是策略引擎;其余的是元数据、血缘事件和一个测试,它们存放在同一个仓库中,并以同样的方式评审。
第 2 章使目录与模式变更保持同步,并度量本地模型起草的元数据。第 4 章把血缘用于影响分析,并用它提出分类建议。第 5 章编写完整的策略及其测试,第 6 章在每一次查询上强制执行它。第 10 章在 CI 中运行第 2 至 7 章的检查,并加入所有者的批准与豁免。
有四章补充本章略去的内容。第 3 章发现元数据所遗漏的个人数据。第 7 章讲述保留、同意与擦除。第 8 章维护审计轨迹,第 9 章把同样的规则应用于训练 AI 或为 AI 提供依据的数据。
共享模块是位于 examples/lib/ 中、存放许多示例所需代码的文件,可供你自行打开阅读,各节会指明它所用到的模块。
用所有者和分类为表编制目录
平台的第一个问题是它的电子表格,一份没有任何机制检查的列清单。数据目录列出平台的各张表及其模式,而治理还需要数据本身不携带的两项事实:每张表由谁负责,每一列必须如何保护。这些事实若保存在电子表格或目录服务器的网页中,修改时无需评审,也没有测试能判断它们是否完整。若保存在仓库中、与数据管道并列的文件里,每次修改都是一份有人批准的差异,测试也可以将该文件与目录相比对。
平台的这个文件是 governance/metadata.toml。它为每张表给出一个所有者、一个领域(表所属的业务部分)和一段描述,并为每一列给出一段描述和一个分类。数据目录工具对所有者的含义各不相同,如第 2 章所比较的那样;我们全书只采用一种含义:对该表负责的团队,而不是某个人,这样即使有人离职,答案依然有效。分类取以下四个值之一,按保护程度从低到高排列:
public:可以向平台之外的任何人展示;internal:供平台员工使用,且本身不能识别出某个人;personal:能识别出某个人,或透露关于某个人的信息;sensitive:一旦泄露会直接伤害某人的数据,例如银行卡号。
各列以其 Iceberg 字段 ID为键,字段 ID 在重命名后依然不变,旁边附上名称以便阅读:
[tables."crm.learners"]
owner = "team-crm"
domain = "crm"
description = "Every learner who has signed up, with their contact details and consents. One row per learner."
[tables."crm.learners".columns]
3 = { name = "email", classification = "personal", description = "The email the learner signs in with." }
本书的三个共享模块承担这项工作。lib.catalog.open_catalog 在临时文件夹中为平台的各张表构建一个全新的 Iceberg 目录,示例退出时将其删除。lib.metadata 加载该文件,用 apply 将其写入目录,并用 problems 对照该文件检查目录;它会报告任何没有元数据或没有所有者的表,以及任何没有分类的列。
加载该文件并据此为 crm.learners 编制目录只需几行代码,检查则针对构建出的结果运行:
1"""Apply governance/metadata.toml to crm.learners: write its owner, domain and descriptions into the catalog, then run the2check on the catalog. The point rests on the last line, the problems the check finds once the file is applied."""3from collections import Counter45from lib import metadata6from lib.catalog import open_catalog78platform = open_catalog(["crm.learners"]) # a fresh Iceberg catalog holding the learners table9described = metadata.load()10learners = described["crm.learners"]11print("owner:", learners.owner)12print("domain:", learners.domain)13counts = Counter(column.classification for column in learners.columns.values())14print("columns classified personal:", counts["personal"])15print("personal columns:", ", ".join(c.name for c in learners.columns.values() if c.classification == "personal"))16print("columns classified internal:", counts["internal"])1718# Write the descriptions into the table's schema, where readers see them, and the owner and domain into its properties.19metadata.apply(platform, described)20table = platform.load_table("crm.learners")21print("table properties:", ", ".join(f"{key}={value}" for key, value in sorted(table.properties.items())))22print("email's description, in the Iceberg schema:", table.schema().find_field("email").doc)2324# The check: every table in the catalog has an owner, and every column one of the four classifications.25print("problems the check finds:", len(metadata.problems(platform, described)))
python catalog-the-learners.pyowner: team-crm
domain: crm
columns classified personal: 7
personal columns: full_name, email, phone, city, street_address, date_of_birth, notes
columns classified internal: 8
table properties: domain=crm, owner=team-crm
email's description, in the Iceberg schema: The email the learner signs in with.
problems the check finds: 0
该文件把 crm.learners 的所有者定为 team-crm,领域为 crm,并将其中 7 列分类为 personal(个人数据):full_name, email, phone, city, street_address, date_of_birth, notes。notes 也在其中,尽管它的名称看不出任何端倪,因为客服人员会在里面随意输入内容。其余各列为 internal:学习者的 ID、两个日期、所在国家、两个同意标志、账户状态和支持区域。
应用该文件还会把所有者和描述放到每位读者都能看到的地方。表的属性读作 domain=crm, owner=team-crm,email 的描述则位于 Iceberg 模式中,任何读取该表的人都能找到:"The email the learner signs in with."分类仍保留在文件中,因为访问策略和各项检查正是从那里读取它们的。apply 之后,检查发现 0 个问题。
电子表格给不出这个零。检查会遍历目录,所以文件中没有描述的列会作为问题显现出来,正如 work_email 在关于电子表格的那一节中那样。
工作准则: 将所有者、领域、描述和分类保存在仓库中同一个经过评审的文件里,每张表的所有者是一个团队,每一列带有封闭列表中的一个分类。把描述写入目录,让每位读取该表的人都能看到,并通过遍历目录(而不是遍历文件)来对照该文件测试目录。如果修改发生在目录的网页上,就选定一方作为来源,另一方由它生成(第 2 章)。
捕获一条管道的列级血缘
目录说明一张表存放了什么。血缘则说明它的数据来自哪里、又流向何处,治理两者都需要:一张由 crm.learners 构建的表,可能复制了邮箱,也可能没有。OpenLineage 是一项开放的血缘规范,它描述作业(例如一条管道)、作业的运行,以及每次运行读取和写入的数据集。一次运行通过事件报告自身:开始时发出 START,结束时再发出一个结束事件,例如 COMPLETE 或 FAIL。事件携带facet,这是 OpenLineage 对附加在运行、作业或数据集上的元数据块的称呼。
列级血缘 facet位于 COMPLETE 事件的输出上,为每个输出列列出它所来自的输入列,并将每个输入列标记为 DIRECT 或 INDIRECT。DIRECT表示输出由该输入派生而来。INDIRECT表示输入影响了输出,但并未被复制:输出列的值"受 inputField 列的值影响,但并非由它派生"(列级血缘 facet)。连接条件和过滤条件中的列属于 INDIRECT 输入,因为它们决定了哪些行存在;分组、排序、窗口或条件中的列也是如此。
客户端库自身不计算任何血缘,第 4 章在其源码中发现了这一点:你给它什么,它就发送什么。因此由本书的模块 lib.lineage 来完成这项工作。它用 sqlglot 解析每条管道的 SQL,并补上 sqlglot 自带的血缘所遗漏的 INDIRECT 输入。它还封装了 lib.pipelines.run,后者在 DuckDB 上运行平台的一条 SQL 管道,并把输出写入目录:运行前发出 START,运行后发出带有血缘的 COMPLETE。这些事件经由 OpenLineage 的文件传输写入同一个文件,时间固定,运行 ID 由该时间和作业名称派生而来,所以该示例每次写出的字节都相同。
平台的 marketing_audience 管道是一条 create table ... as select ... 语句,它写入 marketing.audience,即营销部门可以发送邮件的学习者名单。它从 crm.learners 中选出每位学习者的 ID、姓名、邮箱和国家,将每位学习者与其最喜爱的课程类别连接起来(该类别按其选课记录排序得出),并保留满足 l.marketing_consent and l.status = 'active' 的学习者。
开启事件运行这条管道,仅从事件中读回它的血缘,并通过 lib.warehouse.connect 统计行数,该函数为 DuckDB 提供每张目录表的视图:
1"""Run the marketing_audience pipeline with OpenLineage events, then read its column lineage back from the events alone.2The point rests on the two INDIRECT lines: the filter and the joins decide which rows the audience table holds."""34from lib import lineage, metadata, warehouse5from lib.catalog import open_catalog6from lib.platform import scratch78platform = open_catalog(["crm.learners", "learning.courses", "learning.enrollments"])9events_file = scratch() / "events.jsonl" # the file that stands in for a lineage server: one event per line10lineage.run(platform, "marketing_audience", lineage.client(events_file))11events = lineage.read_events(events_file)1213print("events:", ", ".join(f"{e['eventType']} of {e['job']['name']}" for e in events))14print("the run read:", ", ".join(dataset["name"] for dataset in events[0]["inputs"])) # the START event lists the inputs15print("the run wrote:", events[-1]["outputs"][0]["name"]) # the last event, COMPLETE, names the output1617graph = lineage.graph(events) # built from the COMPLETE events' column lineage, not from the SQL18for column in ("full_name", "email"):19 inputs = [e for e in graph.edges if e.target == f"marketing.audience.{column}"]20 print(f"marketing.audience.{column} comes from:", ", ".join(f"{e.source} ({e.type} {e.subtype})" for e in inputs))21# The INDIRECT inputs of each kind: the inputs that decide which rows exist, not the ones copied into them.22for subtype in ("FILTER", "JOIN"):23 shaping = [e.source for e in graph.edges if e.target == "marketing.audience" and e.subtype == subtype]24 print(f"its rows are shaped by, INDIRECT {subtype}:", ", ".join(shaping))25duckdb = warehouse.connect(platform) # DuckDB on the catalog's tables, to count what the join dropped26# The learners the filter keeps, counted on crm.learners before any join drops one.27consenting = "select count(*) from crm.learners where marketing_consent and status = 'active'"28print("consenting, active learners in crm.learners:", f"{duckdb.sql(consenting).fetchall()[0][0]:,}") # one row, one column29print("rows in marketing.audience:", f"{duckdb.sql('select count(*) from marketing.audience').fetchall()[0][0]:,}") # one row, one column30# Every column a change to email can reach, at any depth of the graph.31reached = lineage.columns_reached(graph, lineage.downstream(graph, "crm.learners.email"))32print("a change to crm.learners.email reaches:", ", ".join(reached))3334# The catalog check, run on the catalog the pipeline wrote to.35for problem in metadata.problems(platform, metadata.load()):36 print("the catalog check finds:", problem)
python audience-pipeline-lineage.pyevents: START of marketing_audience, COMPLETE of marketing_audience
the run read: crm.learners, learning.courses, learning.enrollments
the run wrote: marketing.audience
marketing.audience.full_name comes from: crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email comes from: crm.learners.email (DIRECT IDENTITY)
its rows are shaped by, INDIRECT FILTER: crm.learners.marketing_consent, crm.learners.status
its rows are shaped by, INDIRECT JOIN: crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id
consenting, active learners in crm.learners: 1,861
rows in marketing.audience: 1,829
a change to crm.learners.email reaches: marketing.audience.email
the catalog check finds: marketing.audience: no metadata
仅从事件读取,血缘显示了表名所没有说出的内容。marketing.audience.email 来自 crm.learners.email (DIRECT IDENTITY),full_name 来自 crm.learners.full_name (DIRECT IDENTITY),二者都是个人信息列的原样副本。
过滤条件和连接共同决定这张表保留哪些行。过滤条件的输入是 crm.learners.marketing_consent, crm.learners.status,所以每一行都表明其学习者已同意。连接的输入是 crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id。没有选课记录的学习者没有最喜爱的类别,所以与该类别的连接会把他们排除。crm.learners 中有 1,861 位同意且处于活跃状态的学习者,而 marketing.audience 中有 1,829。
对 crm.learners.email 的更改在这条管道的表中只会影响 marketing.audience.email,因此重命名该列或更改其格式,只会涉及这一列。平台的全部四条管道都发送血缘后,第 4 章发现它不会影响其他任何列。
上一节中的目录检查在同一个目录上运行,发现了 marketing.audience: no metadata。这条管道创建了一张无人描述的表,而血缘改变了这一缺口的含义:它从一张平台一无所知的表,变成了按同意情况筛选出的、含有两个个人信息列的副本。在有人描述它之前,平台的访问策略会把每一个找不到分类的列都视为 sensitive。因此,任何通过平台的执行层(该层将在后面的一节介绍)读取该副本的人,都得不到其中的任何值,第 6 章的谁能看到什么测试展示了这一点。
**工作准则:**血缘要由运行每个作业的代码发出,而不要事后补写(第 4 章):先发出 START 事件,再发出 COMPLETE 或 FAIL 事件,输出的列级血缘放在 COMPLETE 事件中,由作业的 SQL 解析得出,并且既包括提供值的输入,也包括连接和过滤行的输入。血缘要从事件中读回,而不要从 SQL 中读取,这样血缘图显示的才是实际运行的内容。如果某个作业不是解析器能读懂的 SQL,例如一个 Python 脚本,就在同一段代码中写出它的血缘,并将二者一同审查。
对查询执行一次访问决策
平台的各张表已经编入目录,接下来的问题是谁可以读取什么。策略即代码出自 OPA 的一个前提:策略可以与它所管辖的软件解耦,这样负责策略的人就能够单独编写、分析、版本化和分发策略(OPA philosophy)。策略以一项决策来回应一次请求。平台的策略给出的不是简单的是或否,而是一份文档:是否允许读取、只保留用户可见行的行过滤器,以及改变某一列展示方式的列掩码。过滤器和掩码都是数据,绝不是 SQL。决策本身不起任何作用。必须由位于用户与引擎之间的某个环节,把它应用到查询读取的每一张表上。
平台上的客服坐席各自负责一个区域,即 americas、emea 或 apac,他们需要学员的联系方式来回复工单。策略文件 governance/policies/access.rego 默认拒绝所有请求。它为每个组(即用户在 governance/users.toml 中所担任的角色)开放其工作所需的表,并从同一个文件中读取坐席所属的区域。下面这条规则把坐席限定在自己的区域内:
row_filters contains {"column": "support_region", "equals": user.region} if {
"support" in groups
table in {"crm.learners", "support.tickets"}
}
lib.policy.decide 使用 regopy 对策略求值。lib.enforce 中是执行层:它的 Enforcer 就查询读取的每一张表向策略发出询问,把每张表替换为应用该决策的子查询,并在 DuckDB 上运行改写后的查询。lib.enforce.answer 是服务调用的唯一函数,它返回的是数据行,绝不是引擎的连接。
EMEA 区域的客服坐席 agent-a05 通过该层查询 crm.learners 的三个列,并按区域统计学员人数。接着,对照 lib.platform.load 读取的生成文件,核对这些电子邮件:
1"""One access decision from the platform's Rego policy, and the same agent's query enforced in the layer. The point rests on2the emea line: the query names no region, yet only EMEA learners come back."""3import json45from lib import policy6from lib.catalog import open_catalog7from lib.enforce import Enforcer, answer8from lib.platform import load910# agent-a05, a support agent in EMEA, asks to read three columns of crm.learners.11request = policy.request("agent-a05", "crm.learners", ["learner_id", "email", "support_region"]) # the shape Trino's OPA plugin sends12decision = policy.decide(request)13assert decision["allow"] is True # support may read crm.learners; a policy that denies it fails this run14print("allow:", decision["allow"])15print("row filters:", json.dumps(decision["row_filters"]))16print("column masks:", json.dumps(decision["column_masks"]))17print("reasons:", "; ".join(decision["reasons"]))1819# The enforcement layer asks the policy about every table a query reads, then rewrites the query to apply the answer.20layer = Enforcer(open_catalog(["crm.learners"]))21query = "select support_region, count(*) as learners from crm.learners group by support_region order by support_region"22for region, learners in answer(layer, query, "agent-a05"): # the layer adds the agent's region to the query before the engine runs it23 print(f"learners agent-a05 sees in {region}:", f"{learners:,}")2425# Compare the emails the agent gets with the platform's data, without printing any.26stored = load("crm.learners") # the generated data, not the catalog's copy27emails = dict(zip(stored.column("learner_id").to_pylist(), stored.column("email").to_pylist()))28seen = answer(layer, "select learner_id, email from crm.learners", "agent-a05") # the agent's own query, with no region in it29print("emails agent-a05 sees, as stored:", f"{sum(email == emails[learner] for learner, email in seen):,} of {len(seen):,}")30print("learners in the platform's data:", f"{stored.num_rows:,}")
python support-region-enforced.pyallow: True
row filters: [{"column": "support_region", "equals": "emea"}]
column masks: {}
reasons: support may read crm.learners
learners agent-a05 sees in emea: 1,581
emails agent-a05 sees, as stored: 1,581 of 1,581
learners in the platform's data: 5,000
该层在查询内部应用了区域限制,因此坐席的 SQL 不必自己写出这一限制,也无法绕开它。决策允许读取,行过滤器 [{"column": "support_region", "equals": "emea"}] 作用于坐席自己的区域,列掩码为 {},因为客服正是需要联系方式的组。决策给出的理由是 support may read crm.learners。坐席的查询没有对区域设置任何过滤,却只返回了 1,581 名学员,全部位于 EMEA,而平台数据中共有 5,000 名学员。他们的电子邮件按存储时的样子返回,即 1,581 of 1,581,因为这项决策没有掩码任何内容。
regopy 是 rego-cpp 的 Python 绑定,rego-cpp 是微软用 C++ 实现的 Rego,支持 OPA 所定义的 Rego v1.18.1(rego-cpp README at v1.5.2)。regopy 与 OPA 之间存在一些差异,第 5 章列出了这些差异,因此凡涉及生产环境的事项,仍以 OPA 为准。第 6 章说明了为什么该层要替换整张表,而不是在查询点名某个列的地方对该列做掩码。
工作准则: 把访问规则写成仓库中的一份策略,默认拒绝,并以数据形式返回决策:允许、行过滤器和列掩码。把这项决策应用在单独一个执行层中,该层在引擎运行之前改写每一条查询,并且只向用户提供该层的答复,绝不提供引擎的连接。当引擎能够自己向策略发出询问时,比如 Trino 可以询问 OPA,就让引擎本身充当这一层。
这部分尚未翻译,因此以英文显示。
Fail the pull request that adds an unclassified column
The catalog check proves the catalog complete on the day it runs, and the platform changes every week. A check that waits for someone to remember it gets run rarely and read less. Run on every pull request, where its exit code can fail it, the same check becomes a gate, and its findings appear where engineers already work. The authors of Tricorder, Google's platform for code analysis, found time and again that when developers have to open a dashboard or run a separate command line tool, their use of analysis drops off (Sadowski et al., 2015). That is one company's experience with code analysis, not data, but governance checks meet the same people. A gate also fails a pull request only for what it changes, so a finding already on main is reported with its owner and fails no pull request that leaves it as it is.
The book's module for this is lib.ci, the platform's governance job. A schema change on the platform is a migration, an alter table statement in a file in the branch's migrations/ folder. Each side of a pull request is a checkout in its own folder, one of main and one of the branch. For each side, lib.ci.run builds a scratch platform, a fresh catalog made the way the platform runs. It holds the source tables, the side's migrations, its metadata written into the catalog, and its pipelines. It runs the checks it's given on the branch's platform, runs them on main's alone to learn what is already there, and compares the findings by ID.
This section runs one check, coverage. The coverage check is the catalog check, lib.metadata.problems, run on the branch's platform, and for each column nobody classified it adds what a rule on column names suggests, from the rules chapter 3 writes.
The CRM team's pull request adds a column for learners' work email in one migration and forgets the column's metadata, so the coverage check has something to find:
1"""The CRM team's pull request adds a column to crm.learners and no metadata for it; the coverage check runs on its branch and2on main. The point rests on the exit code line: the pull request's own finding fails the job, and the findings on main don't."""3from lib import ci45main = ci.checkout() # the main branch: the governance files and pipelines, and no migrations yet6branch = ci.checkout(main) # the pull request's branch, starting from main7# The pull request's only change: a migration that adds the column, with no metadata for it.8(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")9print("files the pull request changes:", ", ".join(ci.changed(main, branch)))1011# Build a scratch platform from each side, run the coverage check on both, and compare their findings by ID.12report = ci.run(main, branch, checks=["coverage"])13for line in ci.lines(report):14 if not line.startswith("change "): # schema changes are chapter 10's report15 print(line)16for line in ci.totals(report):17 if line.startswith(("findings the pull request introduces", "findings already on main", "exit code")):18 print(line)
python unclassified-column-fails.pyfiles the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
job: failed
findings the pull request introduces: 1
findings already on main: 2
exit code: 1
The pull request fails on 1 finding of its own, the column nobody classified. The file it changes is migrations/0001-add-work-email.sql, and the coverage check reports failed, findings 3, new 1, on main 2, waived 0, 12 tables checked, which counts the platform's eight source tables and the four its pipelines write. The new finding is crm.learners.work_email, reported as not classified; its name suggests personal. The name rule matches email, which suggests the classification, and its ID, coverage/crm.learners.work_email, stays the same from run to run, so it can be discussed and counted.
The findings already on main were there before the pull request: coverage/ai.review_corpus, and coverage/marketing.audience, the table the lineage section found undescribed. Nobody owns either table, so the job names the platform team, the team answerable for tables without an owner. The job exits with code 1, so the pull request fails until the column has a classification, and the findings on main fail no pull request that leaves them as they are. Chapter 10's job runs the checks of chapters 2 to 7 on the same pull request, 11 of them.
Working rule: Run every governance check as one job on every pull request, required by branch protection, and fail it by exit code on any finding the pull request introduces. Report the findings already on the main branch with their owners, without failing the pull request for them. Put every finding in the pull request itself, under an ID that stays the same from run to run, with what to do about it. Run the same checks on a schedule against the production catalog too, for what changes outside pull requests.
生产环境中会出什么问题
治理会以两种方式在不报错的情况下失效,上文介绍的各个部分正好应对了这两种情形:一份无人更新的电子表格,以及一项被其他工具绕过的策略。前者说明了元数据为何要以代码形式管理,后者说明了为何要把执行机制放在同一个位置。
无人更新的治理电子表格
本章最初依据的那份电子表格列出了各个表和列,并在每一项旁标注了分类,而电子表格描述的只是编写当天的模式。模式不会一成不变。一项针对十个开源数据库应用的研究涵盖了它们总计超过 160,000 次修订,其主要发现中的第一条就是:数据库模式在应用的生命周期中频繁演变(Qiu, Li and Su, 2013)。这项研究针对的是应用数据库,而不是数据仓库,但每一次新增列的变更,都会带来一个电子表格没有列出的列。
以平台的电子表格为例,它是 crm.learners 的清单,导出为 CSV,每列一行。随后,CRM 团队在上文的拉取请求中提交的迁移新增了 work_email。迁移之后,将电子表格与目录对照,就能找出它遗漏了什么:
1"""A governance inventory kept as a spreadsheet, exported before a schema change, then compared with the catalog after it.2The point rests on the two work_email lines: the spreadsheet doesn't list the column, and the check finds it unclassified."""3import csv4import io56from lib import ci, metadata7from lib.catalog import open_catalog89platform = open_catalog(["crm.learners"])10described = metadata.load()1112# The inventory: one row per column of crm.learners, exported on the day someone was asked for one.13inventory = io.StringIO()14writer = csv.writer(inventory)15writer.writerow(["table", "column", "classification"])16for field in platform.load_table("crm.learners").schema().fields:17 # Metadata is keyed by Iceberg field ID, not by name, so the lookup survives a rename.18 writer.writerow(["crm.learners", field.name, described["crm.learners"].columns[field.field_id].classification])1920# Later, the CRM team's migration adds a column for learners' work email, applied as an Iceberg schema change.21ci.migrate(platform, "alter table crm.learners add column work_email varchar;")2223rows = list(csv.DictReader(io.StringIO(inventory.getvalue())))24print("columns the spreadsheet lists:", len(rows))25print("of them without a classification, by the spreadsheet:", sum(not row["classification"] for row in rows)) # a blank classification cell: a column the spreadsheet never classified26live = [field.name for field in platform.load_table("crm.learners").schema().fields]27print("columns crm.learners has:", len(live))28print("columns the spreadsheet doesn't list:", ", ".join(sorted(set(live) - {row["column"] for row in rows})))29for problem in metadata.problems(platform, described):30 print("the check against the catalog finds:", problem)
python spreadsheet-misses-new-column.pycolumns the spreadsheet lists: 15
of them without a classification, by the spreadsheet: 0
columns crm.learners has: 16
columns the spreadsheet doesn't list: work_email
the check against the catalog finds: crm.learners.work_email: not classified
这份电子表格看上去完整,却遗漏了一列。按它自己的记录,它列出了 15 列,其中有 0 列没有分类。该表实际有 16 列,而它没有列出的那一列是 work_email。电子表格本身无从显示这一点,因为没有任何东西把它与目录相比较。lib.metadata.problems(目录检查)会遍历目录并发现 crm.learners.work_email: not classified,这与导致CRM 团队的拉取请求失败的结论相同。根据我们的经验,手工维护的元数据副本,无论是电子表格、维基页面还是数据字典,在第一次模式变更时就会过时,而且不会有任何提示。
**工作准则:**绝不要把治理元数据放在没有任何东西检查的文档里。应把它放在经过评审的文件中,由测试在每个拉取请求上并按计划将其与线上目录比较。如果人们需要电子表格,就从该文件生成,并在表上注明生成时间。
一个工具强制执行、另一个工具绕过的策略
策略只保护经过其执行者的读取。Trino 只有在被告知从何处获取行过滤器时才会应用它们,因此如果没有 opa.policy.row-filters-uri 设置,它就完全不做行过滤(Trino 484, OPA access control)。Iceberg 的 REST 目录规范(1.12.0 版)允许目录随表返回读取限制,并规定:"A reader that supports read restrictions must fail any read against the loaded table that cannot apply a returned restriction in full"(REST catalog spec, 1.12.0)。规范为读取方提供了委托访问的信号,即 X-Iceberg-Access-Delegation 请求头,却没有为读取限制提供信号。因此,目录无法得知哪些读取方会应用它的限制。PyIceberg 0.12.0 的 REST 客户端一个也不读取(源码)。我们建议,返回读取限制的目录只向会应用这些限制的引擎提供存储访问权限。
强制执行层把 agent-a05 限制在 EMEA。不过,这一层存在绕过它的路径,因为任何拥有目录访问权限的对象都能读取表的文件。绕过这一层读取该表两次:一次是像笔记本那样,用 DuckDB 通过 lib.warehouse.connect 读取,这一层自身也使用它;另一次是用 PyIceberg 自带的扫描,就像任何持有目录访问权限的读取方那样:
1"""The same table, read three ways: through the enforcement layer, and around it with DuckDB and with PyIceberg. The point2rests on the two lines around the layer, which return every region where the layer returns only EMEA's."""3from collections import Counter45from lib import warehouse6from lib.catalog import open_catalog7from lib.enforce import Denied, Enforcer, answer89platform = open_catalog(["crm.learners"])10query = "select support_region, count(*) from crm.learners group by support_region order by support_region"111213def show(rows):14 return ", ".join(f"{region} {learners:,}" for region, learners in rows)151617layer = Enforcer(platform)18print("through the enforcement layer, as agent-a05:", show(answer(layer, query, "agent-a05")))1920# A notebook with DuckDB on the same catalog: the same SQL, and no policy in the way.21print("DuckDB on the same catalog:", show(warehouse.connect(platform).sql(query).fetchall()))2223# Any reader that holds the catalog's access, here PyIceberg's own scan: no SQL to rewrite at all.24scan = platform.load_table("crm.learners").scan(selected_fields=("support_region", "email")).to_arrow()25print("PyIceberg's scan of the table:", show(sorted(Counter(scan.column("support_region").to_pylist()).items())))26print("emails PyIceberg's scan returns:", f"{scan.num_rows - scan.column('email').null_count:,}") # rows whose email isn't null2728# lib.warehouse builds each table's view on a scan it registers under a name of its own. Through the layer, it's denied.29try:30 answer(layer, "select count(*) from scan_crm_learners", "agent-a05")31except Denied as reason:32 print("the layer, asked for the raw scan:", reason)
python read-around-the-layer.pythrough the enforcement layer, as agent-a05: emea 1,581
DuckDB on the same catalog: americas 2,276, apac 1,143, emea 1,581
PyIceberg's scan of the table: americas 2,276, apac 1,143, emea 1,581
emails PyIceberg's scan returns: 5,000
the layer, asked for the raw scan: not a table in the catalog: scan_crm_learners
这一层之外的读取方会得到所有区域的数据,因为策略从未被询问过。通过这一层时,智能体的查询返回 emea 1,581。在前面没有这一层的情况下,同样的 SQL 在 DuckDB 上返回 americas 2,276, apac 1,143, emea 1,581。PyIceberg 的扫描没有可改写的 SQL,返回 americas 2,276, apac 1,143, emea 1,581,其中有 5,000 个邮箱。每当第二个引擎、Spark 作业或持有存储凭据的脚本读取受管引擎所读的文件时,生产环境中也会发生同样的绕过。
lib.warehouse 把每张表的视图建立在一个扫描之上,该扫描以它自己的名称注册,这张表的名称是 scan_crm_learners,所以这一层的连接中同样持有原始扫描。输出的最后一行显示了查询指名该扫描时这一层的回答:not a table in the catalog: scan_crm_learners。这正是第 6 章所遵循的设计:在一处强制执行,并拒绝其他一切读取数据的操作,即不是目录表的任何对象。在生产环境中,这一处就是各工具用来查询的引擎,如第 6 章所论证的。而且由于存储权限是强制执行的一部分,只有该引擎和写入这些表的作业才应能读取这些文件。
工作准则: 选定一处强制执行策略,即所有工具都通过其查询的引擎,并且只把存储凭据交给它和写入这些表的作业,绝不交给个人或笔记本。在该处拒绝所有不通过目录表读取数据的查询。在把一张表算作受保护之前,列出所有能读取其文件的对象,并把任何不询问策略的读取方视为绕过,除非它是一个作业,其输出已像它所读取的表那样被编入目录并分级分类。
本章的决定
- **治理是同一个代码仓库中的四类代码:**一个元数据文件、发送每个作业血缘事件的代码、一份用 Rego 编写的访问策略,以及在每个拉取请求上运行的检查。其中只有一类是策略引擎。
- **每张表都有一个所属团队,**而不是某个人;每一列都属于四种分类之一:
public、internal、personal或sensitive。这些信息保存在governance/metadata.toml中,以 Iceberg 字段 ID 为键,并通过遍历目录,对照该文件来测试目录。 - **每次流水线运行都会发送 OpenLineage 事件,**其中的列级血缘由其 SQL 解析得出,包括用于连接和过滤行的列。
- **访问决策就是数据:**允许、行过滤和列掩码,均来自同一份默认拒绝的策略。执行层将这些决策应用于每次查询,并拒绝所有读取对象不是目录中表的请求。
- **元数据缺失时按失败关闭处理:**访问策略将无人分类的列视为
sensitive;无人描述的表,在每份报告中都会一直作为发现项保留,直到有人为其添加描述。 - **治理作业在每个拉取请求上运行,**对该拉取请求引入的发现项判定失败,并报告主分支上已有的发现项及其所属团队。
- **一切都在单个进程中离线运行,**使用本章开头列出的替身组件。以这种方式做治理,就是一组小规则,每条规则都在数据发生变化的地方检查,因此元数据中的缺口会表现为一次失败的检查,而不是报告中的意外。后续各章将依次在平台自身的表上测试每条规则。
本章,在本版次中
版次 2026.11.31示例于 2026年11月19日 通过本章 6个示例 个示例中有 6 个通过
译自英文版图书,版次 2026.11.31。