CI 中的数据治理
阅读约需 85分钟 分钟,14节中含10个示例。
读完本章,你可以把第 2 章到第 7 章的治理检查合并为一个作业,在每个拉取请求上运行,并且只评判该拉取请求所改动的内容。当某一列没有分类时,作业会失败;它会运行访问策略和强制执行层的测试,报告每次模式变更在下游影响到的范围,并强制执行两条破坏性变更检查器(即比较模式的两个版本、找出会破坏其读取方的变更的工具)不会检查的规则。你会把所有权保存在开放数据契约标准(Open Data Contract Standard,ODCS)的契约中,这些契约由元数据生成,每个领域对应一个文件夹和一个代码所有者;你还会在全局规则之上叠加各领域自己的规则,并以无人处理的发现来衡量每项检查的噪声。
前面各章为一个在线学习平台构建了治理检查,该平台的各团队每周都会修改其模式,而没有任何拉取请求会运行这些检查。第 2 章依据已审核的元数据测试目录,第 4 章从每次流水线运行中读取血缘。第 5 章测试访问策略,第 6 章测试在查询上强制执行该策略的那一层。第 7 章根据分类生成表属性。这些检查只有在有人运行相应示例时才会执行,团队修改平台时从不会执行。有三项变更即将到来。CRM 团队想增加一列,用来存放学习者的工作邮箱;学习团队想重命名一列,而报表流水线正在读取这一列;数据科学团队想不加掩码地查看学习者所在的城市。每一项都是评审者会批准的小改动,却都会改变什么会出问题,或者谁能看到什么。
本章加入全书共用的模块 lib.ci,它就是平台的治理作业。在本章中,这个作业指的是这个 CI 作业,而不是 OpenLineage 的作业。每个模拟的拉取请求都离线运行,表现为两个称为检出(checkout)的文件夹,一个是主分支的,一个是拉取请求分支的。每个文件夹都包含:governance/,即平台的治理文件;pipelines/,即其流水线的 SQL;migrations/,即分支用来变更模式的 alter table 语句,lib.ci.migrate 会把它们作为 Iceberg 模式变更来应用;以及由元数据生成的文件。对于每一侧,lib.ci.build 都按平台的实际运行方式构建一个临时平台:先建源表,再执行该侧的迁移,然后部署,部署会把元数据写入目录并设置每张表的生成属性,最后是该侧自有流水线的夜间运行,并带有血缘。在这些流水线中,review_corpus 仍是第 4 章中未经治理的副本,因此作业会继续检查它的血缘;平台则基于第 9 章的受治理构建进行训练。lib.ci.run 运行各项检查并比较两侧。整个过程不涉及 git 或网络,每个日期都与本书设定的“今天”,即 2026 年 10 月 1 日比较,而从不使用系统时钟。
这部分尚未翻译,因此以英文显示。
Run every check as one job
A check that runs when someone remembers it is a report, and reports go unread. On a pull request, a check is a gate: it runs on every change, before the change merges, and it fails the pull request through its exit code. That is how Open Policy Agent (OPA) advises running policies in CI, with flags, --fail and --fail-defined, that set the exit code to 1 based on query results (OPA CI/CD docs). A job made of several checks needs four things more:
- Every check runs and reports all its findings, so one failure doesn't hide the next. A check that crashes becomes one finding, "the check could not run", which fails the job while the other checks run.
- Every finding has a stable ID, so the same problem has the same name on every run and can be discussed, counted and waived.
lib.ciwrites it as the check and the subject, such ascoverage/crm.learners.work_email. - The job judges what the pull request changes. It runs the main branch's checks too, and compares the two. A finding the pull request introduces fails it. A finding already on main is reported on every run, counted and named with the team that owns it, but it fails no pull request that leaves it as it is. We judge that a check that fails every pull request for findings nobody in it caused gets switched off. Google's code-analysis platform, Tricorder, shows most of its analyses' results only on the lines a change touches, by default (Sadowski et al., 2015).
- The list of checks lives in a reviewed file,
governance/ci.toml, so switching one off is a diff that someone approves.
The platform's job runs these checks, in this order:
| Check | From | Fails when |
|---|---|---|
coverage |
Chapters 2 and 3 | A table has no metadata or owner, or a column no classification or one below its name's suggestion |
contract-rules |
This chapter | A contract breaks a global or a domain rule |
contracts |
This chapter | A generated file is out of date |
policy-tests |
Chapter 5 | A Rego test or a who-sees-what row fails |
enforcement-tests |
Chapter 6 | A regression case or a who-sees-what query fails |
migrations |
This chapter | Iceberg refuses a migration |
pipelines |
Chapter 4 | A pipeline can't run, or its lineage can't be read |
lineage-classifications |
Chapter 4 | A column is declared less protected than lineage proposes |
owner-review |
This chapter | A classification drops, or an owner or domain changes, unapproved |
removals |
This chapter | A column is removed before it was deprecated |
properties |
Chapter 7 | A table lacks the properties generated for it |
The first pull request the job sees is the CRM team's. Employers buy team licences for their staff, so the team adds work_email to crm.learners in one migration, and forgets the column's metadata:
1"""The platform's governance job on one pull request, run as a CI step runs it: the CRM team adds a work email to2crm.learners. The prose's point rests on the coverage finding for the new column and on the exit code the step returns."""34from lib import ci5from lib.platform import scratch67main = ci.checkout() # the main branch: governance files, pipelines and the files generated from them8branch = ci.checkout(main) # the pull request's branch, starting from main9(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")10print("files the pull request changes:", ", ".join(ci.changed(main, branch)))1112# What a CI step runs, `python -m lib.ci MAIN BRANCH --summary FILE`: it prints the report and its totals, writes the13# report as Markdown for the pull request, and exits with the code it returns.14summary = scratch() / "summary.md"15code = ci.main([str(main), str(branch), "--summary", str(summary)]) # 1 when a finding this pull request adds fails it16print("the step exits with:", code)17print("the summary it writes for the pull request:")18print(summary.read_text(encoding="utf-8"), end="")
python governance-ci-report.pyfiles the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
policy-tests: passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked
enforcement-tests: passed, findings 0, new 0, on main 0, waived 0, 44 cases checked
migrations: passed, findings 0, new 0, on main 0, waived 0, 1 migrations checked
pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
owner-review: passed, findings 0, new 0, on main 0, waived 0, 77 columns checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
properties: passed, findings 4, new 0, on main 4, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
properties/ai.review_corpus: 7 of 7 generated properties missing; on main, owner team-platform
properties/analytics.course_performance: 4 of 4 generated properties missing; on main, owner team-analytics
properties/analytics.learner_activity: 4 of 4 generated properties missing; on main, owner team-analytics
properties/marketing.audience: 7 of 7 generated properties missing; on main, owner team-platform
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
checks run: 11
checks failed: 1
findings: 7
findings the pull request introduces: 1
findings already on main: 6
findings waived: 0
findings approved: 0
exit code: 1
the step exits with: 1
the summary it writes for the pull request:
## Governance job: failed
| Check | Result | New | On main | Waived | Checked |
| --- | --- | ---: | ---: | ---: | --- |
| `coverage` | failed | 1 | 2 | 0 | 12 tables |
| `contract-rules` | passed | 0 | 0 | 0 | 10 contracts |
| `contracts` | passed | 0 | 0 | 0 | 11 files |
| `policy-tests` | passed | 0 | 0 | 0 | 24 tests and rows |
| `enforcement-tests` | passed | 0 | 0 | 0 | 44 cases |
| `migrations` | passed | 0 | 0 | 0 | 1 migrations |
| `pipelines` | passed | 0 | 0 | 0 | 4 pipelines |
| `lineage-classifications` | passed | 0 | 0 | 0 | 23 columns |
| `owner-review` | passed | 0 | 0 | 0 | 77 columns |
| `removals` | passed | 0 | 0 | 0 | 87 columns |
| `properties` | passed | 0 | 4 | 0 | 12 tables |
### To fix in this pull request
- `coverage/crm.learners.work_email`: not classified; its name suggests personal
### Already on main, and whose it is
- `coverage/ai.review_corpus`: no metadata (team-platform)
- `coverage/marketing.audience`: no metadata (team-platform)
- `properties/ai.review_corpus`: 7 of 7 generated properties missing (team-platform)
- `properties/analytics.course_performance`: 4 of 4 generated properties missing (team-analytics)
- `properties/analytics.learner_activity`: 4 of 4 generated properties missing (team-analytics)
- `properties/marketing.audience`: 7 of 7 generated properties missing (team-platform)
### Schema changes
- `crm.learners`, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
The pull request introduces 1 finding, and it is the one that fails the job: crm.learners.work_email is not classified; its name suggests personal. The suggestion comes from chapter 3's name rules, which match e_?mail, so the finding says what to add. Of the job's checks, 1 fails, and the step exits with code 1. The coverage findings on main are marketing.audience and ai.review_corpus, the tables nobody has described since chapter 2 found them. Neither table has an owner, so the job names the platform team, which ci.toml makes answerable for tables nobody owns. Chapter 4's lineage shows that both hold personal data, and every report repeats them until someone describes them.
The other checks pass, among them chapter 5's policy tests, run on the data the branch's files generate, and chapter 6's enforcement tests, run on the branch's tables. The properties check reports findings on main, such as 4 of 4 generated properties missing; on main, owner team-analytics: the tables pipelines write lose their generated properties on every nightly run, as chapter 7 found, on main as on the branch. The schema change, added as work_email string; named by 0 pipelines, reaches 0 columns, reaches nothing yet.
Working rule: Run every check as one job on every pull request, each reporting all its findings under stable IDs, and a crashed check as a failing finding. Fail the job by exit code on any finding the pull request introduces that is neither waived nor approved. Report every finding already on main, counted and named with its owner, without failing a pull request that leaves it as it is. Keep the list of checks in a reviewed file.
把关拉取请求,而不是提交
平台的任务在 CI 中运行。预提交钩子是检查可以运行的另一个位置,它在作者的机器上、提交产生之前运行。pre-commit 是一个 git 钩子框架,通常只对一次提交所改动的文件运行钩子,其文档将钩子描述为在代码评审之前发现简单问题的一种方式(pre-commit)。这适合只需要一个文件的检查:metadata.toml 能否解析,以及每个分类是否都是四个取值之一。
它不适合覆盖率检查。CRM 团队的拉取请求改动了migrations/0001-add-work-email.sql,却没有改动任何元数据文件,因此只针对元数据文件过滤的钩子本来没有任何东西可检查。缺口存在于迁移所改动的目录与拉取请求从未触及的文件之间,只有构建整个目录的任务才能发现它。钩子可以通过 pre-commit 的 always_run 设置为无论如何都运行,这样即使没有文件匹配它也会运行。但它只在有人安装过的地方运行,任何人都可以用 pre-commit 的 SKIP 变量或 git 的 --no-verify 跳过它,而评审者无法看到这两种跳过。
因此,把关的是拉取请求上的任务,并且要将其设为必需。GitHub 的分支保护可以要求所有必需的状态检查通过后,协作者才能把更改合并到受保护分支(GitHub,受保护分支),而不是必需的状态检查只是建议。发现结果出现在哪里同样重要。Google 的 Tricorder 作者发现,当开发者必须打开仪表盘或运行单独的命令行工具时,分析的使用率会下降(Sadowski et al., 2015)。那只是一家公司的情况,而且针对的是代码分析而非数据,但这正是 lib.ci.main 不止于返回退出码的原因。CI 步骤以 python -m lib.ci MAIN BRANCH 运行它,它为每条发现打印一行,包含其 ID 和需要补充的内容。它还会把报告写成 Markdown,放到评审者阅读差异的拉取请求中。发布摘要,以及从拉取请求的评审中读取批准,属于 CI 服务的职责,不在本书的讨论范围内。
工作准则: 用钩子在作者的机器上对单个文件做快速检查,并让拉取请求上的 CI 任务成为关卡,由主分支上的分支保护强制要求。把每条发现连同其 ID 和处理方式都放进拉取请求本身,绝不只放在仪表盘上。也要按计划针对生产目录运行这些检查,以覆盖在任何拉取请求之外发生变化的表。
报告模式变更的影响范围
CRM 团队新增的列没有影响到任何下游,但重命名会产生影响,而且它的三类读取方,即 Iceberg、SQL 和治理元数据,对重命名的判断各不相同。Iceberg 读取方按字段 ID 查找列,规范指出“projection must be done using field ids”(Iceberg table spec, 1.12.0),因此重命名后的列仍能读取重命名之前写入的所有文件。SQL 按名称查找列,所以同一次重命名会破坏所有引用该列名的查询和管道。平台的治理元数据以字段 ID 为键,这是第 2 章的决定,因此重命名只需要更新 ID 旁边的名称。规范中有效的基本类型提升包括:int 到 long,float 到 double,以及 decimal 的精度(只能加宽);在格式 v3 中还有 date 到 timestamp 或 timestamp_ns,以及类型为 unknown 的列到任意类型。其他任何变更都意味着需要新建一列。
一次变更会破坏哪些 SQL,是一个血缘问题,第 4 章通过遍历列图回答了它。评审也必须沿着 INDIRECT 边追踪,这是第 4 章对下面这类输入的称呼:它们决定输出包含哪些行,而不是决定输出的值是什么。OpenLineage 的列血缘 facet 为影响整个数据集的列血缘保留了一个 dataset 列表,这类影响来自过滤、排序、分组、连接、窗口函数等操作(ColumnLineageDatasetFacet, 1.54.0)。只用于连接或分组的列会影响它所作用的每一行,只报告复制值的报告会漏掉这一点。lib.ci.changes 按字段 ID 逐个比较两侧的模式。对于分支上每个被重命名、改变类型或被删除的列,它会遍历主分支的血缘图,找出 SQL 中引用该列的管道、值来自该列的列、它可能改变的每一列,以及拥有这些列的团队。
接下来是学习团队在迁移和元数据文件中将 learning.courses.category 重命名为 subject。同一个拉取请求还修复了引用该列的两条管道:
1"""What the learning team's rename of learning.courses.category reaches, through Iceberg readers, SQL and lineage. The2prose's point rests on the courses an Iceberg reader still finds, the columns reached, and the job's verdict once the fix merges with the rename."""3import pyarrow.compute as pc45from lib import ci, retention67main = ci.checkout()8branch = ci.checkout(main)9(branch / "migrations" / "0001-rename-category.sql").write_text(10 "alter table learning.courses rename column category to subject;\n")11ci.edit(branch, "governance/metadata.toml", '3 = { name = "category", classification = "public"',12 '3 = { name = "subject", classification = "public"') # the same field ID, renamed1314before, after = ci.build(main), ci.build(branch)15# matched by field ID, so the rename is one change, not a column dropped and another added16change = next(c for c in ci.changes(before, after) if c.table == "learning.courses")17print("change:", change.kind, "from", change.before, "to", change.after)18print("field ID, before and after:", change.field_id)19# read by the new name: Iceberg maps it to the same field ID as before20subjects = after.catalog.load_table("learning.courses").scan(selected_fields=("subject",)).to_arrow()21print("courses whose subject an Iceberg reader finds:", pc.count(subjects["subject"]).as_py())22print("pipelines whose SQL names it:", ", ".join(change.jobs))23for job, why in sorted(after.failed.items()):24 print(f"pipeline {job}, on the branch:", why)25print("columns whose values come from it:", len(change.values))26for column in change.values:27 print(" value of:", column)28print("columns it can change, DIRECT and INDIRECT:", len(change.reached))29# a column's table is the first two parts of its name, schema.table.column30print("tables those columns are in:", ", ".join(sorted({".".join(c.split(".")[:2]) for c in change.reached})))31print("owners the report goes to:", ", ".join(change.owners))32print("tables on main:", len(retention.tables(before.catalog)))33print("tables on the branch:", len(retention.tables(after.catalog)))34coverage = ci.run(before, after, checks=["coverage"]).checks[0]35print("coverage findings on the branch:", ", ".join(finding.id for finding in coverage.findings))3637# The same pull request, with both pipelines changed to read the new name. Each side runs its own pipelines/.38ci.edit(branch, "pipelines/course_performance.sql", " c.category,", " c.subject as category,")39ci.edit(branch, "pipelines/course_performance.sql", "c.title, c.category", "c.title, c.subject")40sql = (branch / "pipelines" / "marketing_audience.sql").read_text(encoding="utf-8")41(branch / "pipelines" / "marketing_audience.sql").write_text(sql.replace("c.category", "c.subject").replace(42 "f.category", "f.subject"), encoding="utf-8")43fixed = ci.build(branch)44ci.generate(fixed) # and the contracts generated again, as the author's hook would45print("files the fixed pull request changes:", ", ".join(ci.changed(main, branch)))46for line in ci.lines(ci.run(before, fixed, checks=["pipelines", "coverage", "contracts", "lineage-classifications"])):47 print("with the fix,", line)
python schema-change-impact.pychange: renamed from category string to subject string
field ID, before and after: 3
courses whose subject an Iceberg reader finds: 200
pipelines whose SQL names it: course_performance, marketing_audience
pipeline course_performance, on the branch: Unknown column: category
pipeline marketing_audience, on the branch: Unknown column: category
columns whose values come from it: 2
value of: analytics.course_performance.category
value of: marketing.audience.favourite_category
columns it can change, DIRECT and INDIRECT: 12
tables those columns are in: analytics.course_performance, marketing.audience
owners the report goes to: no owner, team-analytics
tables on main: 12
tables on the branch: 10
coverage findings on the branch: coverage/ai.review_corpus
files the fixed pull request changes: governance/contracts/learning/courses.odcs.yaml, governance/metadata.toml, migrations/0001-rename-category.sql, pipelines/course_performance.sql, pipelines/marketing_audience.sql
with the fix, pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
with the fix, coverage: passed, findings 2, new 0, on main 2, waived 0, 12 tables checked
with the fix, contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
with the fix, lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
with the fix, coverage/ai.review_corpus: no metadata; on main, owner team-platform
with the fix, coverage/marketing.audience: no metadata; on main, owner team-platform
with the fix, change learning.courses, field 3: renamed from category string to subject string; named by 2 pipelines, reaches 12 columns
with the fix, job: passed
对 Iceberg 而言,什么都没有损坏。重命名保留了该列的字段 ID,因此分支上的 Iceberg 读取方仍然能为 200 门课程找到 subject。对 SQL 而言,引用该列的管道,即 course_performance, marketing_audience,各自在向分支写入任何内容之前就停止了,因为第 4 章规定无法读取的血缘属于错误。就值而言,这次重命名影响到 2 列,即类别的两份副本。就所有边而言,它影响到 12 列,即 analytics.course_performance, marketing.audience 的每一列,因为类别用于对课程报告分组,并为每位学习者的最爱排序。报告发送给 no owner, team-analytics:分析团队,以及无人接收 marketing.audience 的报告,因为它还没有所有者。
失败的构建会掩盖它所破坏的内容。分支的目录中有 10 张表,而主分支上有 12 张,因为损坏的管道什么也没有写入。覆盖率检查只报告 coverage/ai.review_corpus:不存在的表无法被发现缺少描述,所以 marketing.audience 的发现项消失了。双方各自运行自己的 pipelines/,因此修复要与重命名一同提交:当两条管道都读取 subject 时,该作业报告 passed。在生产环境中,合并也可以保护评审所见的状态。REST 目录的提交可以携带 AssertCurrentSchemaId,它要求表的当前模式 ID 与该要求中指明的 ID 一致(REST catalog spec, 1.12.0),因此针对某一模式评审过的迁移会失败,而不会被应用到另一个模式上。
工作准则: 按字段 ID 比较模式,不要按名称或模式 ID 比较,并从三个方面报告每一次重命名、类型变更和删除:Iceberg 读取方是否受影响,哪些管道的 SQL 引用了该列,以及它通过血缘(包括 INDIRECT 边)影响到什么。把报告发送给它所涉及的每张表的所有者。会破坏管道的重命名,要与该管道的修复合并在同一个拉取请求中,绝不能先于修复合并。
降低分类前先询问所有者
破坏性变更检查器保护的是模式的读取者。dbt 的模型契约把删除列或更改列的数据类型视为可能影响下游使用方的破坏性变更(dbt 模型契约)。datacontract-cli 的 breaking 命令会比较两份 ODCS 契约,并让破坏了使用方所依赖保证的变更失败(datacontract-cli 1.2.7)。两者都没有把保护当作一项保证。在 datacontract-cli 1.2.7 中,没有任何规则涉及属性的分类或团队成员的角色,因此对二者的更改都会落入一条将其标记为未测试的规则(规则文件)。检测器以 INFO 级别报告这类变更(检测器文件),该级别不会导致失败。该命令仅在出现 ERROR 条目时才以 1 退出(命令实现文件、破坏性变更模型文件)。examples/requirements.txt 没有固定它的版本,所以没有任何示例运行它。2026 年 10 月 22 日,datacontract-cli 1.2.7 在本章为 crm.learners 生成的契约上运行,对被重命名的 email 以退出码 1 判定失败。尽管该属性保留了它的 id,它仍把重命名读作删除,因为它按名称匹配属性(规范化文件)。而对于 email 从 personal 重新分类为 public,以及所有者的变更,它都以退出码 0 放行。ODCS 本身也说分类“可以是任何值”(ODCS v3.2.0 模式)。
然而每项策略都会读取分类,因此降低分类是元数据文件所能携带的后果最严重的变更之一。我们加入第一条自己的规则:降低某列的分类、更改某表的所有者,或把某表移到另一个域,都需要在主分支上拥有该表的团队批准。移动算作所有权变更,因为域自身的规则适用于其下的表(后文一节)。所有者取自主分支,否则拉取请求可能把表交给新的所有者,而新所有者又可以批准这次降级。第 2 章为模型的草稿定下了同样的规则:只有评审者可以降低分类,并且要在该列的描述中写明理由。
数据科学团队的拉取请求排在最后。它把 crm.learners.city 从 personal 降为 internal,以便数据科学家按城市对学习者分组。访问策略会对数据科学家假名化个人列,而这次降级把 city 从该规则中移除。这次运行先后两次询问策略,再运行检查模式和策略的各项检查以及所有者规则:
1"""A pull request from the data-science team lowers crm.learners.city from personal to internal. The prose's point rests2on analyst-1's mask on each side, and on the owner rule failing alone."""3from lib import ci, policy45main = ci.checkout()6branch = ci.checkout(main)7ci.edit(branch, "governance/metadata.toml", '6 = { name = "city", classification = "personal"',8 '6 = { name = "city", classification = "internal"')9before, after = ci.build(main), ci.build(branch)1011reading_city = policy.request("analyst-1", "crm.learners", ["learner_id", "city"])12for side, name in ((before, "main"), (after, "the branch")):13 rego = policy.interpreter(side.governance / "policies", data=ci.policy_data(side))14 # a column with no mask in the decision is shown as it is stored15 print(f"analyst-1 reading city, on {name}:", policy.decide(reading_city, rego)["column_masks"].get("city", "unmasked"))1617checks = ["policy-tests", "enforcement-tests", "removals", "owner-review"]18report = ci.run(before, after, checks=checks)19print("schema changes:", len(report.changes))20for line in ci.lines(report):21 print(line)2223# The approvals come from the pull request's reviews: one entry per approving review, named by the reviewer's team.24# The owner rule passes when it raises no unapproved finding.25owner_review = lambda approvals: "passed" if ci.run(before, after, approvals, checks=["owner-review"]).passed else "failed"26print("owner-review, once team-crm approves:", owner_review(["team-crm"]))27print("owner-review, once team-analytics approves:", owner_review(["team-analytics"]))28ci.edit(after.root, "governance/ci.toml", "teams = {}", "teams = { team-crm = 2 }") # team-crm asks for two29# one approval from team-crm and one from another team: team-crm still needs a second of its own30print("owner-review, two required, one from team-crm:", owner_review(["team-crm", "team-platform"]))31print("owner-review, two required, two from team-crm:", owner_review(["team-crm", "team-crm"]))
python classification-downgrade.pyanalyst-1 reading city, on main: pseudonymize
analyst-1 reading city, on the branch: unmasked
schema changes: 0
policy-tests: passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked
enforcement-tests: passed, findings 0, new 0, on main 0, waived 0, 44 cases checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
owner-review: failed, findings 1, new 1, on main 0, waived 0, 77 columns checked
owner-review/crm.learners.city: lowered from personal to internal; needs team-crm's approval
job: failed
owner-review, once team-crm approves: passed
owner-review, once team-analytics approves: failed
owner-review, two required, one from team-crm: failed
owner-review, two required, two from team-crm: passed
元数据文件中的一个词就能让整个角色看到某列的明文,而除了所有者规则之外,没有任何检查会察觉。analyst-1 在主分支上读取 city 的结果是 pseudonymize,在该分支上则是 unmasked。作业报告的模式变更为 0,所以破坏性变更检查器没有可比较的内容。策略测试通过,即 passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked,强制执行测试也通过,因为它们都不读取 city;删除检查同样通过,因为没有任何列被删除。只有所有者规则失败,失败项为 lowered from personal to internal; needs team-crm's approval。
决定所有者规则结果的是所有者的批准。批准以每条批准评审一个条目的形式到达 lib.ci.run,并以评审者所属团队命名。有了 team-crm 的批准,所有者规则报告 passed;而有 team-analytics 的批准时,由于它并不拥有该表,结果为 failed。
工作准则: 把降低分类、更改所有者和移到另一个域视为需要该表在主分支上的所有者批准的变更,并在批准给出之前让作业失败;绝不要依赖破坏性变更检查器来处理这些变更。要求在同一个拉取请求中于该列的描述里写明理由。提高分类不需要这样的批准,因为它只会隐藏更多内容。
删除列之前先将其弃用
所有者规则防范的是一类破坏性变更检查工具会忽略的变更。删除列则是它们全都能发现的变更,但到那时删除已进入评审,对受其影响的读取方来说为时已晚。ODCS 3.2.0 增加了一种事先警告读取方的方式:契约中的任何元素,无论是模式对象还是属性,包括嵌套的属性,都可以设置可选的布尔字段 deprecated,表示不再建议使用。该标准还要求 description 指明替代方案并给出迁移指引(ODCS v3.2.0 schema)。我们的第二条规则把弃用警告变成必经步骤:只有主分支已将某列标记为弃用,该列才可以从表中移除。这需要两个拉取请求,一个负责弃用该列,另一个负责删除该列,两者之间的时间留给该列的读取方迁移。
平台的元数据文件原本没有这样的字段,因此本章增加了一个:在列的条目中写入 deprecated = true。lib.metadata.load 将其读取为 Column.deprecated,缺省时为 false,lib.ci.contract 则把它作为 ODCS 的 deprecated 写入该列的契约。
平台上最合适的候选是 billing.payments.card_number。它自己的描述写着"Should not be stored at all",而第 3 章发现它保存着完整的银行卡号。team-billing 对它执行了两次删除,每次都是在迁移中删除该列并删掉其元数据条目:一次是立即删除,另一次是在弃用它的拉取请求合并之后:
1"""team-billing removes the legacy card_number column from billing.payments: once straight away, and once after a pull2request that deprecates it has merged. The prose's point rests on the two removal lines."""3import yaml45from lib import ci67entry = ('8 = { name = "card_number", classification = "sensitive", '8 'description = "The full card number, from a legacy export. Should not be stored at all." }\n')9# the note goes after the description's sentence ending "at all.", and deprecated = true goes inside the entry's braces10deprecated_entry = entry.replace("all.", "all. Deprecated: reconcile payments by payment_id.").replace(" }", ", deprecated = true }")111213def remove_card_number(base, line):14 """A pull request on top of `base` that drops the column and deletes its metadata entry."""15 branch = ci.checkout(base)16 (branch / "migrations" / "0001-drop-card-number.sql").write_text("alter table billing.payments drop column card_number;\n")17 ci.edit(branch, "governance/metadata.toml", line, "")18 return branch192021main = ci.checkout()22deprecated = ci.checkout(main) # the first pull request: mark the column deprecated, and say what replaces it23ci.edit(deprecated, "governance/metadata.toml", entry, deprecated_entry)2425for base, line, name in ((main, entry, "straight away"), (deprecated, deprecated_entry, "after the deprecation merged")):26 report = ci.run(ci.build(base), ci.build(remove_card_number(base, line)), checks=["removals"])27 print(f"removal, {name}:", "passed" if report.checks[0].passed else "failed")28 for finding in report.checks[0].findings:29 print(f" {finding.id}:", finding.message)3031# property 7 is the eighth column, field 8: card_number, the column the deprecation marks32card_number = ci.contract(ci.build(deprecated), "billing.payments")["schema"][0]["properties"][7]33print("the deprecated column in its ODCS contract:")34print(yaml.safe_dump(card_number, sort_keys=False, width=200), end="")
python deprecate-then-remove.pyremoval, straight away: failed
removals/billing.payments.card_number: removed without being deprecated on the main branch first
removal, after the deprecation merged: passed
the deprecated column in its ODCS contract:
id: field-8
name: card_number
logicalType: string
physicalType: string
required: false
description: 'The full card number, from a legacy export. Should not be stored at all. Deprecated: reconcile payments by payment_id.'
classification: sensitive
deprecated: true
立即删除时,检查失败,removed without being deprecated on the main branch first。弃用合并之后,同样的删除则报告 passed。在此期间,契约把警告传达给了所有读取契约的人,而不只是代码仓库中的人:deprecated: true,其描述说明了如何在没有该列的情况下对账支付。该规则检查的是两次变更的先后顺序,而不是两者之间的时间间隔。给读取方留多长时间,由所属团队与该列的读取方共同决定,影响报告会说明这些读取方是谁。
工作准则: 分两个拉取请求删除一列:先将其标记为弃用,并在描述中写明替代方案,然后合并;再从表中删除该列,并从元数据中删除它的条目。凡是删除主分支尚未弃用的列,一律判为失败。弃用期持续多久,要与该列的读取方商定,因为该任务只检查弃用是否在先。
这部分尚未翻译,因此以英文显示。
Route each domain's reviews by path
The owner rule asks a table's team to approve its downgrades, and the repository can route that request. GitHub requests code owners' review automatically when a pull request modifies code they own, and a branch protection rule can require that review (GitHub's documentation, About code owners). The platform keeps all its metadata in one file, governance/metadata.toml, which suits 5 teams in one repository. A file has one set of owners, though, so the request can reach the right team only if each team's metadata lives under a path of its own.
ODCS draws the same lines. A contract names one domain, the logical data domain it belongs to (ODCS v3.2.0 fundamentals). ODCS has no owner field: ownership is a free-form role on a member of the contract's team (ODCS v3.2.0 team). lib.ci.contract writes each described table as an ODCS contract. It records the domain, the owning team as a member with the role owner, and each column as a property whose id holds its Iceberg field ID. ODCS defines a property's id as "A unique identifier for the element used to create stable, refactor-safe references". lib.ci.generate writes each contract into a folder for its domain, and a CODEOWNERS file in .github/, the first place GitHub looks for one. Both are generated from the metadata, never typed. Ask whom GitHub would request for the data-science team's downgrade of city:
1"""Ownership as ODCS contracts, a folder per domain, and who GitHub asks to review the downgrade of crm.learners.city2when the metadata is one shared file, and when each domain has a folder of contracts generated from it. The prose's3point rests on the first review line and on the contracts check failing before the files are generated again."""4import yaml56from lib import ci78main = ci.checkout() # the main branch holds the contracts and CODEOWNERS generated from its metadata9branch = ci.checkout(main)10ci.edit(branch, "governance/metadata.toml", '6 = { name = "city", classification = "personal"',11 '6 = { name = "city", classification = "internal"')1213shared = ["/governance/ @example-org/team-platform"] # one file for every domain, owned by the platform team14for path, owners in ci.reviewers(shared, ["governance/metadata.toml"]).items():15 print(f"one shared file, review of {path}:", " ".join(owners))1617crm = yaml.safe_load((main / "governance/contracts/crm/learners.odcs.yaml").read_text(encoding="utf-8"))18print(yaml.safe_dump({"domain": crm["domain"], "team": crm["team"]}, sort_keys=False), end="")19lines = (main / ".github/CODEOWNERS").read_text(encoding="utf-8").splitlines()20for line in lines[1:]: # the first line says the file is generated21 print("CODEOWNERS", line)22before, after = ci.build(main), ci.build(branch) # the pull request changes the metadata only23print("teams that own tables:", len({table.owner for table in before.described.values()}))24# drops the job's verdict, the last line, which this example doesn't print25for line in ci.lines(ci.run(before, after, checks=["contracts"]))[:-1]:26 print("before generating again,", line)27print("files generated again:", len(ci.generate(after)))28print("after generating again, contracts:", "passed" if ci.run(before, after, checks=["contracts"]).passed else "failed")29# the last matching CODEOWNERS line wins, as GitHub applies it30for path, owners in ci.reviewers(lines, ci.changed(main, branch)).items():31 print(f"a folder per domain, review of {path}:", " ".join(owners))
python domain-contracts.pyone shared file, review of governance/metadata.toml: @example-org/team-platform
domain: crm
team:
name: team-crm
members:
- username: team-crm
role: owner
CODEOWNERS /.github/CODEOWNERS @example-org/team-platform
CODEOWNERS /governance/ @example-org/team-platform
CODEOWNERS /migrations/ @example-org/team-platform
CODEOWNERS /pipelines/ @example-org/team-platform
CODEOWNERS /pipelines/course_performance.sql @example-org/team-analytics
CODEOWNERS /pipelines/learner_activity.sql @example-org/team-analytics
CODEOWNERS /governance/contracts/analytics/ @example-org/team-analytics
CODEOWNERS /governance/contracts/billing/ @example-org/team-billing
CODEOWNERS /governance/contracts/crm/ @example-org/team-crm
CODEOWNERS /governance/ci/domains/crm.rego @example-org/team-crm
CODEOWNERS /governance/contracts/learning/ @example-org/team-learning
CODEOWNERS /governance/contracts/support/ @example-org/team-support
teams that own tables: 5
before generating again, contracts: failed, findings 1, new 1, on main 0, waived 0, 11 files checked
before generating again, contracts/governance/contracts/crm/learners.odcs.yaml: differs from what the metadata generates: generate it again
files generated again: 11
after generating again, contracts: passed
a folder per domain, review of governance/contracts/crm/learners.odcs.yaml: @example-org/team-crm
a folder per domain, review of governance/metadata.toml: @example-org/team-platform
With one shared file, the downgrade changes governance/metadata.toml, and GitHub asks @example-org/team-platform to review it. That is the team that owns the file, not team-crm, whose column it is. The generated CODEOWNERS gives the platform team the file itself, because About code owners says a repository is fully protected against unauthorized changes only when the CODEOWNERS file itself has an owner (GitHub's documentation). The platform team also gets the governance files, the migrations and the pipelines. Each pipeline then goes to the team that owns its table, and each folder of contracts goes to its domain's team. The pull request changed only the metadata, so the contracts check, which generates every file again and compares, fails: differs from what the metadata generates: generate it again. Generated again, the pull request also changes the crm contract, and its review goes to @example-org/team-crm, because GitHub gives the last matching pattern precedence. A hook that generates the files saves the author a failed run. The check makes the routing certain.
When code-owner review is required, an approval from any one of a file's owners meets it (About code owners). Branch protection asks for one number of approvals for the whole branch, met by reviewers with write permissions (GitHub, protected branches), so any such reviewer counts toward it, and it can't ask for two of team-crm's members. The job can, from the approvals it's given. The owner rule counts them per team, against a number per team in ci.toml. With team-crm asking for two, the downgrade of city with one team-crm approval reports failed, and with two it reports passed. Iceberg 1.12.0's REST catalog spec offers another place for classifications, labels per table and field ID (REST catalog spec). Coverage and the contract rules could read a catalog server's labels, but changes to them would need the server's own review, since the spec doesn't require that labels be persisted or versioned.
Working rule: Keep each domain's metadata under a path of its own, as one ODCS contract per table with one domain and one owning team, generated from the metadata and checked by the job, with a CODEOWNERS line for that team. Give the platform team the CODEOWNERS file, the job's settings, its global rules, the waivers and the migrations. Require code-owner review, and count approvals per team in the job when one isn't enough.
在全局规则之上叠加领域规则
拥有各自领域契约的团队仍然共用同一个平台,而且有些规则必须在每个领域中都成立。Dehghani 在描述数据网格时把这称为联邦式计算治理,即“一组全局规则——适用于所有数据产品及其接口的规则——以确保生态系统健康且可互操作”。这些全局规则与各领域自行做出的决定并列存在,两者都由平台执行(Dehghani, 2020)。这是一种框架性的说法,而非测量结果,但它可以对应到代码上。平台团队维护一个全局规则包,由各领域共同认可;在 Dehghani 的版本中,这些规则由领域产品负责人和平台产品负责人共同决定。每个领域有自己的规则包,由该领域的团队负责,并由同一个任务运行所有规则包。
OPA 项目中针对文件的规则工具是 Conftest,它使用 Rego 测试结构化的配置数据(Conftest)。lib.ci 以进程内方式运行同类规则,使用的是 regopy,与第 5 章运行访问策略的方式相同。governance/ci/global.rego 存放全局规则:只能有一个负责人,无论其角色的大小写如何;只能属于一个领域,缺失、为空或仅含空白的值都视为不通过,因为空字符串在 Rego 中是已定义的值,单靠 not 会让它通过;分类必须取自四个类别之一,因为 ODCS 接受任意字符串;每一列都必须有描述。governance/ci/domains/crm.rego 存放 team-crm 的规则:crm 中没有任何内容是公开的,因为其中的一切都描述学员或讲师。contract-rules 检查会对每份契约运行全局规则包,并对该分支上属于某个领域的契约运行该领域的规则包。规则以数据的形式返回每一项违规,包含规则名称、主体和消息。违规的主体与规则名称共同构成该发现项的 ID。这些文件位于访问策略所在的文件夹 governance/policies/ 之外,因此修改它们不会改变第 8 章随每次决策一并记录的策略修订版本。
CRM 团队的变更再次提交,这次带上了元数据。work_email 被分类为 public,理由是工作邮箱地址属于业务信息,而它的描述为空。第二次尝试加上了描述,并把 crm.learners 移到没有自身规则的分析领域:
1"""The platform's global rules and the crm domain's own rule, run on the CRM team's two tries at work_email: classified2public with no description, then described with crm.learners moved to the analytics domain, which has no rules of its own.3The prose's point rests on the two contract-rules findings of the first try and on the owner-review finding after the move."""4from lib import ci56main = ci.checkout()7branch = ci.checkout(main)8(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")9ci.edit(branch, "governance/metadata.toml", '15 = { name = "notes"',10 '16 = { name = "work_email", classification = "public", description = "" }\n15 = { name = "notes"')1112for path in sorted((branch / "governance" / "ci").rglob("*.rego")):13 print("rules:", path.relative_to(branch).as_posix())14before = ci.build(main)15report = ci.run(before, branch, checks=["coverage", "contract-rules"])16for line in ci.lines(report):17 print(line)1819# The second try: a description, and the table moved to a domain whose rules allow public columns.20ci.edit(branch, "governance/metadata.toml", 'description = "" }', 'description = "The learner\'s work address." }')21# the table's domain changes from crm to analytics, which has no rules of its own22ci.edit(branch, "governance/metadata.toml", 'owner = "team-crm"\ndomain = "crm"\ndescription = "Every learner',23 'owner = "team-crm"\ndomain = "analytics"\ndescription = "Every learner')24report = ci.run(before, branch, checks=["coverage", "contract-rules", "owner-review"])25for line in ci.lines(report):26 print("moved to analytics,", line)
python global-and-domain-rules.pyrules: governance/ci/domains/crm.rego
rules: governance/ci/global.rego
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: failed, findings 2, new 2, on main 0, waived 0, 10 contracts checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: classified public; its name suggests personal; needs team-crm's approval
coverage/marketing.audience: no metadata; on main, owner team-platform
contract-rules/crm.learners.work_email/described: no description (global rule described)
contract-rules/crm.learners.work_email/no-public: crm classifies no column public (domains.crm rule no-public)
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
moved to analytics, coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
moved to analytics, contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
moved to analytics, owner-review: failed, findings 1, new 1, on main 0, waived 0, 77 columns checked
moved to analytics, coverage/ai.review_corpus: no metadata; on main, owner team-platform
moved to analytics, coverage/crm.learners.work_email: classified public; its name suggests personal; needs team-crm's approval
moved to analytics, coverage/marketing.audience: no metadata; on main, owner team-platform
moved to analytics, owner-review/crm.learners/domain: domain changed from crm to analytics; needs team-crm's approval
moved to analytics, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
moved to analytics, job: failed
此时覆盖检查标记了分类问题:classified public; its name suggests personal; needs team-crm's approval。名称只是一种证据,而拉取请求里没有可供读取的值,因此低于名称所提示级别的分类,是一项只有主干分支上的负责人才能回答的发现。
契约规则要求更多。全局规则包标记了 no description (global rule described),crm 规则包标记了 crm classifies no column public (domains.crm rule no-public)。team-crm 无需询问任何其他团队就决定了第二条规则,平台在每个拉取请求上为他们运行这条规则,包括来自 team-crm 的拉取请求。
第二次尝试时,适用的是分析领域的规则,契约规则全部通过:passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked。但这次迁移本身在 domain changed from crm to analytics; needs team-crm's approval 上失败,分类同样失败。
**工作准则:**把每个领域都必须遵守的规则写成一个全局规则包,由平台团队维护,并让每个领域增加一个由其团队负责、只对该领域契约运行的规则包。以带有稳定主体的数据形式返回违规,而不是只返回简单的通过或失败。领域规则只能补充全局规则,不能取代它们;并且要把表迁移到另一个领域视为所有权变更,这样任何团队都无法通过改动一个词来逃避其所在领域的规则。
统计每项检查中被豁免的发现
迄今为止的各项检查都会因某个理由使拉取请求失败。但当人们不再相信一道关卡时,它同样会失效。Google 的 Tricorder 作者将有效误报定义为用户选择不采取行动的任何报告,并把代码评审阶段的这一比率控制在 10% 以下。他们实际采取行动的对象范围更窄,即“无用率”:某个分析器的“无用”点击数,占其“无用”“请修复”和“应用修复”点击总数的比例。达到 10% 或以上时,该分析器进入观察期;超过 25% 时,可能被立即关闭(Sadowski et al., 2015)。这是一家公司的代码分析,阈值由其自行选定,而对于数据治理,我们尚未见到已发表的对应标准。这一度量可以作为类比来借用。对每项检查,统计无人处理的发现数,无论是已豁免的,还是留在 main 上的。
在平台上,对新发现决定不采取行动,就是一次豁免。governance/waivers.toml 中的一项豁免按 ID 指明一个发现、对其负责的团队、一条理由,以及其生效的最后一天。lib.ci 只接受自作业运行当天起最长不超过 90 天的豁免。这一上限是保存在 governance/ci.toml 中的经验法则,因此每项豁免都会回来接受一次决定。同一文件还列出了无论日期如何都不能用豁免压制的检查:覆盖率检查,因为没有负责人的表或没有分类的列应当在拉取请求中修复;以及各项强制执行测试。已在 main 上的发现不需要豁免。过期的豁免不会压制任何发现,报告会将其列出。平台团队负责豁免文件,因此每项豁免也是一次经过评审的变更。
CRM 的变更需要两项豁免。第一项针对未分类的列。第二项针对该列被分类为 personal 之后,尚未撰写的描述。团队起初请求第二项运行到夏季,随后改为运行到 10 月 31 日:
1"""Waivers on the CRM team's pull request: one for the unclassified work_email column, which coverage never allows;2then, with the column classified but not yet described, one that asks for too long, and one that fits, on the book's3today and a month later. The prose's point rests on the two refusals and on the expired waiver the report lists."""4from datetime import date56from lib import ci7from lib.platform import TODAY8910def waive(branch, finding, expires, reason):11 with open(branch / "governance" / "waivers.toml", "a", encoding="utf-8") as waivers:12 # the date is written unquoted, so TOML reads it as a date, which the job compares with the day it runs13 waivers.write(f'\n[[waivers]]\nfinding = "{finding}"\nowner = "team-crm"\nreason = "{reason}"\nexpires = {expires}\n')141516main = ci.checkout()17before = ci.build(main)18unclassified = ci.checkout(main)19(unclassified / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")20waive(unclassified, "coverage/crm.learners.work_email", "2026-10-31", "Classified once the sign-up form is live.")21for line in ci.lines(ci.run(before, unclassified, checks=["coverage"])):22 print("unclassified,", line)2324undescribed = ci.checkout(main) # the second try: classified, with its description still to come25(undescribed / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")26ci.edit(undescribed, "governance/metadata.toml", '15 = { name = "notes"',27 '16 = { name = "work_email", classification = "personal", description = "" }\n15 = { name = "notes"')28waive(undescribed, "contract-rules/crm.learners.work_email/described", "2027-06-30", "Described once the form's copy is final.")29after = ci.build(undescribed)30for line in ci.lines(ci.run(before, after, checks=["coverage", "contract-rules"])):31 print("undescribed, asked until 2027-06-30,", line)3233# the same waiver, shortened to the end of October, within the longest a waiver may run34ci.edit(undescribed, "governance/waivers.toml", "2027-06-30", "2026-10-31")35for day in (TODAY, date(2026, 11, 1)): # the book's today, then a month later, after the waiver has expired36 report = ci.run(before, after, today=day, checks=["coverage", "contract-rules"])37 print(f"undescribed, asked until 2026-10-31, on {day}, contract-rules:", "passed" if report.checks[1].passed else "failed")38 print(f"undescribed, asked until 2026-10-31, on {day}, findings waived:", sum(len(c.waived) for c in report.checks))39 print(f"undescribed, asked until 2026-10-31, on {day}, waivers that silence nothing:",40 "; ".join(why for _, why in report.unused) or "none")
python waived-findings.pyunclassified, coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
unclassified, coverage/ai.review_corpus: no metadata; on main, owner team-platform
unclassified, coverage/crm.learners.work_email: not classified; its name suggests personal
unclassified, coverage/marketing.audience: no metadata; on main, owner team-platform
unclassified, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
unclassified, waiver coverage/crm.learners.work_email: silences nothing, coverage findings are never waived
unclassified, job: failed
undescribed, asked until 2027-06-30, coverage: passed, findings 2, new 0, on main 2, waived 0, 12 tables checked
undescribed, asked until 2027-06-30, contract-rules: failed, findings 1, new 1, on main 0, waived 0, 10 contracts checked
undescribed, asked until 2027-06-30, coverage/ai.review_corpus: no metadata; on main, owner team-platform
undescribed, asked until 2027-06-30, coverage/marketing.audience: no metadata; on main, owner team-platform
undescribed, asked until 2027-06-30, contract-rules/crm.learners.work_email/described: no description (global rule described)
undescribed, asked until 2027-06-30, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
undescribed, asked until 2027-06-30, waiver contract-rules/crm.learners.work_email/described: silences nothing, runs to 2027-06-30, past the longest allowed, 2026-12-30
undescribed, asked until 2027-06-30, job: failed
undescribed, asked until 2026-10-31, on 2026-10-01, contract-rules: passed
undescribed, asked until 2026-10-31, on 2026-10-01, findings waived: 1
undescribed, asked until 2026-10-31, on 2026-10-01, waivers that silence nothing: none
undescribed, asked until 2026-10-31, on 2026-11-01, contract-rules: failed
undescribed, asked until 2026-10-31, on 2026-11-01, findings waived: 0
undescribed, asked until 2026-10-31, on 2026-11-01, waivers that silence nothing: expired on 2026-10-31
针对未分类列的豁免被拒绝,尽管它只请求一个月:silences nothing, coverage findings are never waived。没有任何豁免能让该拉取请求通过。针对描述的豁免在运行到夏季时被拒绝,silences nothing, runs to 2027-06-30, past the longest allowed, 2026-12-30。缩短到 10 月 31 日后,它就能压制该发现。10 月 1 日,合约规则报告 passed。11 月 1 日,它们报告 failed。10 月 1 日,不压制任何发现的豁免清单是 none;11 月 1 日则是 expired on 2026-10-31:日期已经过去,问题却没有消失。
工作准则: 让一项豁免只指明拉取请求引入的某一个发现,并带有负责团队、一条理由和最多几个月之后的截止日期,同时列出任何豁免都不得压制的检查,例如缺失的分类或负责人。对每项检查报告有多少发现无人处理,无论是已豁免的还是留在 main 上的,并评审其中大部分发现都无人处理的任何检查:要么检查有问题,要么平台有问题,而这个数字本身无法说明是哪一种。
生产环境中会出什么问题
依我们的判断,治理作业被关掉的次数多于它出故障的次数:团队会移除那些运行缓慢或告警嘈杂的检查,每次只改动 governance/ci.toml 中的一行,而关卡依然显示通过,实际守住的范围却变小了。下面用两项分阶段检查说明这一过程,其中以计数代替耗时;第三个例子则展示一个出了故障的作业。
逐行读取的检查
支持团队向 support.tickets 添加了两列:callback_number,分类为 internal;以及 reply_email,分类为 public。如果团队希望不仅检查列名,还要检查取值,就会把第 3 章对该表的分类加入这项作业,分别按抽样和逐行两种方式进行:
1"""The support team's pull request adds two columns to support.tickets, which coverage judges by their names and chapter23's value checks judge by reading a sample, or every row. The prose's point rests on the callback number's empty column3and on the values each check reads; counts of values stand in for time, which depends on the machine."""4import pyarrow as pa56from lib import ci, pii78main = ci.checkout()9branch = ci.checkout(main)10(branch / "migrations" / "0001-callback-and-reply.sql").write_text(11 "alter table support.tickets add column callback_number varchar;\n"12 "alter table support.tickets add column reply_email varchar;\n")13ci.edit(branch, "governance/metadata.toml", '8 = { name = "status"',14 '9 = { name = "callback_number", classification = "internal", description = "A number to call the learner back on." }\n'15 '10 = { name = "reply_email", classification = "public", description = "Where the reply goes." }\n8 = { name = "status"')16before, after = ci.build(main), ci.build(branch)1718report = ci.run(before, after, checks=["coverage", "classify-sampled", "classify-every-row"])19for name in ("body", "callback_number", "reply_email"):20 values = after.catalog.load_table("support.tickets").scan(selected_fields=(name,)).to_arrow()[name]21 print(f"support.tickets.{name}, values on the branch:", f"{len(values) - values.null_count:,}")22 for check in report.checks:23 found = [f.message for f in check.findings if f.id == f"{check.name}/support.tickets.{name}"]24 print(f"support.tickets.{name}, {check.name}:", found[0] if found else "no finding")25for check in report.checks[1:]:26 print(f"{check.name}, values read:", f"{check.checked:,}")2728text_columns, sampled, every_row = 0, 0, 029for name in sorted(before.described):30 data = before.catalog.load_table(name).scan().to_arrow()31 # the non-null values of each text column, the ones a value check classifies32 filled = [len(column) - column.null_count for column in data.columns if pa.types.is_string(column.type)]33 text_columns += len(filled)34 # the sampled check reads at most pii.ROWS values of each column35 sampled += sum(min(pii.ROWS, count) for count in filled)36 every_row += sum(filled)37print("text columns in the described tables:", text_columns)38print("values the sampled check would read in them:", f"{sampled:,}")39print("values the every-row check would read in them:", f"{every_row:,}")
python slow-check-every-row.pysupport.tickets.body, values on the branch: 2,000
support.tickets.body, coverage: no finding
support.tickets.body, classify-sampled: values support sensitive, reviewed as personal
support.tickets.body, classify-every-row: values support sensitive, reviewed as personal
support.tickets.callback_number, values on the branch: 0
support.tickets.callback_number, coverage: no finding
support.tickets.callback_number, classify-sampled: no finding
support.tickets.callback_number, classify-every-row: no finding
support.tickets.reply_email, values on the branch: 0
support.tickets.reply_email, coverage: classified public; its name suggests personal; needs team-support's approval
support.tickets.reply_email, classify-sampled: its name suggests personal, reviewed as public
support.tickets.reply_email, classify-every-row: its name suggests personal, reviewed as public
classify-sampled, values read: 1,000
classify-every-row, values read: 10,000
text columns in the described tables: 32
values the sampled check would read in them: 5,800
values the every-row check would read in them: 429,453
覆盖检查按名称判定 reply_email 不通过,即 classified public; its name suggests personal; needs team-support's approval;而 callback_number 通过,即 no finding,因为没有任何名称规则与它匹配。该分支中新列包含 0 个值,所以读取取值同样帮不上忙。迁移添加的是空列,每一行得到的结果都是 no finding。待该列填入数据后,第 3 章中按计划运行的抽样会发现其中的电话号码。
抽样检查读取 1,000 个值,逐行检查读取 10,000 个值。两者都认定 support.tickets.body 在个人信息分类下属于敏感数据,而第 3 章已经把它交给了 team-support。对已描述的各表中的所有文本列,抽样会读取 5,800 个值,逐行则会读取 429,453 个值。后一个数字随数据增长,而且每个拉取请求都要付出这样的代价,无论该拉取请求改动了什么。我们认为,让人久等的作业会被跳过,随后被移除。
工作准则: 拉取请求检查的规模应由变更决定,而不是由数据决定:检查元数据、模式、策略,以及拉取请求所添加列的名称。读取取值的检查应按计划运行,使用固定的样本,如第 3 章所做的那样,并把检查结果交给所属团队。要记录作业的检查在何处止步,例如添加空列的情形。
标记每个 ID 的检查
一个谨慎的团队新增了一项检查,把所有名称形如 ID 的列都当作个人数据。平台对其用于连接的 ID 另有决定:第 2 章将 learner_id 归类为 internal,并保护用于解析它的表。这项检查的发现都是新增的,因此各负责团队都对其予以豁免:
1"""A noisy check added to the job: every column named like an ID is personal. The platform decided otherwise for IDs2used for joining, so each owning team waives its findings, and the waiver count climbs. The prose's point rests on the3not-acted-on line of each check."""4from collections import Counter56from lib import ci78main = ci.checkout()9branch = ci.checkout(main) # the platform team adds the check to the job's settings10ci.edit(branch, "governance/ci.toml", '"properties"]', '"properties", "id-columns"]')11before, after = ci.build(main), ci.build(branch)1213report = ci.run(before, after)14print("waivers in the file, before:", len(ci.waivers(after.governance)))15noisy = next(check for check in report.checks if check.name == "id-columns")16print("job, before the waivers:", "passed" if report.passed else "failed")17print("id-columns, findings:", len(noisy.findings))18print("id-columns, findings on main:", len(noisy.on_main))19# the last word of each message is the classification20classified = Counter(finding.message.rpartition(" ")[2] for finding in noisy.findings)21for level, count in sorted(classified.items()):22 print(f"id-columns, findings classified {level}:", count)23# the part of the finding's ID after the check's name: the column's full name24print("id-columns, the public one:", ", ".join(f.id.split("/")[1] for f in noisy.findings if f.message.endswith("public")))2526with open(after.governance / "waivers.toml", "a", encoding="utf-8") as waivers:27 for finding in noisy.findings:28 waivers.write(f'\n[[waivers]]\nfinding = "{finding.id}"\nowner = "{finding.owner}"\n'29 'reason = "An ID for joining, internal by decision."\nexpires = 2026-10-31\n')3031report = ci.run(before, after)32print("waivers in the file, after:", len(ci.waivers(after.governance)))33print("job, with the waivers:", "passed" if report.passed else "failed")34for check in (check for check in report.checks if check.findings):35 idle = [f for f in check.findings if f.waiver or f.on_main] # findings nobody acted on36 print(f"{check.name}, not acted on:", f"{len(idle)} of {len(check.findings)} findings, {len(idle) / len(check.findings):.0%}: "37 f"{len(check.waived)} waived, {len(check.on_main)} on main")
python noisy-check-waivers.pywaivers in the file, before: 0
job, before the waivers: failed
id-columns, findings: 21
id-columns, findings on main: 0
id-columns, findings classified internal: 20
id-columns, findings classified public: 1
id-columns, the public one: learning.courses.course_id
waivers in the file, after: 21
job, with the waivers: passed
coverage, not acted on: 2 of 2 findings, 100%: 0 waived, 2 on main
properties, not acted on: 4 of 4 findings, 100%: 0 waived, 4 on main
id-columns, not acted on: 21 of 21 findings, 100%: 21 waived, 0 on main
该检查发现了 21 个列,每个都是平台有意分类的 ID。其中 20 个为 internal,learning.courses.course_id 为 public。
豁免文件中的豁免数从 0 条增至 21 条,ID 检查显示为 21 of 21 findings, 100%: 21 waived, 0 on main。这已超过 Tricorder 的作者所用的任何阈值,此处仅作类比,因为他们统计的是点击数,而不是豁免数。属性检查显示出相同的占比,即 4 of 4 findings, 100%: 0 waived, 4 on main,而且理应如此:它的发现是流水线运行器所需的修复。计数无法区分这两种情形,原因却可以。
**工作准则:**添加检查之前,先在主分支上运行它并阅读每一条发现;如果大多数发现都会被豁免,就修改规则,而不是豁免其发现。对于发现长期大多被豁免的检查,应将其移除;对于正确的检查出现高占比,则应视为需要安排的工作,并指定负责人和日期。
损坏的文件会成为一条发现项
一位支持人员的条目写入 governance/users.toml 时少了一个引号。同一个拉取请求还把 crm.learners.country 从文本改成了数字,并且带有 CRM 团队尚未分类的 work_email:
1"""A pull request that breaks two files the job reads: a support agent added to the users file with a quote missing,2and a migration Iceberg refuses. It also carries the CRM team's unclassified work_email column. The prose's point rests3on the two checks that could not run and on the exit code."""4from lib import ci56main = ci.checkout()7branch = ci.checkout(main)8# the closing quote after "support" is missing, so the users file no longer parses9ci.edit(branch, "governance/users.toml", '[users.agent-a09]', '[users.agent-a12]\ngroups = ["support]\nregion = "emea"\n\n'10 '[users.agent-a09]')11(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")12# a string column cannot become an int: that isn't one of Iceberg's type promotions, so the migration is refused13(branch / "migrations" / "0002-country-as-number.sql").write_text("alter table crm.learners alter column country type int;\n")14print("files the pull request changes:", ", ".join(ci.changed(main, branch)))1516report = ci.run(main, branch)17for line in ci.lines(report):18 print(line)19print("checks that could not run:", ", ".join(check.name for check in report.checks if check.error))20print("checks that ran:", sum(check.error is None for check in report.checks))21print("exit code:", report.exit_code)
python broken-files-become-findings.pyfiles the pull request changes: governance/users.toml, migrations/0001-add-work-email.sql, migrations/0002-country-as-number.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
policy-tests: failed, the check could not run
enforcement-tests: failed, the check could not run
migrations: failed, findings 1, new 1, on main 0, waived 0, 2 migrations checked
pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
owner-review: passed, findings 0, new 0, on main 0, waived 0, 77 columns checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
properties: passed, findings 4, new 0, on main 4, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
policy-tests/error: the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)
enforcement-tests/error: the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)
migrations/0002-country-as-number.sql: refused: Cannot change column type: country: string -> int
properties/ai.review_corpus: 7 of 7 generated properties missing; on main, owner team-platform
properties/analytics.course_performance: 4 of 4 generated properties missing; on main, owner team-analytics
properties/analytics.learner_activity: 4 of 4 generated properties missing; on main, owner team-analytics
properties/marketing.audience: 7 of 7 generated properties missing; on main, owner team-platform
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
checks that could not run: policy-tests, enforcement-tests
checks that ran: 9
exit code: 1
用户文件使读取它的两项检查都出了错:the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)。Iceberg 拒绝这次类型变更,因为它不属于类型提升,表现为 refused: Cannot change column type: country: string -> int。这两项无法运行的检查不会阻止其他检查(包括迁移检查),所以作者还会看到 not classified; its name suggests personal,作业以退出码 1 结束。如果没有这层捕获,作业会在第一个错误处停止,作者每运行一次只能修复一个问题。
**工作准则:**把崩溃的检查转成一条使作业失败的发现项,附上错误的第一行和它所指的文件,照常运行其他检查,并且绝不让豁免将其静默。
其他失效情形
- 无人要求的关卡。 如果分支保护并不要求某个作业的状态检查,该作业仍会报告其发现,而拉取请求照样合并。应将其设为必需检查,并统计作业失败但仍已合并的拉取请求。
- 构建损坏会掩盖发现。 损坏的流水线没有写出的表,无法作为未描述的表被发现。应先查看
pipelines检查。 - 豁免比其理由存续得更久。 某项发现已被修复,其豁免却仍然保留,随时可能掩盖下一个具有相同 ID 的发现。报告会列出与任何发现都不匹配的豁免,应将其删除。
- 重命名会改变发现 ID。 ID 以列命名,因此被重命名的列,其发现会以新的 ID 重新出现,报告则会把它们原有的豁免列为不匹配任何发现。应在重命名所在的拉取请求中迁移这些豁免。
- 批准比被批准的内容存续得更久。
team-crm批准了city的降级,随后作者又推送了email的降级,这一批准便同时涵盖了两者。应让分支保护启用“当推送的提交影响拉取请求的差异时,撤销过期的拉取请求批准”。 - 一份批准代表整个团队。 任何一位代码所有者的批准都满足 GitHub 的要求,而分支保护的计数是按分支计算的。如果某个领域需要更多批准,应在作业中按团队统计批准数。
- 被阻塞的团队修改关卡。 拉取请求未通过的团队会从
ci.toml中移除该检查、添加豁免,或修改 CODEOWNERS。应让平台团队成为这三者的代码所有者。 - 拉取请求之外的变更。 笔记本创建了一张表,或者手工运行了一次迁移。还应让该作业按计划针对生产环境运行,正如第 2 章运行
problems那样。
本章的决定
- 一项作业在每个拉取请求上运行第 2 章到第 7 章的检查, 即
lib.ci.run,检查项列在governance/ci.toml中;只有第 3 章对值的抽样改为按计划运行。lib.ci.main返回退出码,并以 Markdown 形式写出报告。 - 该作业评判的是拉取请求所做的改动。 由它引入的发现项会使作业失败,除非已获批准或已被豁免;main 上已有的发现项会连同其负责人一起报告,但不会使任何检查失败。崩溃的检查按一条失败的发现项计。
- 每条发现项都有稳定的 ID, 由检查及其对象构成,例如
coverage/crm.learners.work_email。 - 拉取请求与 main 分支的比较在临时平台上进行, 这些平台按照平台实际运行的方式构建,各自带有自己的迁移和管道。
- 模式变更按字段 ID 做差异比较, 并从三个方面报告:Iceberg 读取方、SQL 中点名该列的管道,以及血缘声明该列可到达的每一列,其中包括 INDIRECT 边。
- 我们自己的两条规则: 降低分类、更改负责人或把表移到另一个域,需要 main 分支上该表负责人的批准,并按团队计数;而一个列只有在 main 分支已将其标记为
deprecated之后才能删除。 - 所有权是每张表一份 ODCS 合约, 带有一个域,并由负责团队担任角色
owner,生成到每个域一个的文件夹中,并附一行 CODEOWNERS,由该作业检查。平台团队拥有 CODEOWNERS、作业的设置、其全局规则、豁免和迁移。 - 全局规则和域规则都是 Rego, 通过 regopy 在进程内对每份合约运行,位于访问策略所在文件夹之外。
- 刻意留下的缺口仍保留在 main 上:
marketing.audience和ai.review_corpus会作为覆盖率发现项出现在每份报告中,以平台团队为其负责人,直到它们得到描述为止。 - 噪声按每项检查中无人处理的发现项来衡量:被豁免的,或留在 main 上的。 一项检查的发现大多被忽略,要么是检查有误,要么是平台有误,仅凭数量无法判断是哪一种。一项豁免只针对一条发现项,写明一位负责人、一条理由和一个最多在 90 天之内的截止日期,而覆盖率和强制执行测试从不豁免。
- 拉取请求上的检查按改动的规模确定大小; 读取值的检查按计划运行。这些检查只有在人们一直开着它们时才有用,所以每一项都按改动的规模来确定。
本章,在本版次中
版次 2026.11.31示例于 2026年11月19日 通过本章 10个示例 个示例中有 10 个通过
译自英文版图书,版次 2026.11.31。