可测试的目录

阅读约需 73分钟 分钟,12节中含11个示例。

版次 2026.11.31 示例于 2026年11月19日 通过 108个示例中有108个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.31。 阅读原文

读完本章,你就能把目录的元数据当作代码来维护。每张表都有负责人和所属领域,每一列都有描述和分类,这些内容都写在一个由团队评审的文件里。该文件以稳定的键来索引,因此重命名不会丢失任何信息;它会被写入 Iceberg 目录,并由一项测试检查,只要有任何内容缺失,测试就会失败。你还会让本地模型起草描述和分类,并在有人采纳之前,衡量它的草稿有多常出错。

该平台的目录有三个问题。它的数据仓库是一个 Iceberg 目录,由一家在线学习平台维护,平台的管道不断新增表。仓库中有八张源表,位于以运营团队命名的命名空间里,例如 crm.learners 和 support.tickets。管道基于这些表构建报表,并不断创建新表,而没有任何记录说明谁拥有 marketing.audience(市场部可以发送邮件的学员名单),也没有记录说明 ai.review_corpus(平台的课程助手所读取的文本)是否包含个人数据。模式变更中被重命名的列,可能在无人察觉的情况下丢失它的分类。此外,管道新增的每张表,都需要为每一列手工编写描述和分类,这十分缓慢,以至于各团队会跳过这一步。

第 1 章构建了一个完整的受治理平台的粗略版本。本章则让它的目录变得牢固。

本章的示例使用本书的五个共享模块。lib.catalog.open_catalog 在临时文件夹中为平台的表构建一个全新的目录,基于 PyIceberg 的 SQL 目录和 SQLite,PyIceberg 的文档只建议将其用于探索和开发(PyIceberg 配置)。lib.pipelines.run 运行平台的某条管道,并把输出写入目录。lib.warehouse.connect 在目录的表上打开 DuckDB,使查询可以像平台那样按名称引用这些表;lib.policy 负责评估平台的访问策略,这些策略由第 5 章编写。最后是 lib.metadata,它由本章逐步构建,负责加载平台的元数据,将其写入目录,并对照元数据检查目录。

这部分尚未翻译,因此以英文显示。

Keep the metadata in a reviewed file

最近一次修改见于版次 2026.11.27

A catalog's metadata is what you know about each table that the data doesn't say: who owns it, which part of the business it belongs to, what it holds, and how each column must be protected. Catalog servers keep this in their own database and edit it in a web page. That works for browsing, but a change made there is invisible to code review, and no test can say whether the catalog is complete. Keeping the metadata in a file in the repository, beside the pipelines that create the tables, makes every change a diff that someone approves, and gives a test something to compare the catalog with.

The platform's file is governance/metadata.toml. For each table it records:

  • an owner, the team answerable for the table, never a person;
  • a domain, the part of the business the table belongs to;
  • a description of the table;
  • for each column, its name, a description and a classification.

A classification is one of four values, ordered from least to most protected:

Classification Meaning On the platform
public May be shown to anyone outside the platform A course's title
internal For the platform's staff, and doesn't identify a person by itself When an enrollment started
personal Identifies a person or tells something about one A learner's email, or text they wrote
sensitive Data whose leak would directly harm someone A card number, an instructor's bank account

The order matters: when a column is built from several others, as a pipeline's output columns often are, the stricter classification wins, and lib.metadata.stricter returns it. Loaded and counted, the file shows what it covers:

metadata-model.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Loads the metadata file and counts what it describes, then prints four of the learners table's columns, looked up byfield ID. The prose's point rests on the `country` and `city` lines, whose descriptions give the reason for each classification."""from collections import Counterfrom lib import metadatadescribed = metadata.load()                      # governance/metadata.tomlcolumns = [column for table in described.values() for column in table.columns.values()]print("tables described:", len(described))print("columns described:", len(columns))counts = Counter(column.classification for column in columns)for level in metadata.LEVELS:    print(f"columns classified {level}:", counts[level])owners = sorted({table.owner for table in described.values()})print("teams that own tables:", len(owners))print("owners:", ", ".join(owners))learners = described["crm.learners"]print("crm.learners owner:", learners.owner)print("crm.learners domain:", learners.domain)for field_id in (1, 3, 5, 6):                    # field IDs, which the file keys columns by, not positions in the table    column = learners.columns[field_id]    print(f"crm.learners field {field_id}:", column.name, "|", column.classification, "|", column.description)
输出python metadata-model.py
tables described: 10
columns described: 77
columns classified public: 6
columns classified internal: 57
columns classified personal: 12
columns classified sensitive: 2
teams that own tables: 5
owners: team-analytics, team-billing, team-crm, team-learning, team-support
crm.learners owner: team-crm
crm.learners domain: crm
crm.learners field 1: learner_id | internal | The learner's ID, which the platform's other tables use for the learner. Identifies a person through this table's personal columns.
crm.learners field 3: email | personal | The email the learner signs in with.
crm.learners field 5: country | internal | The learner's country, as an ISO 3166 two-letter code.
crm.learners field 6: city | personal | The learner's city. Identifying together with date of birth.

Most columns are internal, because most of a warehouse is IDs, dates and counts: 57 of its 77 columns are. The protected columns are few. Of the file's columns, 12 are personal, and sensitive holds the card number in billing.payments and the payout account in crm.instructors. The file covers the eight source tables and the two reporting tables the platform's analysts rely on, and every table belongs to one of 5 owning teams. The point is that each description carries the reason for its classification. country is internal because a country alone doesn't single anyone out, while city is personal because, as its description says, it identifies a learner together with their date of birth.

The four values and the one-owner rule are ours, not a standard's. The Open Data Contract Standard (ODCS) records a domain, an owner as a free-text role on a team, and a free-text classification (ODCS v3.2.0), and its JSON schema accepts any string as a classification (ODCS JSON schema). The catalog servers differ too. DataHub types its owners (ownership types), and it lets an asset belong to only one domain at a time (domains). It classifies columns with glossary terms (business glossary).

OpenMetadata lets an owner be either a team or several users, a rule enforced at the instance level and on by default (data ownership). Its classifications are a controlled vocabulary of tags (classification), such as PII.Sensitive (PII tags). In Databricks' Unity Catalog every object has an owner, who holds every privilege on it (privileges). Tables and columns carry tags (tags), and a governed tag takes values "only from a predefined set of allowed values" (governed tags). A file like the platform's maps onto each of them. None of these documents describes the check this chapter's test makes: a team, never a person, as the owner of every table, and one of the four classifications on every column.

Working rule: Keep owners, domains, descriptions and classifications in a file in the repository, reviewed like the pipelines. Use a closed list of classifications, ordered by how much protection each needs, and one owning team per table. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it; two sources that both accept edits drift apart.

以字段 ID 为键保存列元数据

最近一次修改见于版次 2026.11.27

平台的元数据文件以数字而不是名称为键来标识每一列,因为名称会变。Iceberg 为每一列分配一个字段 ID,即在表的模式内唯一的整数,规范要求重命名时保持该 ID 不变:“重命名现有字段必须更改名称,但不得更改字段 ID”(Iceberg 表规范)。读取数据时同样依靠该 ID 查找值,因为 Iceberg 按字段 ID 选取数据文件中的列,所以重命名后的列仍能读取重命名之前写入的文件(Iceberg 表规范)。以名称为键的元数据则无所依附:重命名之后,旧名称指向空,新名称又没有条目。

平台的 CRM 团队希望把 crm.learners.email 改名为 contact_email,为工作邮箱腾出位置。在临时目录(scratch catalog)中演练这次重命名,可以看出以名称为键的副本会丢失什么。下面分别在以字段 ID 为键的元数据文件中,以及在以名称为键的副本(列清单电子表格通常就是这样)中查找该列:

rename-orphans-name-keys.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 通过
"""Renames crm.learners.email in a scratch catalog, then looks the column up by field ID and by name. The prose's pointrests on the two classification lines, which disagree, and on the check's `renamed from email` line."""from lib import catalog, metadata, policyplatform = catalog.open_catalog(["crm.learners"])described = metadata.load()by_id = described["crm.learners"].columns                                    # the book's metadata: keyed by field IDby_name = {column.name: column.classification for column in by_id.values()}  # a spreadsheet's: keyed by namelearners = platform.load_table("crm.learners")with learners.update_schema() as update:    update.rename_column("email", "contact_email")                           # renames in place: the field ID staysfield = learners.schema().find_field("contact_email")print("contact_email field ID:", field.field_id)values = learners.scan(selected_fields=("contact_email",)).to_arrow().column("contact_email")print("values read under the new name:", f"{len(values) - values.null_count:,} of {len(values):,}")print("classification of contact_email, looked up by name:", by_name.get("contact_email"))print("classification of contact_email, looked up by field ID:", by_id[field.field_id].classification)names = {f.name for f in learners.schema().fields}print("name-keyed entries that match no column:", ", ".join(sorted(set(by_name) - names)))for problem in metadata.problems(platform, described):    print("check:", problem)# The policies read classifications by name, generated from the file, which still says email.decision = policy.decide(policy.request("agent-a01", "crm.learners", ["contact_email"]))print("a support agent reading contact_email sees it as:", decision["column_masks"].get("contact_email", "shown"))
输出python rename-orphans-name-keys.py
contact_email field ID: 3
values read under the new name: 5,000 of 5,000
classification of contact_email, looked up by name: None
classification of contact_email, looked up by field ID: personal
name-keyed entries that match no column: email
check: crm.learners.contact_email: renamed from email since it was reviewed
a support agent reading contact_email sees it as: redact

两次查找的结果不一致,只有以字段 ID 为键的查找知道这仍是同一列。重命名保留了字段 ID 3,5,000 of 5,000个值在新名称下顺利读出。以名称为键时,contact_email 没有分类,而 email 的条目不匹配任何列。以字段 ID 为键时,该列仍然是 personal。基于名称键副本构建的检查,会报告一个未分类的列,却发现不了一个已分类的列已经消失。

以 ID 为键的文件能够察觉这次重命名,本章所依托的检查 lib.metadata.problems 会将其报告出来:contact_email: renamed from email since it was reviewed。数据在重命名后得以保留,并不意味着重命名无害:引用 email 的 SQL 会失效,只按名称识别列的系统也会失效。平台的访问策略由第 5 章编写,它们按名称接收查询所读取的列,因此从由 lib.policy 根据该文件生成的、以名称为键的视图中读取分类。检查失败时,有人在文件中更新名称并重新阅读描述,生成的视图随之更新。在此之前,视图中没有 contact_email 的条目,而策略会把找不到的列视为 sensitive。支持人员能看到 crm.learners 中其他所有个人列,读取 contact_email 时看到的是 redact:重命名以关闭访问的方式失败,最先察觉的是支持人员。

工作准则: 以字段 ID 为键保存列元数据,并在旁边附上名称供人阅读。名称不再匹配时,应将其视为需要复查该列描述的理由,而不是一个新列。对于只按名称识别列的系统,应从以 ID 为键的文件生成其以名称为键的视图,绝不手工维护。

将描述写入表的 schema

最近一次修改见于版次 2026.11.27

元数据文件是平台元数据接受审阅的地方。但读取表的人和工具会查看 catalog,因此一部分元数据必须复制到那里。问题在于每项事实在 catalog 中应放在哪里,Iceberg 回答了其中一部分:每一列都可以带有一个 doc,这是一条随 schema 一起传递的可选注释(Iceberg 表规范)。表还有属性(properties),即一个字符串映射,但规范指出,该映射"用于控制影响读写的设置,不应用于任意元数据"。

因此我们尽可能遵循规范。列描述写入 doc,分类则保留在文件中,由策略和检查读取。所有者和所属领域写入两个表属性,作为只读取表的工具的指引,这样任何人即使只有表而没有代码仓库,也能找到应当询问的团队。最后这一选择违背了规范所述的意图,我们只把它限于这两个简短的值。

Iceberg 项目自己的答案是一个独立的通道。Iceberg 1.12.0 的 REST catalog 规范增加了 labels,即 catalog 随表返回的所有权或分类信息,按表并按字段 ID 记录,有意不放入属性中(REST catalog 规范)。这一变更随 pull request 15750 引入。PyIceberg 0.12.0 的 REST 客户端不会读取它们(源码)。

复制工作由 lib.metadata.apply 完成。对于文件中描述且 catalog 中存在的每张表,它把每条描述写入对应列的 doc,并把所有者和所属领域写入属性,只修改有差异的部分。针对平台八张源表组成的全新 catalog,apply 显示了它所做的修改:

apply-metadata.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Writes the metadata file into a fresh catalog of the eight source tables, then applies it again. The prose's point restson the second `tables changed` line, which is zero: applying a file the catalog already matches writes nothing."""from lib import catalog, metadataplatform = catalog.open_catalog()                # all eight source tables, with no metadata yetdescribed = metadata.load()print("crm.learners schema ID before:", platform.load_table("crm.learners").schema().schema_id)print("tables changed:", len(metadata.apply(platform, described)))learners = platform.load_table("crm.learners")print("crm.learners schema ID after:", learners.schema().schema_id)print("email's doc, in the Iceberg schema:", learners.schema().find_field("email").doc)scanned = learners.scan(selected_fields=("email",), limit=1).to_arrow()   # one row is enough: the schema comes back with it# Arrow keeps an Iceberg column's doc in the field's metadata, under the bytes key b"doc".print("email's doc, in a scan's Arrow schema:", scanned.schema.field("email").metadata[b"doc"].decode())print("crm.learners properties:", ", ".join(f"{k}={v}" for k, v in sorted(learners.properties.items())))print("tables changed by applying again:", len(metadata.apply(platform, described)))
输出python apply-metadata.py
crm.learners schema ID before: 0
tables changed: 8
crm.learners schema ID after: 1
email's doc, in the Iceberg schema: The email the learner signs in with.
email's doc, in a scan's Arrow schema: The email the learner signs in with.
crm.learners properties: domain=crm, owner=team-crm
tables changed by applying again: 0

应用该文件会把元数据写入 catalog,再次应用则不会写入任何内容,因此 apply 可以在每次合并时运行。第一次运行修改了 8 张表。email 的描述现在位于 Iceberg schema 中,也位于扫描返回的 Arrow schema 中,所以通过 PyIceberg 读取该表的代码可以随数据一并获得描述,而无需知道该文件的存在。属性中保存了 owner=team-crm 和 domain=crm。第二次运行修改了 0 张表。crm.learners 的 schema ID 从 0 变为 1,尽管没有任何列的 ID、名称或类型发生变化,这正是下一小节的主题。

这部分尚未翻译,因此以英文显示。

A description edit is a schema change

最近一次修改见于版次 2026.11.27

Iceberg's spec gives every schema change a new schema with its own ID, which becomes the table's current schema and joins the list of schemas the table keeps (Iceberg table spec). In PyIceberg 0.12.0, editing nothing but a column's doc counts as such a change. Change nothing but the description of learning.reviews.body, after lib.metadata.apply has written the platform's descriptions, and watch the schema ID:

doc-edit-schema-id.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Changes one column's description after the metadata is applied, then compares schema IDs with field IDs, names andtypes. The prose's point rests on the schema ID changing while the shape stays the same, which the assert checks."""from lib import catalog, metadataplatform = catalog.open_catalog(["learning.reviews"])metadata.apply(platform, metadata.load())    # the platform's descriptions go in first, so the edit below is the only changereviews = platform.load_table("learning.reviews")before = reviews.schema()print("schema ID before the edit:", before.schema_id)with reviews.update_schema() as update:                                        # change one description, nothing else    update.update_column("body", doc="The review's text, as the learner wrote it. May hold names and emails.")after = reviews.schema()assert metadata.shape(before) == metadata.shape(after), "a doc edit changed a field ID, name or type"print("schema ID after the edit:", after.schema_id)print("schemas the table keeps:", len(reviews.metadata.schemas))print("schema ID changed:", before.schema_id != after.schema_id)print("field IDs, names or types changed:", metadata.shape(before) != metadata.shape(after))
输出python doc-edit-schema-id.py
schema ID before the edit: 1
schema ID after the edit: 2
schemas the table keeps: 3
schema ID changed: True
field IDs, names or types changed: False

A description edit changes the schema ID and nothing a review should compare. The ID went from 1 to 2, and the table keeps 3 schemas: the original, the one apply wrote, and this one. The field IDs, names and types are the same, which lib.metadata.shape compares. A rule that asks for a new classification whenever the schema ID changes would fire on every description edit, and teams learn to ignore a check that fires for nothing.

Working rule: Write column descriptions into Iceberg's doc, so every reader of the table gets them, and keep classifications in the reviewed file. Put no more than an owner and a domain in table properties, and say in your code why. To decide whether a schema change needs a review, compare field IDs, names and types, never schema IDs.

这部分尚未翻译,因此以英文显示。

Test the catalog against the metadata

最近一次修改见于版次 2026.11.27

With the metadata applied to the catalog, the remaining question is whether the file covers everything the catalog holds. A test answers it, and it has to start from the catalog: a check that walks the file finds problems only in tables someone already described. lib.metadata.problems lists every table in every namespace of the catalog, and reports a table with no entry or no owner, a column whose field ID has no classification, a classification that isn't one of the four, and a column renamed since it was reviewed.

On the platform, the gaps come from pipelines. Each pipeline is one create table ... as select ... in pipelines/, and running it creates a table. Run all four pipelines on a fresh catalog and check it, and the gaps show:

metadata-gaps.py Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · sqlglot 30.33.0 通过
"""Runs the platform's four pipelines on a fresh catalog, then checks every table in it against the metadata file. Theprose's point rests on the `problems found` line and the two `no metadata` lines that follow it."""from lib import catalog, metadata, pipelinesplatform = catalog.open_catalog()for name in ("course_performance", "learner_activity", "marketing_audience", "review_corpus"):   # each one a file in pipelines/    print("pipeline output:", pipelines.run(platform, name))# The eight source tables plus the four outputs the pipelines just created.tables = [f"{namespace}.{name}" for (namespace,) in platform.list_namespaces() for _, name in platform.list_tables(namespace)]print("tables in the catalog:", len(tables))found = metadata.problems(platform, metadata.load())print("problems found:", len(found))for problem in found:    print("problem:", problem)
输出python metadata-gaps.py
pipeline output: analytics.course_performance
pipeline output: analytics.learner_activity
pipeline output: marketing.audience
pipeline output: ai.review_corpus
tables in the catalog: 12
problems found: 2
problem: ai.review_corpus: no metadata
problem: marketing.audience: no metadata

The two tables that fail are the ones that matter most. The catalog holds 12 tables, and the check finds 2 problems: ai.review_corpus: no metadata and marketing.audience: no metadata. The two analytics tables pass, because the file describes them, with field IDs that match the order of their select lists.

marketing.audience copies learners' names and emails out of crm.learners, and ai.review_corpus copies the free text of reviews, so both hold personal data, and neither has an owner who would answer for it. Until someone classifies them, the platform's access policy, governance/policies/access.rego, treats every column it can't find a classification for as sensitive, so the gap fails closed rather than open. Chapter 4 proposes their classifications from lineage, from the columns each output column was built from.

The test is then one line, assert not metadata.problems(catalog, metadata.load()), run wherever the catalog changes: in the pull request that adds a pipeline, which chapter 10 sets up, and on a schedule against the production catalog, which catches tables created outside any pull request.

Working rule: Walk the catalog, not the metadata file, so a table nobody described shows up. Fail the test on any table without an owner and any column without one of the four classifications, and never default a missing classification to anything but the most protected value. A pipeline that creates a table ships with that table's metadata, in the same pull request.

在元数据过时之前发现模式变更

最近一次修改见于版次 2026.11.27

针对目录的测试能够发现新表。表也会在其元数据之下发生变化,而每一种变化都需要各自的应对方式。重命名会保留字段 ID,因此条目仍然有效,但其描述可能已经不准确。新增的列会获得一个无人审阅过的字段 ID。被删除的列会留下一个什么也不描述的条目。而先删除一列、再添加一个同名的列,并不会让旧列回来:规范为每个新增字段分配新的 ID(Iceberg 表规范),所以重新添加的列是另一列。

模式变更足够频繁,这一点值得重视。在一项针对十个开源应用的研究中,“80% 的模式变更发生在 20% 至 30% 的表中”(Qiu、Li 和 Su,2013)。该研究针对的是应用数据库,而不是数据仓库。如果某个数据仓库的变更也以同样方式集中,那么最繁忙的表就是元数据最先过时的表。

在 crm.learners 上暂存的两项变更展示了这两项检查如何工作:一项添加了 preferred_name,它有描述但没有分类;另一项删除了 country 并再将其添加回来。problems 与 lib.metadata.dropped:第二项检查报告其字段 ID 已不在表中的条目,并按 Iceberg 时间旅行的方式,即指定一个快照,读取被删除列的值:

schema-changes-checked.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Adds an unreviewed column, drops a column and adds it back, then runs both checks and reads the dropped values by timetravel. The prose's point rests on the two field IDs, and on the time-travel line, which still returns every country."""from pyiceberg.types import StringTypefrom lib import catalog, metadataplatform = catalog.open_catalog(["crm.learners"])described = metadata.load()learners = platform.load_table("crm.learners")written = learners.current_snapshot()                                        # the snapshot the rows were written inprint("country field ID, before it was dropped:", learners.schema().find_field("country").field_id)with learners.update_schema() as update:                                     # a column nobody has reviewed    update.add_column("preferred_name", StringType(), doc="What the learner asks to be called.")with learners.update_schema() as update:                                     # drop a column, then add it back    update.delete_column("country")with learners.update_schema() as update:    update.add_column("country", StringType())                               # an added field gets a new ID, never the old onecountry = learners.schema().find_field("country")print("country field ID, after it was added back:", country.field_id)values = learners.scan(selected_fields=("country",)).to_arrow().column("country")print("country values after it was added back:", f"{values.null_count:,} nulls of {len(values):,}")# Minus the one snapshot that wrote the rows: the schema changes wrote none of their own.print("snapshots the schema changes wrote:", len(learners.metadata.snapshots) - 1)old = learners.scan(snapshot_id=written.snapshot_id, selected_fields=("country",)).to_arrow().column("country")print("country values, read by time travel to that snapshot:", f"{len(old) - old.null_count:,} of {len(old):,}")for problem in metadata.problems(platform, described):    print("problems():", problem)for problem in metadata.dropped(platform, described):    print("dropped():", problem)
输出python schema-changes-checked.py
country field ID, before it was dropped: 5
country field ID, after it was added back: 17
country values after it was added back: 5,000 nulls of 5,000
snapshots the schema changes wrote: 0
country values, read by time travel to that snapshot: 5,000 of 5,000
problems(): crm.learners.preferred_name: not classified
problems(): crm.learners.country: not classified
dropped(): crm.learners.country: field 5 is described but no longer in the table

重新添加的列是新列:country 回来时的字段 ID 是 17,而不是 5;普通扫描通过当前模式读取,而旧字段在当前模式中已不存在,因此该列返回 5,000 nulls of 5,000。两个新增的列都未分类,而给 preferred_name 的描述不算数:描述只是模式中的一段文档,而分类是文件中经过审阅的一行。

学习者所在的国家并没有消失。模式变更不会写入快照(此处为 0),所以表的当前快照仍是写入其行时的那个快照。快照会记录创建时表的当前模式 ID(Iceberg 表规范),而在 PyIceberg 0.12.0 中,指定快照的扫描会使用该模式读取。因此,对该快照进行时间旅行会返回 5,000 of 5,000 的国家值。删除列并不会删除其任何值:平台永久擦除数据的内容见第 7 章。dropped 报告旧字段那条经过审阅的条目现在什么也不描述。字段 ID 一节中的重命名是第三种变更,problems 会将其报告为需要审阅的列。

工作准则: 每次模式变更都运行这两项检查:problems 用于发现无人分类的列,或无人重新审阅的重命名列,dropped 用于发现什么也不描述的条目。在删除某列的同一个拉取请求中移除该列的条目。删除个人信息列并不会移除其值,在保存这些值的快照过期、其文件被删除之前,时间旅行仍可读取它们。切勿通过删除再重新添加列来重命名:应原地重命名,这样它会保留其字段 ID 和数据。

为连接而对假名化标识符分级

最近一次修改见于版次 2026.11.31

元数据文件把一些指向个人的列分级为 internal,其中最常见的是 learner_id。它是一个本身没有含义的数字,平台的分析师和数据科学家需要用它把选课记录、事件、支付和工单相互连接起来。像 learner_id 这样本身没有含义的 ID 称为假名化标识符:它不指明任何人,却能通过另一张表指向某个人。一个 ID 所能触及的范围,比它的分级所暗示的更广。这段代码统计携带该 ID 的表,把选课记录连接到学员,并向访问策略询问谁能进行这一连接并看到姓名:

pseudonymous-learner-id.py Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 通过
"""Counts the tables that carry learner_id and how many enrollments one join resolves to an email, then asks the accesspolicy who sees learners' names and also reads another table's learner_id. The prose's point rests on the enrollmentsline and the last line's list of users."""from lib import catalog, metadata, policy, warehousedescribed = metadata.load()carrying = {name: table.column("learner_id") for name, table in described.items() if table.column("learner_id")}   # column() gives None when absentprint("tables with a learner_id column:", len(carrying))print("learner_id classified as:", ", ".join(sorted({column.classification for column in carrying.values()})))personal = [c.name for c in described["crm.learners"].columns.values() if c.classification == "personal"]print("personal columns in crm.learners:", ", ".join(personal))con = warehouse.connect(catalog.open_catalog(["crm.learners", "learning.enrollments"]))found, total = con.execute("""    -- the left join keeps every enrollment; count(l.email) counts only those whose learner has an email    select count(l.email), count(*)    from learning.enrollments e left join crm.learners l on l.learner_id = e.learner_id""").fetchall()[0]                              # fetchall returns a list of rows, and this query has exactly oneprint("enrollments whose learner_id leads to an email:", f"{found:,} of {total:,}")# Who can put a name on another table's learner_id: users who read that table and see crm.learners' names unmasked.rego, seeing, naming = policy.interpreter(), [], []for user in policy.data_document()["platform"]["users"]:    names = policy.decide(policy.request(user, "crm.learners", ["full_name"]), rego)    if names["allow"] and "full_name" not in names["column_masks"]:          # allowed, and not masked: the name shows in clear        seeing.append(user)        tables = [t for t in carrying if t != "crm.learners"                 # the other tables with learner_id this user may read                  and policy.decide(policy.request(user, t, ["learner_id"]), rego)["allow"]]        naming += [f"{user} ({', '.join(tables)})"] if tables else []print("users who see learners' names in crm.learners:", ", ".join(seeing))print("of them, users who read learner_id in another table:", ", ".join(naming))
输出python pseudonymous-learner-id.py
tables with a learner_id column: 7
learner_id classified as: internal
personal columns in crm.learners: full_name, email, phone, city, street_address, date_of_birth, notes
enrollments whose learner_id leads to an email: 20,000 of 20,000
users who see learners' names in crm.learners: agent-a01, agent-a05, agent-a09, marketing-1
of them, users who read learner_id in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets)

一次连接就能触及个人:20,000 of 20,000 条选课记录指向某个电子邮箱,随邮箱而来的还有学员的姓名、电话号码、地址和出生日期,它们都在该表的 personal 列之中。然而在携带 learner_id 的全部 7 张表中,它都是 internal。

欧盟《通用数据保护条例》(GDPR)在第 26 条鉴于条款中论及此类数据:“经过假名化处理的个人数据,若借助额外信息即可归属于某一自然人,则应被视为关于可识别自然人的信息”(GDPR,第 26 条鉴于条款)。这一条款对某个具体平台意味着什么,是该平台的律师要回答的问题,我们不作回答。

我们要决定的,是本书的代码如何对待这个 ID。internal 描述的是该列在其所在表中所透露的内容,其余的则由元数据文件中每个 learner_id 的描述来说明,其中指明 crm.learners 是解析它的表,learning.enrollments 的描述正是如此:“报名的学员。与 crm.learners 连接时可识别个人。”保护措施落在解析该 ID 的那张表上。第 5 章决定谁可以读取 crm.learners 的个人列,示例的最后几行则向其策略询问谁能在那里看到学员姓名,结果是:agent-a01, agent-a05, agent-a09, marketing-1。其中,只有支持坐席还会读取另一张表中的 learner_id:agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets)。他们可以为本区域的每张工单对应上姓名,这正是他们的工作。市场部能看到姓名,但不读取任何其他持有该 ID 的表。

因此,决定谁能进行这一连接的是策略,而不是 ID 的分级。把 support.tickets.learner_id 分级为 personal,对支持团队不会有任何改变,因为他们是读取该表的唯一群体,而策略本来就向支持团队展示每一个个人列。第 7 章中的删除操作必须覆盖每一张持有学员 learner_id 的表,无论其分级如何。我们把这类 ID 称为已假名化的,绝不称为已匿名化的。

工作准则: 按假名化 ID 自身所在表所透露的内容对它分级,使分析师所需的连接继续可用,并在其描述中写明它通过什么来解析。保护解析它的那张表,并在每一次删除中都包含该 ID。谁能解析该 ID 是一项访问决策:列出可以未经掩码地同时读取该 ID 和解析表个人列的用户,只把这一组权限授予工作职责就是把数据行关联到个人的人。

用本地模型起草元数据

最近一次修改见于版次 2026.11.27

针对目录的测试能告诉你哪些列还没有人描述,但它不会撰写描述。为每一列写出描述并选定分类,是维护目录时最耗时的工作,而且流水线每新增一张表,这项工作就会再来一次。语言模型可以根据目录工具能展示的内容来起草这两项:列名、列的类型和少量取值。研究既支持这种帮助,也指出了它的局限。在一项基准测试中,在给定指令并采用两步流水线的条件下,ChatGPT 为表格列标注语义类型,取得了"零样本和单样本设置下超过 85% 的 F1 分数"(Korini and Bizer, 2023)。一项针对生成列描述的研究发现,模型难以处理含义模糊的列,这些列需要专家手工补充信息(Wretblad et al., 2024)。因此,模型的草稿只是一份提议,由人在拉取请求中接受、修改或否决。首要的工作是衡量草稿的错误程度,使评审知道该重点检查哪里。

本书中的草稿,在示例中称为元数据建议,由 recordings/metadata_suggestions.py 一次性录制,示例读取的是这份录制结果,所以没有任何示例会调用模型。该脚本运行的是固定修订版本的 Qwen3-1.7B。该模型是开放的,采用 Apache-2.0 许可证,任何人都可以下载相同的权重。它有 17 亿个参数,可以在 CPU 上运行,无需 GPU,这使样本取值始终留在运行它的机器上。它还经过指令微调,带有一个可以关闭"思考"的开关,所谓"思考",是模型在作答之前另外写出的推理过程。脚本采用贪心解码,即每次取最可能的下一个词元,尽管模型卡片建议在这种模式下采用采样:"Temperature=0.7, TopP=0.8, TopK=20, and MinP=0"(Qwen3-1.7B)。录制结果必须在他人重新录制时保持一致,而采样得到的答案只是众多抽样中的一次。

对于平台八张源表中的每一列,模型看到的内容包括:表名、列名和 Iceberg 类型、用固定种子选出的该列五个取值,以及四种分类及其含义,这些含义取自分类表。每种含义附有两个例子,且这些例子都不是平台已有的列。对于 sensitive,例子是"政府身份证号码或健康状况"。模型被要求以 JSON 格式给出一句话的描述和一个分类。录制的清单记录了模型、其修订版本、提示词与设置的哈希值以及日期,因此任何人都可以重新录制并进行比较。

只要该列属于个人数据,这五个取值就是个人数据:对于 support.tickets.body,模型读取的工单中包含学习者的姓名、街道地址、一个电话号码、一个出生日期和一个银行账号。在你自己的机器上运行模型,这些数据就留在本机。把同样的提示词发送给托管模型,则是把平台的个人数据发送给另一家公司,这需要与其他任何转移个人数据的情形一样做出决定。

与已评审的文件对照,每份草稿的分类是否一致一目了然:

suggestion-agreement.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Compares the recorded model's classification of each source column with the reviewed one, and counts how many agree.The prose's point rests on the agreement line and on the two lines counting personal and sensitive columns."""from collections import Counterfrom lib import metadatareviewed = metadata.load()                                   # governance/metadata.toml, reviewed by peoplesuggested = metadata.suggestions()                           # the recorded answers under the book's prompt: no model runs hereassert all(reviewed[s.table].columns[s.field_id].name == s.column for s in suggested)   # each answer names the file's column at its field IDprint("columns the model was asked about:", len(suggested))print("answers that parse as the JSON asked for:", sum(s.description is not None for s in suggested))   # None when the answer isn't that JSONprint("answers with one of the four classifications:", sum(s.classification in metadata.LEVELS for s in suggested))pairs = Counter((reviewed[s.table].columns[s.field_id].classification, s.classification) for s in suggested)agree = sum(n for (was, said), n in pairs.items() if was == said)print("classifications that agree with the review:", f"{agree} of {len(suggested)} ({agree / len(suggested):.0%})")for level in metadata.LEVELS:    said = sorted(((s, n) for (was, s), n in pairs.items() if was == level), key=lambda p: -p[1])    print(f"reviewed {level}, the model said:", ", ".join(f"{s} {n}" for s, n in said))for level in ("personal", "sensitive"):    reviewed_as = sum(n for (was, _), n in pairs.items() if was == level)    print(f"{level} columns the model called {level}:", f"{pairs[level, level]} of {reviewed_as}")
输出python suggestion-agreement.py
columns the model was asked about: 64
answers that parse as the JSON asked for: 64
answers with one of the four classifications: 64
classifications that agree with the review: 40 of 64 (62%)
reviewed public, the model said: internal 3, public 3
reviewed internal, the model said: internal 37, public 5, sensitive 2
reviewed personal, the model said: internal 9, public 3
reviewed sensitive, the model said: internal 2
personal columns the model called personal: 0 of 12
sensitive columns the model called sensitive: 0 of 2

全部 64 个答案都是有效的 JSON,且分类是四种之一,这说明格式有效,但不能说明答案是否正确。它们在 40 of 64 (62%) 列上与评审一致,而这单一数字掩盖了错误所在。一致的部分大多是被判为 internal 的 internal 列。在平台的个人列中,模型把 0 of 12 判为 personal;而在 sensitive 列中,即卡号和收款账户,0 of 2:它把两者都判为 internal。

提示词中的示例可能泄露答案

最近一次修改见于版次 2026.11.27

提示词中的示例也是测量的一部分。脚本还会记录同一模型在同样的列上使用第二个提示词的结果,这个提示词除示例之外与原提示词完全相同。它只为 personal 和 sensitive 给出示例,而且用的都是平台自己的列。对于 personal,示例是“姓名、联系方式、地址或用户自己写的文字”;对于 sensitive,示例是银行卡号或银行账户。这两类对应的列是 billing.payments.card_number 和 crm.instructors.payout_iban,也是仅有的两个敏感列。把两组答案并排来看,就能看出示例带来的变化:

suggestions-with-platform-examples.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Scores the recorded answers under two prompts against the reviewed classifications: the book's prompt, whose examplescome from outside the platform, and one whose examples are the platform's own columns. The prose's point rests onthe sensitive-column lines and the agreement counts."""from lib import metadatareviewed = metadata.load()prompts = {"the book's prompt": metadata.SUGGESTIONS,           "the prompt with the platform's columns as examples": metadata.SUGGESTIONS_WITH_PLATFORM_EXAMPLES}for name, path in prompts.items():    # Each recorded answer, paired with the classification the file reviewed for its column.    suggested = [(s, reviewed[s.table].columns[s.field_id].classification) for s in metadata.suggestions(path)]    agree = sum(s.classification == was for s, was in suggested)    print(f"{name}, classifications that agree with the review:", f"{agree} of {len(suggested)}")    for level in ("personal", "sensitive"):        reviewed_as = [s for s, was in suggested if was == level]        print(f"{name}, {level} columns called {level}:",              f"{sum(s.classification == level for s in reviewed_as)} of {len(reviewed_as)}")    for s, was in suggested:        if was == "sensitive":            print(f"{name}, {s.table}.{s.column}:", s.classification)
输出python suggestions-with-platform-examples.py
the book's prompt, classifications that agree with the review: 40 of 64
the book's prompt, personal columns called personal: 0 of 12
the book's prompt, sensitive columns called sensitive: 0 of 2
the book's prompt, crm.instructors.payout_iban: internal
the book's prompt, billing.payments.card_number: internal
the prompt with the platform's columns as examples, classifications that agree with the review: 41 of 64
the prompt with the platform's columns as examples, personal columns called personal: 0 of 12
the prompt with the platform's columns as examples, sensitive columns called sensitive: 2 of 2
the prompt with the platform's columns as examples, crm.instructors.payout_iban: sensitive
the prompt with the platform's columns as examples, billing.payments.card_number: sensitive

写明平台自身列的提示词能把敏感列判对,而本书的提示词做不到。以平台的列作为示例时,模型判对了 2 of 2 个敏感列。本书的提示词只判对 0 of 2 个,并把这两列都判为 internal。

整体一致率几乎没有变化,分别是 41 of 64 和 40 of 64,所以只看一个数字永远看不出差别。个人列并未从示例中获益:在该提示词下,0 of 12 同样被判为 personal。

“银行卡号”几乎一字不差地指向 card_number,而“姓名”则要模型自行判断 full_name 是否属于姓名,结果它没有判对。示例与目录相吻合的提示词,会让模型在最需要保护的列上显得比实际更好,因此本书的提示词从平台数据之外选取示例。

工作准则: 只用允许发送给模型的输入来起草元数据,并且最好交给自己机器上的模型。记录每个答案对应的模型、模型版本、提示词和设置,使审阅者能看出每份草稿是由什么产生的;在任何程序读取答案之前,先解析并验证每个答案。提示词的示例应取自用于测量它的表之外。按分类报告与已审阅元数据的一致率,不要只给出一个数字。

衡量模型草稿在哪里出错

最近一次修改见于版次 2026.11.27

已记录的草稿是一个本地模型为平台源表给出的描述和分类。在大多数列上,它们与经过审核的分类一致。对数据治理而言,更重要的是它们往往朝哪个方向出错。分类比应有的级别更严格,代价是审核者多花一分钟,分析师多看到一个被屏蔽的列。分类比应有的级别更宽松,则会使个人数据在策略中被当作员工数据,或被当作任何人都可以查看的数据。按方向拆分这些错误,就能看出模型朝哪个方向失误:

suggestion-mistakes.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Counts the recorded classifications that protect a column less than the review does, and those that protect it more,then lists each kind of miss. The prose's point rests on the two counts and on the `personal columns the model calledinternal` line."""from lib import metadatareviewed = metadata.load()rank = metadata.LEVELS.index                                   # a column's position in LEVELS: a larger rank is more protectedweaker, stronger = [], []for s in metadata.suggestions():    if s.classification not in metadata.LEVELS:                    # an answer nobody can act on goes to a person        continue    column = reviewed[s.table].columns[s.field_id]    if rank(s.classification) < rank(column.classification):        weaker.append((s, column.classification))    elif rank(s.classification) > rank(column.classification):        stronger.append((s, column.classification))print("columns the model protects less than the review:", len(weaker))print("columns the model protects more than the review:", len(stronger))# Each kind of miss, the most protected review first.for misses in (weaker, stronger):    for was, said in sorted({(was, s.classification) for s, was in misses}, key=lambda p: (-rank(p[0]), -rank(p[1]))):        names = [f"{s.table}.{s.column}" for s, w in misses if (w, s.classification) == (was, said)]        print(f"{was} columns the model called {said}:", len(names))        print(f"which, {was} called {said}:", ", ".join(names))
输出python suggestion-mistakes.py
columns the model protects less than the review: 19
columns the model protects more than the review: 5
sensitive columns the model called internal: 2
which, sensitive called internal: crm.instructors.payout_iban, billing.payments.card_number
personal columns the model called internal: 9
which, personal called internal: crm.learners.full_name, crm.learners.email, crm.learners.phone, crm.learners.date_of_birth, crm.learners.notes, crm.instructors.full_name, crm.instructors.email, learning.events.ip_address, support.tickets.body
personal columns the model called public: 3
which, personal called public: crm.learners.city, crm.learners.street_address, learning.reviews.body
internal columns the model called public: 5
which, internal called public: learning.events.event_type, learning.events.occurred_at, learning.events.user_agent, learning.reviews.posted_at, support.tickets.opened_at
internal columns the model called sensitive: 2
which, internal called sensitive: crm.learners.marketing_consent, crm.learners.ai_training_consent
public columns the model called internal: 3
which, public called internal: learning.courses.course_id, learning.courses.title, learning.courses.price_usd

模型的错误集中在最需要保护的列上。它给出的保护级别低于审核结果的列有 19 个,高于审核结果的列有 5 个。所有个人列和敏感列都在保护不足之列。它把 9 个个人列标为 internal,其中包括 crm.learners 和 crm.instructors 中的姓名和电子邮件、学员的电话号码、出生日期和备注、事件的 IP 地址,以及支持工单的文本。它把其中 3 个标为 public:学员所在的城市和街道地址,以及他们的评价文本。提示词自身对 personal 的定义以“识别一个人,或说明关于某人的某些信息”开头,而姓名和电子邮件两者兼具。

其他保护不足的列是时间戳、事件的类型及其用户代理。模型把它们标为 public,而审核结果选择了 internal:这是较小的错误,但审核者仍然需要发现它。反方向的错误代价较小。模型把 2 个同意标志标为 sensitive,并把课程目录的公开 ID、标题和价格标为 internal。

描述以另一种方式出错,下面把模型的四条描述与审核后的描述并列展示:

suggestion-descriptions.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通过
"""Prints the recorded description beside the reviewed one for four columns, where the review knows what a name, a typeand five values can't say. The prose's point rests on the pairs themselves, read one after another."""from lib import metadatareviewed = metadata.load()suggested = {(s.table, s.column): s for s in metadata.suggestions()}   # keyed by name, so the four columns below can be named# Four columns where the platform knows more than a name, a type and five values say.for table, column in [("crm.learners", "ai_training_consent"), ("crm.learners", "support_region"),                      ("crm.learners", "notes"), ("billing.payments", "card_number")]:    print(f"{table}.{column}, the model:", suggested[table, column].description)    print(f"{table}.{column}, the review:", reviewed[table].column(column).description)   # looked up by name: the file keys columns by field ID, but `column()` finds one by name
输出python suggestion-descriptions.py
crm.learners.ai_training_consent, the model: This column indicates whether a learner has consented to receive AI training.
crm.learners.ai_training_consent, the review: Whether the learner agreed to their text training the platform's AI. Unknown for sign-ups before the question was asked.
crm.learners.support_region, the model: The 'support_region' column indicates the regional support area for learners in the online learning platform.
crm.learners.support_region, the review: The support team that serves the learner: americas, emea or apac.
crm.learners.notes, the model: Stores notes from learners about their move, billing address, or preferences.
crm.learners.notes, the review: Free text from support agents. May hold anything, so it is treated as personal.
billing.payments.card_number, the model: Stores the card numbers of customers for payment processing.
billing.payments.card_number, the review: The full card number, from a legacy export. Should not be stored at all.

每条描述都流畅而可信,其中三条遗漏了只有了解平台如何填充该列的人才能写出的内容。ai_training_consent 记录的是学员的文本是否可用于训练平台的 AI,而不是学员是否希望进行训练,空值表示从未询问过这个问题。notes 如其取值所示,是由支持人员撰写的关于学员的记录,其中有一条是关于学员雇主发票的备注,而不是由学员撰写。card_number 是一个根本不该存在的遗留列,这正是其描述能说出的最有用的信息。第四条在实质上是正确的:平台为每位学员指定负责其所在国家的支持团队的 support_region,而模型把它称为“区域支持范围”。流畅的描述本身无法说明它是正确的还是有遗漏,因此审核者要逐条阅读。一种生成列描述方法的作者指出,单提示词模型存在同样的失败,它们会写出虚构或不完整的描述(Cai et al., 2026)。

以上是一个小型模型对两个提示词的回答,采用贪心解码。这份记录无法说明,模型漏掉个人列是因为规模太小,还是因为它起草时只有名称、类型和五个取值可依据。更大的模型或更好的提示词也许能让更多列判断正确,要查明这一点,办法是以同样的方式对照审核过的文件进行衡量,然后才能放心让它处理无人审核过的表。

工作准则: 只允许模型提高列的分类级别,不允许降低:只有审核者可以在拉取请求中降低级别,并在该列的描述中写明理由。把模型的描述当作初稿,交给了解该列的人重写。在让模型为无人审核过的表起草元数据之前,按分类和错误方向,对照你已审核的元数据衡量该模型,并在模型或提示词发生变化时再次衡量。

生产环境中会出什么问题

最近一次修改见于版次 2026.11.27

元数据过时时不会有任何报错,而下面每一种失效在有人依赖它之前,看起来都是完整的。

  • 负责人过时。 团队会被重命名、合并和解散,而文件中的负责人比这些团队存续得更久。团队与个人不同,能经受一次人员离开,但文件仍需要一项检查:对照贵组织的团队列表,核实每个负责人都是现存的团队。如果你在 ODCS 中保存归属信息,其团队成员带有 dateOut 和 replacedByUsername(ODCS v3.2.0),检查可以读取这两项,找出在 owner 角色上没有现任成员的团队。
  • 描述写的是去年的模式。 列的含义发生了变化,而名称和类型保持不变:amount 开始用负数记录退款,或者 status 增加了一个取值。任何模式检查都发现不了这一点。代码注释也以同样的方式失效。针对注释的研究发现,检测器在看到提交之前的变更时,比只看到变更后的代码时,能更准确地发现不一致(Panthaplackel et al., 2021)。对于列,这种比较只在拉取请求展示了变更的地方成立。流水线的代码变更会出现在差异中,但源系统可以在没有任何差异的情况下改变它写出的内容。在有差异的地方,应让修改流水线逻辑的拉取请求同时展示它所写入各列的描述,使评审者两者都能读到。
  • 以名称为键的元数据在重命名时过时。 列的电子表格、仪表盘的数据字典,或者针对列名编写的策略,每次重命名都会丢失一列,正如字段 ID 一节所示,而且不会有任何报告。应从以 ID 为键的文件生成所有以名称为键的视图,并在每次模式变更时运行 problems。
  • 两个元数据来源互相覆盖。 目录服务器的网页和仓库中的文件都接受编辑,任一方向的同步都会丢掉另一方的修改。DataHub 将在其界面中所做的编辑与摄取的元数据分开保存,使摄取流水线不会覆盖用户录入的内容(DataHub dataset entity)。应选定一个来源,并将另一个设为只读或自动生成。
  • 描述的编辑触发了无人需要的评审。 在 PyIceberg 中,每次文档编辑都会给模式一个新的 ID,正如上文的文档编辑所示,因此以模式 ID 为键的评审会无故触发。应改为比较字段 ID、名称和类型。
  • 起草的分类被批量批准。 一个添加了四十个由模型起草的列的拉取请求,其描述会被阅读,其分类却只被略读,记录下来的草稿显示了这会导致什么结果。应将每条起草的分类与描述分开列出,并将保护程度最低的排在最前,使评审者看到每条草稿会让什么处于无保护状态。
  • 在流水线之外创建的表逃过了测试。 笔记本或一次性的回填会创建一张没有人会添加进文件的表。应按计划针对生产目录运行 problems,而不只是在拉取请求中运行。

本章的决定

最近一次修改见于版次 2026.11.27
  • **元数据是一份经过评审的文件:**负责人、领域、描述和分类都保存在 governance/metadata.toml 中,只通过拉取请求修改。
  • 四个有序的分类:public、internal、personal 和 sensitive,当一列由多列构建而成时,取较严格的一个。没有分类的列按 sensitive 处理。
  • 每张表只有一个负责团队, 不是个人,旁边另有一个领域。
  • 列以 Iceberg 字段 ID 为键, 同时保留列名,这样重命名会显现出来供评审。
  • 描述写入 Iceberg 的 doc, 由 lib.metadata.apply 写入。负责人和领域作为指针写入表属性,分类则保留在文件中。
  • 目录依据该文件进行测试:problems 遍历目录中的每张表,dropped 找出对应已不存在的列的条目。比较模式变更时,依据字段 ID、名称和类型,而不是模式 ID。
  • 在每张表中,learner_id 都是 internal, 其描述说明它通过 crm.learners 解析。保护措施设在那张表上,由访问策略决定谁可以在能看到姓名的情况下连接这两张表。
  • apply 和 problems 在每次变更时运行:apply 在每次合并时把该文件写入目录,problems 在每个拉取请求中运行,并按计划针对生产目录运行。
  • 起草元数据的模型运行在平台自己的机器上, 因为只要该列是个人数据,展示给模型的样本值就是个人数据。
  • 模型起草的元数据只是一份提议。 草稿连同所用模型一起记录,与经过评审的文件对照衡量,并由人重写。模型可以提高分类,但不能降低。提示词中若出现目录自身的列名,就会让模型的表现显得比实际更好,所以提示词中的示例取自平台数据之外。过时的描述会变成一次失败的检查。

本章,在本版次中

版次 2026.11.31示例于 2026年11月19日 通过本章 11个示例 个示例中有 11 个通过

译自英文版图书,版次 2026.11.31。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 2 章,共 10 章: 可测试的目录

  1. Keep the metadata in a reviewed file
  2. 以字段 ID 为键保存列元数据
  3. 将描述写入表的 schema
  4. A description edit is a schema change
  5. Test the catalog against the metadata
  6. 在元数据过时之前发现模式变更
  7. 为连接而对假名化标识符分级
  8. 用本地模型起草元数据
  9. 提示词中的示例可能泄露答案
  10. 衡量模型草稿在哪里出错
  11. 生产环境中会出什么问题
  12. 本章的决定

返回顶部全部章节