可測試的目錄

閱讀約需 76 分鐘 分鐘,12 節 中共有 11 個範例。

版次 2026.11.31 範例於 2026年11 月 19 通過測試 108 個範例中有 108 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.31。 閱讀原文

讀完本章,你就能把目錄的中繼資料當成程式碼來維護。每個資料表都有擁有者與領域,每個欄位都有描述與分級,這些內容寫在一個由團隊審查的檔案裡。這個檔案以穩定的鍵來索引內容,因此重新命名不會讓任何內容遺失;它會被寫入 Iceberg 目錄,並由一項測試檢查,只要有任何內容缺漏,測試就會失敗。你還會讓本機模型草擬描述與分級,並在有人接受草稿之前,衡量這些草稿出錯的頻率。

這個平台的目錄有三個問題。它的資料倉儲是一個 Iceberg 目錄,由一間線上學習平台維護,其管線不斷新增資料表。倉儲裡有八個來源資料表,放在以負責團隊命名的命名空間中,例如 crm.learners 與 support.tickets。管線以這些來源資料表建立報表資料表,並持續建立新的資料表,但沒有任何紀錄說明 marketing.audience(行銷部門可以寄送電子郵件的學習者名單)由誰擁有,也沒有說明 ai.review_corpus(平台的課程助理所讀取的文字)是否含有個人資料。結構變更時被重新命名的欄位,可能在無人察覺的情況下失去它的分級。此外,管線新增的每個資料表,都需要為每個欄位以手工撰寫描述與分級,這項工作慢到讓各團隊乾脆略過。

第 1 章建立了整個受治理平台的粗略版本,本章則讓它的目錄變得穩固。

本章的範例使用本書五個共用模組。lib.catalog.open_catalog 會在暫存資料夾中,為平台的資料表建立一個全新的目錄,它建立在 PyIceberg 以 SQLite 為基礎的 SQL 目錄之上,而 PyIceberg 的文件僅建議將其用於探索與開發(PyIceberg 設定)。lib.pipelines.run 執行平台的其中一條管線,並將輸出寫入目錄。lib.warehouse.connect 在目錄的資料表上開啟 DuckDB,讓查詢可以像平台一樣直接以名稱引用這些資料表。lib.policy 則評估平台的存取政策,這些政策由第 5 章撰寫。最後是 lib.metadata,它在本章逐步建構,負責載入平台的中繼資料、將其寫入目錄,並依據它檢查目錄。

此部分尚未翻譯,因此以英文呈現。

Keep the metadata in a reviewed file

最近一次變動於第 2026.11.27 版

A catalog's metadata is what you know about each table that the data doesn't say: who owns it, which part of the business it belongs to, what it holds, and how each column must be protected. Catalog servers keep this in their own database and edit it in a web page. That works for browsing, but a change made there is invisible to code review, and no test can say whether the catalog is complete. Keeping the metadata in a file in the repository, beside the pipelines that create the tables, makes every change a diff that someone approves, and gives a test something to compare the catalog with.

The platform's file is governance/metadata.toml. For each table it records:

  • an owner, the team answerable for the table, never a person;
  • a domain, the part of the business the table belongs to;
  • a description of the table;
  • for each column, its name, a description and a classification.

A classification is one of four values, ordered from least to most protected:

Classification Meaning On the platform
public May be shown to anyone outside the platform A course's title
internal For the platform's staff, and doesn't identify a person by itself When an enrollment started
personal Identifies a person or tells something about one A learner's email, or text they wrote
sensitive Data whose leak would directly harm someone A card number, an instructor's bank account

The order matters: when a column is built from several others, as a pipeline's output columns often are, the stricter classification wins, and lib.metadata.stricter returns it. Loaded and counted, the file shows what it covers:

metadata-model.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Loads the metadata file and counts what it describes, then prints four of the learners table's columns, looked up byfield ID. The prose's point rests on the `country` and `city` lines, whose descriptions give the reason for each classification."""from collections import Counterfrom lib import metadatadescribed = metadata.load()                      # governance/metadata.tomlcolumns = [column for table in described.values() for column in table.columns.values()]print("tables described:", len(described))print("columns described:", len(columns))counts = Counter(column.classification for column in columns)for level in metadata.LEVELS:    print(f"columns classified {level}:", counts[level])owners = sorted({table.owner for table in described.values()})print("teams that own tables:", len(owners))print("owners:", ", ".join(owners))learners = described["crm.learners"]print("crm.learners owner:", learners.owner)print("crm.learners domain:", learners.domain)for field_id in (1, 3, 5, 6):                    # field IDs, which the file keys columns by, not positions in the table    column = learners.columns[field_id]    print(f"crm.learners field {field_id}:", column.name, "|", column.classification, "|", column.description)
輸出python metadata-model.py
tables described: 10
columns described: 77
columns classified public: 6
columns classified internal: 57
columns classified personal: 12
columns classified sensitive: 2
teams that own tables: 5
owners: team-analytics, team-billing, team-crm, team-learning, team-support
crm.learners owner: team-crm
crm.learners domain: crm
crm.learners field 1: learner_id | internal | The learner's ID, which the platform's other tables use for the learner. Identifies a person through this table's personal columns.
crm.learners field 3: email | personal | The email the learner signs in with.
crm.learners field 5: country | internal | The learner's country, as an ISO 3166 two-letter code.
crm.learners field 6: city | personal | The learner's city. Identifying together with date of birth.

Most columns are internal, because most of a warehouse is IDs, dates and counts: 57 of its 77 columns are. The protected columns are few. Of the file's columns, 12 are personal, and sensitive holds the card number in billing.payments and the payout account in crm.instructors. The file covers the eight source tables and the two reporting tables the platform's analysts rely on, and every table belongs to one of 5 owning teams. The point is that each description carries the reason for its classification. country is internal because a country alone doesn't single anyone out, while city is personal because, as its description says, it identifies a learner together with their date of birth.

The four values and the one-owner rule are ours, not a standard's. The Open Data Contract Standard (ODCS) records a domain, an owner as a free-text role on a team, and a free-text classification (ODCS v3.2.0), and its JSON schema accepts any string as a classification (ODCS JSON schema). The catalog servers differ too. DataHub types its owners (ownership types), and it lets an asset belong to only one domain at a time (domains). It classifies columns with glossary terms (business glossary).

OpenMetadata lets an owner be either a team or several users, a rule enforced at the instance level and on by default (data ownership). Its classifications are a controlled vocabulary of tags (classification), such as PII.Sensitive (PII tags). In Databricks' Unity Catalog every object has an owner, who holds every privilege on it (privileges). Tables and columns carry tags (tags), and a governed tag takes values "only from a predefined set of allowed values" (governed tags). A file like the platform's maps onto each of them. None of these documents describes the check this chapter's test makes: a team, never a person, as the owner of every table, and one of the four classifications on every column.

Working rule: Keep owners, domains, descriptions and classifications in a file in the repository, reviewed like the pipelines. Use a closed list of classifications, ordered by how much protection each needs, and one owning team per table. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it; two sources that both accept edits drift apart.

以欄位 ID 作為欄位中繼資料的鍵

最近一次變動於第 2026.11.27 版

平台的中繼資料檔案以數字而非名稱作為每個欄位的鍵,因為名稱會改變。Iceberg 為每個欄位指派一個欄位 ID(field ID),這是在資料表結構描述內唯一的整數,而規格要求重新命名時必須保留它:「重新命名既有欄位時,必須變更名稱,但不得變更欄位 ID」(Iceberg table spec)。讀取端同樣以該 ID 找出值,因為 Iceberg 是依欄位 ID 選取資料檔案中的欄位,所以重新命名後的欄位,仍能讀取重新命名之前寫入的檔案(Iceberg table spec)。以名稱作為鍵的中繼資料則無所依附:重新命名之後,舊名稱不指向任何欄位,新名稱也沒有對應的項目。

平台的 CRM 團隊希望將 crm.learners.email 改名為 contact_email,以便為工作用電子郵件騰出空間。在暫存目錄(scratch catalog)中模擬這次重新命名,就能看出以名稱作為鍵的副本會遺失什麼。我們分別在以欄位 ID 為鍵的中繼資料檔案中,以及在以名稱為鍵的副本中(欄位的試算表通常就是這種形式)查詢該欄位:

rename-orphans-name-keys.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 通過
"""Renames crm.learners.email in a scratch catalog, then looks the column up by field ID and by name. The prose's pointrests on the two classification lines, which disagree, and on the check's `renamed from email` line."""from lib import catalog, metadata, policyplatform = catalog.open_catalog(["crm.learners"])described = metadata.load()by_id = described["crm.learners"].columns                                    # the book's metadata: keyed by field IDby_name = {column.name: column.classification for column in by_id.values()}  # a spreadsheet's: keyed by namelearners = platform.load_table("crm.learners")with learners.update_schema() as update:    update.rename_column("email", "contact_email")                           # renames in place: the field ID staysfield = learners.schema().find_field("contact_email")print("contact_email field ID:", field.field_id)values = learners.scan(selected_fields=("contact_email",)).to_arrow().column("contact_email")print("values read under the new name:", f"{len(values) - values.null_count:,} of {len(values):,}")print("classification of contact_email, looked up by name:", by_name.get("contact_email"))print("classification of contact_email, looked up by field ID:", by_id[field.field_id].classification)names = {f.name for f in learners.schema().fields}print("name-keyed entries that match no column:", ", ".join(sorted(set(by_name) - names)))for problem in metadata.problems(platform, described):    print("check:", problem)# The policies read classifications by name, generated from the file, which still says email.decision = policy.decide(policy.request("agent-a01", "crm.learners", ["contact_email"]))print("a support agent reading contact_email sees it as:", decision["column_masks"].get("contact_email", "shown"))
輸出python rename-orphans-name-keys.py
contact_email field ID: 3
values read under the new name: 5,000 of 5,000
classification of contact_email, looked up by name: None
classification of contact_email, looked up by field ID: personal
name-keyed entries that match no column: email
check: crm.learners.contact_email: renamed from email since it was reviewed
a support agent reading contact_email sees it as: redact

兩次查詢的結果不一致,而且只有以欄位 ID 為鍵的那一個知道這是同一個欄位。重新命名保留了欄位 ID 3,且 5,000 of 5,000 的值在新名稱下被讀出。以名稱為鍵時,contact_email 沒有分類,而 email 的項目也不對應任何欄位。以欄位 ID 為鍵時,該欄位仍然是 personal。以名稱為鍵的副本所建立的檢查,會回報一個未分類的欄位,卻會漏掉一個已分類的欄位已經消失這件事。

以 ID 為鍵的檔案確實能察覺重新命名,而本章所依據的檢查 lib.metadata.problems 也會回報這件事:contact_email: renamed from email since it was reviewed。資料在重新命名後仍然存在,並不代表重新命名沒有影響:指名 email 的 SQL 會失效,任何只以名稱識別欄位的系統也會失效。平台的存取政策由第 5 章撰寫,它們以名稱接收查詢所讀取的欄位,因此是從 lib.policy 依此檔案產生的、以名稱為鍵的檢視中讀取分類。檢查失敗時,會有人更新檔案中的名稱並重新閱讀說明,產生的檢視隨之更新。在那之前,檢視中沒有 contact_email 的項目,而政策會將找不到的欄位視為 sensitive。支援人員能看到 crm.learners 中其他所有個人欄位,但讀取 contact_email 時看到的是 redact:重新命名會以拒絕存取的方式失敗,而且支援人員會最先發現。

實用原則: 以欄位 ID 作為欄位中繼資料的鍵,並在旁邊附上名稱供人閱讀。名稱不再相符時,應視為需要重新檢視該欄位說明的理由,而不是新欄位。若系統只以名稱識別欄位,請由以 ID 為鍵的檔案產生其以名稱為鍵的檢視,切勿以手動方式維護。

將描述寫入資料表的結構描述

最近一次變動於第 2026.11.27 版

中介資料檔案是審查平台中介資料的地方。不過,讀取資料表的人與工具是查看目錄(catalog),因此部分中介資料必須複製到目錄中。問題在於每一項事實在目錄中應放在哪裡,而 Iceberg 回答了其中一部分:每個欄位都可以帶有一個 doc,這是隨結構描述一起傳遞的選用註解(Iceberg 資料表規格)。資料表也有屬性(properties),是一個字串對應表,但規格指出該對應表「用於控制影響讀取與寫入的設定,並不打算用來存放任意的中介資料」。

因此,我們盡可能遵循規格。欄位描述放入 doc,分類則留在檔案中,供政策與檢查讀取。擁有者與領域放入兩個資料表屬性,作為只讀取資料表的工具的指引,讓手上只有資料表而沒有儲存庫的人,也能找到該詢問的團隊。最後這個做法違背了規格明言的意圖,因此我們只限於這兩個簡短的值。

Iceberg 專案本身的解法是另一條獨立的通道。Iceberg 1.12.0 的 REST 目錄規格新增了 labels,也就是目錄隨資料表回傳的擁有權或分類資訊,依資料表及欄位 ID 分別記錄,刻意不放在屬性中(REST 目錄規格)。這項變更來自 pull request 15750。PyIceberg 0.12.0 的 REST 用戶端並不會讀取這些資訊(原始碼)。

由 lib.metadata.apply 負責複製。對於檔案所描述且目錄中存在的每個資料表,它會把每項描述寫入對應欄位的 doc,並把擁有者與領域寫入屬性,只變更有差異的部分。針對平台八個來源資料表所組成的全新目錄,apply 會顯示它做了哪些變更:

apply-metadata.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Writes the metadata file into a fresh catalog of the eight source tables, then applies it again. The prose's point restson the second `tables changed` line, which is zero: applying a file the catalog already matches writes nothing."""from lib import catalog, metadataplatform = catalog.open_catalog()                # all eight source tables, with no metadata yetdescribed = metadata.load()print("crm.learners schema ID before:", platform.load_table("crm.learners").schema().schema_id)print("tables changed:", len(metadata.apply(platform, described)))learners = platform.load_table("crm.learners")print("crm.learners schema ID after:", learners.schema().schema_id)print("email's doc, in the Iceberg schema:", learners.schema().find_field("email").doc)scanned = learners.scan(selected_fields=("email",), limit=1).to_arrow()   # one row is enough: the schema comes back with it# Arrow keeps an Iceberg column's doc in the field's metadata, under the bytes key b"doc".print("email's doc, in a scan's Arrow schema:", scanned.schema.field("email").metadata[b"doc"].decode())print("crm.learners properties:", ", ".join(f"{k}={v}" for k, v in sorted(learners.properties.items())))print("tables changed by applying again:", len(metadata.apply(platform, described)))
輸出python apply-metadata.py
crm.learners schema ID before: 0
tables changed: 8
crm.learners schema ID after: 1
email's doc, in the Iceberg schema: The email the learner signs in with.
email's doc, in a scan's Arrow schema: The email the learner signs in with.
crm.learners properties: domain=crm, owner=team-crm
tables changed by applying again: 0

套用該檔案會把中介資料寫入目錄,再次套用則不會寫入任何內容,所以 apply 可以在每次合併時執行。第一次執行變更了 8 個資料表。email 的描述現在已存在於 Iceberg 結構描述,以及掃描所回傳的 Arrow 結構描述中,因此透過 PyIceberg 讀取資料表的程式碼,無須知道該檔案存在,就能隨資料取得描述。屬性中包含 owner=team-crm 與 domain=crm。第二次執行變更了 0 個資料表。crm.learners 的結構描述 ID 從 0 變成 1,儘管沒有任何欄位的 ID、名稱或型別改變,這正是下一小節的主題。

此部分尚未翻譯,因此以英文呈現。

A description edit is a schema change

最近一次變動於第 2026.11.27 版

Iceberg's spec gives every schema change a new schema with its own ID, which becomes the table's current schema and joins the list of schemas the table keeps (Iceberg table spec). In PyIceberg 0.12.0, editing nothing but a column's doc counts as such a change. Change nothing but the description of learning.reviews.body, after lib.metadata.apply has written the platform's descriptions, and watch the schema ID:

doc-edit-schema-id.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Changes one column's description after the metadata is applied, then compares schema IDs with field IDs, names andtypes. The prose's point rests on the schema ID changing while the shape stays the same, which the assert checks."""from lib import catalog, metadataplatform = catalog.open_catalog(["learning.reviews"])metadata.apply(platform, metadata.load())    # the platform's descriptions go in first, so the edit below is the only changereviews = platform.load_table("learning.reviews")before = reviews.schema()print("schema ID before the edit:", before.schema_id)with reviews.update_schema() as update:                                        # change one description, nothing else    update.update_column("body", doc="The review's text, as the learner wrote it. May hold names and emails.")after = reviews.schema()assert metadata.shape(before) == metadata.shape(after), "a doc edit changed a field ID, name or type"print("schema ID after the edit:", after.schema_id)print("schemas the table keeps:", len(reviews.metadata.schemas))print("schema ID changed:", before.schema_id != after.schema_id)print("field IDs, names or types changed:", metadata.shape(before) != metadata.shape(after))
輸出python doc-edit-schema-id.py
schema ID before the edit: 1
schema ID after the edit: 2
schemas the table keeps: 3
schema ID changed: True
field IDs, names or types changed: False

A description edit changes the schema ID and nothing a review should compare. The ID went from 1 to 2, and the table keeps 3 schemas: the original, the one apply wrote, and this one. The field IDs, names and types are the same, which lib.metadata.shape compares. A rule that asks for a new classification whenever the schema ID changes would fire on every description edit, and teams learn to ignore a check that fires for nothing.

Working rule: Write column descriptions into Iceberg's doc, so every reader of the table gets them, and keep classifications in the reviewed file. Put no more than an owner and a domain in table properties, and say in your code why. To decide whether a schema change needs a review, compare field IDs, names and types, never schema IDs.

此部分尚未翻譯,因此以英文呈現。

Test the catalog against the metadata

最近一次變動於第 2026.11.27 版

With the metadata applied to the catalog, the remaining question is whether the file covers everything the catalog holds. A test answers it, and it has to start from the catalog: a check that walks the file finds problems only in tables someone already described. lib.metadata.problems lists every table in every namespace of the catalog, and reports a table with no entry or no owner, a column whose field ID has no classification, a classification that isn't one of the four, and a column renamed since it was reviewed.

On the platform, the gaps come from pipelines. Each pipeline is one create table ... as select ... in pipelines/, and running it creates a table. Run all four pipelines on a fresh catalog and check it, and the gaps show:

metadata-gaps.py Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · sqlglot 30.33.0 通過
"""Runs the platform's four pipelines on a fresh catalog, then checks every table in it against the metadata file. Theprose's point rests on the `problems found` line and the two `no metadata` lines that follow it."""from lib import catalog, metadata, pipelinesplatform = catalog.open_catalog()for name in ("course_performance", "learner_activity", "marketing_audience", "review_corpus"):   # each one a file in pipelines/    print("pipeline output:", pipelines.run(platform, name))# The eight source tables plus the four outputs the pipelines just created.tables = [f"{namespace}.{name}" for (namespace,) in platform.list_namespaces() for _, name in platform.list_tables(namespace)]print("tables in the catalog:", len(tables))found = metadata.problems(platform, metadata.load())print("problems found:", len(found))for problem in found:    print("problem:", problem)
輸出python metadata-gaps.py
pipeline output: analytics.course_performance
pipeline output: analytics.learner_activity
pipeline output: marketing.audience
pipeline output: ai.review_corpus
tables in the catalog: 12
problems found: 2
problem: ai.review_corpus: no metadata
problem: marketing.audience: no metadata

The two tables that fail are the ones that matter most. The catalog holds 12 tables, and the check finds 2 problems: ai.review_corpus: no metadata and marketing.audience: no metadata. The two analytics tables pass, because the file describes them, with field IDs that match the order of their select lists.

marketing.audience copies learners' names and emails out of crm.learners, and ai.review_corpus copies the free text of reviews, so both hold personal data, and neither has an owner who would answer for it. Until someone classifies them, the platform's access policy, governance/policies/access.rego, treats every column it can't find a classification for as sensitive, so the gap fails closed rather than open. Chapter 4 proposes their classifications from lineage, from the columns each output column was built from.

The test is then one line, assert not metadata.problems(catalog, metadata.load()), run wherever the catalog changes: in the pull request that adds a pipeline, which chapter 10 sets up, and on a schedule against the production catalog, which catches tables created outside any pull request.

Working rule: Walk the catalog, not the metadata file, so a table nobody described shows up. Fail the test on any table without an owner and any column without one of the four classifications, and never default a missing classification to anything but the most protected value. A pipeline that creates a table ships with that table's metadata, in the same pull request.

在中繼資料過時之前攔截結構描述變更

最近一次變動於第 2026.11.27 版

針對目錄的測試能攔截新增的資料表。資料表也會在其中繼資料之下發生變化,而每一種變更都需要各自的因應方式。更名會保留欄位 ID,因此條目仍然有效,但其描述可能已經不正確。新增的欄位會取得一個無人審閱過的欄位 ID。被刪除的欄位會留下一個不描述任何東西的條目。而先刪除一個欄位、再新增同名欄位,並不會讓舊欄位回來:規格會為每個新增的欄位指派新的 ID(Iceberg table spec),所以重新加入的欄位是另一個不同的欄位。

結構描述變更的頻率高到足以構成影響。在一項針對十個開源應用程式的研究中,「80% 的結構描述變更發生在 20–30% 的資料表」(Qiu, Li and Su, 2013)。該研究涵蓋的是應用程式資料庫,而非資料倉儲。如果資料倉儲的變更也以相同方式集中,那麼最繁忙的資料表,就是中繼資料最先過時的資料表。

在 crm.learners 上暫存的兩項變更,可以展示兩種檢查如何運作:一項新增 preferred_name,有描述但沒有分類;另一項刪除 country 後再把它加回來。problems 與 lib.metadata.dropped:後者會回報其欄位 ID 已不存在於資料表中的條目,並以 Iceberg 時間旅行的方式,也就是指定一個快照,讀取被刪除欄位的值:

schema-changes-checked.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Adds an unreviewed column, drops a column and adds it back, then runs both checks and reads the dropped values by timetravel. The prose's point rests on the two field IDs, and on the time-travel line, which still returns every country."""from pyiceberg.types import StringTypefrom lib import catalog, metadataplatform = catalog.open_catalog(["crm.learners"])described = metadata.load()learners = platform.load_table("crm.learners")written = learners.current_snapshot()                                        # the snapshot the rows were written inprint("country field ID, before it was dropped:", learners.schema().find_field("country").field_id)with learners.update_schema() as update:                                     # a column nobody has reviewed    update.add_column("preferred_name", StringType(), doc="What the learner asks to be called.")with learners.update_schema() as update:                                     # drop a column, then add it back    update.delete_column("country")with learners.update_schema() as update:    update.add_column("country", StringType())                               # an added field gets a new ID, never the old onecountry = learners.schema().find_field("country")print("country field ID, after it was added back:", country.field_id)values = learners.scan(selected_fields=("country",)).to_arrow().column("country")print("country values after it was added back:", f"{values.null_count:,} nulls of {len(values):,}")# Minus the one snapshot that wrote the rows: the schema changes wrote none of their own.print("snapshots the schema changes wrote:", len(learners.metadata.snapshots) - 1)old = learners.scan(snapshot_id=written.snapshot_id, selected_fields=("country",)).to_arrow().column("country")print("country values, read by time travel to that snapshot:", f"{len(old) - old.null_count:,} of {len(old):,}")for problem in metadata.problems(platform, described):    print("problems():", problem)for problem in metadata.dropped(platform, described):    print("dropped():", problem)
輸出python schema-changes-checked.py
country field ID, before it was dropped: 5
country field ID, after it was added back: 17
country values after it was added back: 5,000 nulls of 5,000
snapshots the schema changes wrote: 0
country values, read by time travel to that snapshot: 5,000 of 5,000
problems(): crm.learners.preferred_name: not classified
problems(): crm.learners.country: not classified
dropped(): crm.learners.country: field 5 is described but no longer in the table

重新加入的欄位是新的欄位:country 以欄位 17 回來,而不是 5,而一般掃描是透過目前的結構描述讀取,舊欄位在其中已不存在,因此對該欄位傳回 5,000 nulls of 5,000。兩個新增的欄位都尚未分類,而為 preferred_name 所給的描述並不算數:描述只是結構描述中的說明文字,分類則是檔案中經過審閱的一行內容。

學習者的國家資料並未消失。結構描述變更不會寫入快照(此處為 0),因此資料表目前的快照仍是寫入那些資料列時的快照。快照會記錄其建立時資料表目前結構描述的 ID(Iceberg table spec),而在 PyIceberg 0.12.0 中,指定快照的掃描會使用該結構描述讀取。因此對該快照進行時間旅行,會傳回 5,000 of 5,000 個國家值。刪除欄位並不會刪除其中任何值:平台永久清除資料的地方在第 7 章。dropped 會回報舊欄位那筆已審閱的條目現在不描述任何東西。欄位 ID 一節中的更名是第三種變更,problems 會將它回報為待審閱的欄位。

實用原則: 每次結構描述變更都要執行兩項檢查:以 problems 找出無人分類的欄位或無人重新閱讀的更名欄位,以 dropped 找出不描述任何東西的條目。刪除欄位時,要在同一個 pull request 中移除其條目。刪除個人資料欄位並不會移除其值;在保存這些值的快照過期、其檔案被刪除之前,時間旅行仍能讀取它們。切勿以刪除再重新加入的方式為欄位更名:應就地更名,使其保有欄位 ID 與資料。

為連接資料而分類假名化識別碼

最近一次變動於第 2026.11.31 版

中繼資料檔將部分指向個人的欄位分類為 internal,其中最常見的是 learner_id。它是一個本身沒有意義的數字,而平台的分析師與資料科學家需要用它來彼此連接選課、事件、付款與工單。像 learner_id 這種本身沒有意義的 ID 稱為假名化識別碼(pseudonymous identifier):它不會指名任何人,卻能透過另一張資料表指向某個人。單一 ID 所能觸及的範圍,比它的分類所顯示的更廣。程式碼會計算帶有這個 ID 的資料表數量,將選課紀錄連接到學習者,並向存取政策詢問誰能進行這項連接並看到姓名:

pseudonymous-learner-id.py Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 通過
"""Counts the tables that carry learner_id and how many enrollments one join resolves to an email, then asks the accesspolicy who sees learners' names and also reads another table's learner_id. The prose's point rests on the enrollmentsline and the last line's list of users."""from lib import catalog, metadata, policy, warehousedescribed = metadata.load()carrying = {name: table.column("learner_id") for name, table in described.items() if table.column("learner_id")}   # column() gives None when absentprint("tables with a learner_id column:", len(carrying))print("learner_id classified as:", ", ".join(sorted({column.classification for column in carrying.values()})))personal = [c.name for c in described["crm.learners"].columns.values() if c.classification == "personal"]print("personal columns in crm.learners:", ", ".join(personal))con = warehouse.connect(catalog.open_catalog(["crm.learners", "learning.enrollments"]))found, total = con.execute("""    -- the left join keeps every enrollment; count(l.email) counts only those whose learner has an email    select count(l.email), count(*)    from learning.enrollments e left join crm.learners l on l.learner_id = e.learner_id""").fetchall()[0]                              # fetchall returns a list of rows, and this query has exactly oneprint("enrollments whose learner_id leads to an email:", f"{found:,} of {total:,}")# Who can put a name on another table's learner_id: users who read that table and see crm.learners' names unmasked.rego, seeing, naming = policy.interpreter(), [], []for user in policy.data_document()["platform"]["users"]:    names = policy.decide(policy.request(user, "crm.learners", ["full_name"]), rego)    if names["allow"] and "full_name" not in names["column_masks"]:          # allowed, and not masked: the name shows in clear        seeing.append(user)        tables = [t for t in carrying if t != "crm.learners"                 # the other tables with learner_id this user may read                  and policy.decide(policy.request(user, t, ["learner_id"]), rego)["allow"]]        naming += [f"{user} ({', '.join(tables)})"] if tables else []print("users who see learners' names in crm.learners:", ", ".join(seeing))print("of them, users who read learner_id in another table:", ", ".join(naming))
輸出python pseudonymous-learner-id.py
tables with a learner_id column: 7
learner_id classified as: internal
personal columns in crm.learners: full_name, email, phone, city, street_address, date_of_birth, notes
enrollments whose learner_id leads to an email: 20,000 of 20,000
users who see learners' names in crm.learners: agent-a01, agent-a05, agent-a09, marketing-1
of them, users who read learner_id in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets)

一次連接就能找到人:20,000 of 20,000 筆選課紀錄可通往電子郵件,而隨著電子郵件而來的,還有學習者的姓名、電話號碼、地址與出生日期,這些都屬於該表的 personal 欄位。然而,在帶有 learner_id 的全部 7 張資料表中,它都是 internal。

歐盟的《一般資料保護規則》(GDPR)在前言第 26 點處理了這類資料:「經過假名化處理的個人資料,若可藉由額外資訊歸屬於特定自然人,應視為與可識別的自然人有關的資訊」(GDPR, Recital 26)。這項前言對特定平台意味著什麼,是該平台法務人員要回答的問題,本書不予回答。

本書要決定的是,書中的程式碼如何對待這個 ID。internal 描述的是該欄位在其所在資料表中揭露了什麼,而中繼資料檔中每個 learner_id 的描述則交代其餘部分,指出 crm.learners 是能解析它的資料表,learning.enrollments 的描述即是如此:「The learner who enrolled. Identifies a person when joined with crm.learners.」保護措施設在能解析該 ID 的資料表上。第 5 章決定誰可以讀取 crm.learners 的個人欄位,而本範例的最後幾行則詢問其政策,誰能在那裡看到學習者的姓名:agent-a01, agent-a05, agent-a09, marketing-1。在這些人之中,只有客服人員還會讀取另一張資料表中的 learner_id:agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets)。他們可以為所屬地區的每一張工單對上姓名,這正是他們的職責。行銷人員看得到姓名,但不會讀取其他帶有該 ID 的資料表。

因此,決定誰能進行這項連接的是政策,而不是 ID 的分類。把 support.tickets.learner_id 分類為 personal 對客服不會有任何改變,因為客服是唯一讀取該表的群組,而政策本來就讓客服看到所有個人欄位。第 7 章的資料抹除,則必須涵蓋每一張持有學習者 learner_id 的資料表,無論其分類為何。我們稱這類 ID 為已假名化,絕不稱為已匿名化。

實用原則: 依 ID 所在資料表本身揭露的內容來分類假名化 ID,使分析師所需的連接得以持續運作,並在其描述中說明它透過哪張資料表解析。保護能解析它的資料表,並將該 ID 納入每一次資料抹除。誰能解析這個 ID 是存取上的決定:列出可以不經遮罩地同時讀取該 ID 與解析資料表個人欄位的使用者,並且只把這組權限授予職責在於將資料列連結到個人的人。

以本機模型草擬中繼資料

最近一次變動於第 2026.11.27 版

針對目錄的測試能告訴你哪些欄位還沒有人描述,但它不會替你撰寫描述。為每個欄位撰寫描述並選定分類,是維護目錄時最耗時的工作,而且每當管線新增一張資料表,這項工作就會再出現一次。語言模型可以根據目錄工具能提供給它的資訊來草擬兩者:欄位的名稱、型別,以及少數幾個值。研究同時支持這種協助與它的限制。在一項基準測試中,給予指示並採用兩步驟管線時,ChatGPT 標註資料表欄位語意型別的成績達到「零樣本與單樣本設定下 F1 分數超過 85%」(Korini and Bizer, 2023)。一項針對自動產生欄位描述的研究發現,模型對語意含糊的欄位處理起來相當吃力,這類欄位需要專家以人工提供意見(Wretblad et al., 2024)。因此,模型的草稿只是一份提案,由人在 pull request 中接受、修正或否決。首要工作是衡量草稿的錯誤程度,讓審查者知道該把注意力放在哪裡。

本書的草稿在範例中稱為中繼資料建議,是由 recordings/metadata_suggestions.py 一次錄製下來的,而範例讀取的是這份錄製結果,所以沒有任何範例會呼叫模型。該腳本以固定的修訂版本執行 Qwen3-1.7B。這個模型是開放的,採用 Apache-2.0 授權,因此任何人都能下載相同的權重。它有 17 億個參數,可在 CPU 上執行,不需要 GPU,這使樣本值留在執行它的機器上。它也經過指令微調,並提供一個開關,可以關閉它的「思考」,也就是它在回答之前原本會先寫出的推理過程。腳本採用貪婪解碼,也就是每次取可能性最高的下一個 token,儘管模型卡建議在這種模式下使用取樣:「Temperature=0.7, TopP=0.8, TopK=20, and MinP=0」(Qwen3-1.7B)。錄製結果必須在有人重新錄製時得到相同的內容,而取樣得到的答案只是眾多抽樣中的一次。

對於平台八張來源資料表中的每個欄位,模型會看到資料表名稱、欄位名稱與 Iceberg 型別、以固定種子選出的五個欄位值,以及四種分類,並附上分類表中的各項定義。每項定義附有兩個例子,而這些例子都不是平台上實際存在的欄位。以 sensitive 而言,例子是「政府核發的身分證號碼或健康狀況」。模型被要求以 JSON 格式給出一句話的描述與一個分類。錄製結果的 manifest 會記載模型、其修訂版本、提示詞與設定的雜湊值,以及日期,因此任何人都可以重新錄製並進行比較。

只要該欄位屬於個人資料,這五個值就是個人資料:以 support.tickets.body 而言,模型讀到的客服單包含學員的姓名、街道地址、電話號碼、出生日期與銀行帳號。在你自己的機器上執行的模型會把這些資料留在本機。若將相同的提示詞送往託管模型,就等於把平台的個人資料送給另一家公司,這需要與其他任何傳輸個人資料的情況相同的決策。

將每份草稿的分類與已審查的檔案對照,就能看出兩者是否一致:

suggestion-agreement.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Compares the recorded model's classification of each source column with the reviewed one, and counts how many agree.The prose's point rests on the agreement line and on the two lines counting personal and sensitive columns."""from collections import Counterfrom lib import metadatareviewed = metadata.load()                                   # governance/metadata.toml, reviewed by peoplesuggested = metadata.suggestions()                           # the recorded answers under the book's prompt: no model runs hereassert all(reviewed[s.table].columns[s.field_id].name == s.column for s in suggested)   # each answer names the file's column at its field IDprint("columns the model was asked about:", len(suggested))print("answers that parse as the JSON asked for:", sum(s.description is not None for s in suggested))   # None when the answer isn't that JSONprint("answers with one of the four classifications:", sum(s.classification in metadata.LEVELS for s in suggested))pairs = Counter((reviewed[s.table].columns[s.field_id].classification, s.classification) for s in suggested)agree = sum(n for (was, said), n in pairs.items() if was == said)print("classifications that agree with the review:", f"{agree} of {len(suggested)} ({agree / len(suggested):.0%})")for level in metadata.LEVELS:    said = sorted(((s, n) for (was, s), n in pairs.items() if was == level), key=lambda p: -p[1])    print(f"reviewed {level}, the model said:", ", ".join(f"{s} {n}" for s, n in said))for level in ("personal", "sensitive"):    reviewed_as = sum(n for (was, _), n in pairs.items() if was == level)    print(f"{level} columns the model called {level}:", f"{pairs[level, level]} of {reviewed_as}")
輸出python suggestion-agreement.py
columns the model was asked about: 64
answers that parse as the JSON asked for: 64
answers with one of the four classifications: 64
classifications that agree with the review: 40 of 64 (62%)
reviewed public, the model said: internal 3, public 3
reviewed internal, the model said: internal 37, public 5, sensitive 2
reviewed personal, the model said: internal 9, public 3
reviewed sensitive, the model said: internal 2
personal columns the model called personal: 0 of 12
sensitive columns the model called sensitive: 0 of 2

全部 64 個答案都是有效的 JSON,且都屬於四種分類之一,這顯示格式有效,但無法說明答案是否正確。它們在 40 of 64 (62%) 個欄位上與審查結果一致,而這單一數字掩蓋了錯誤出現的位置。一致的部分大多是被判為 internal 的 internal 欄位。在平台的個人欄位中,模型將 0 of 12 判為 personal;至於 sensitive 欄位,也就是卡號與撥款帳戶,則是 0 of 2:模型把兩者都判為 internal。

提示中的範例可能洩漏答案

最近一次變動於第 2026.11.27 版

提示中的範例也是量測的一部分。腳本還會記錄同一個模型在同樣的欄位上,使用第二個提示時的答案;這個提示除了範例之外,其餘都與第一個相同。該提示只為 personal 與 sensitive 提供範例,而且這些範例就是平台自己的欄位。對於 personal,範例是「姓名、聯絡資料、地址,或是他們所寫的文字」;對於 sensitive,範例是卡號或銀行帳戶。這兩者就是 billing.payments.card_number 與 crm.instructors.payout_iban,也是僅有的敏感欄位。兩種答案並排之後,就能看出範例改變了什麼:

suggestions-with-platform-examples.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Scores the recorded answers under two prompts against the reviewed classifications: the book's prompt, whose examplescome from outside the platform, and one whose examples are the platform's own columns. The prose's point rests onthe sensitive-column lines and the agreement counts."""from lib import metadatareviewed = metadata.load()prompts = {"the book's prompt": metadata.SUGGESTIONS,           "the prompt with the platform's columns as examples": metadata.SUGGESTIONS_WITH_PLATFORM_EXAMPLES}for name, path in prompts.items():    # Each recorded answer, paired with the classification the file reviewed for its column.    suggested = [(s, reviewed[s.table].columns[s.field_id].classification) for s in metadata.suggestions(path)]    agree = sum(s.classification == was for s, was in suggested)    print(f"{name}, classifications that agree with the review:", f"{agree} of {len(suggested)}")    for level in ("personal", "sensitive"):        reviewed_as = [s for s, was in suggested if was == level]        print(f"{name}, {level} columns called {level}:",              f"{sum(s.classification == level for s in reviewed_as)} of {len(reviewed_as)}")    for s, was in suggested:        if was == "sensitive":            print(f"{name}, {s.table}.{s.column}:", s.classification)
輸出python suggestions-with-platform-examples.py
the book's prompt, classifications that agree with the review: 40 of 64
the book's prompt, personal columns called personal: 0 of 12
the book's prompt, sensitive columns called sensitive: 0 of 2
the book's prompt, crm.instructors.payout_iban: internal
the book's prompt, billing.payments.card_number: internal
the prompt with the platform's columns as examples, classifications that agree with the review: 41 of 64
the prompt with the platform's columns as examples, personal columns called personal: 0 of 12
the prompt with the platform's columns as examples, sensitive columns called sensitive: 2 of 2
the prompt with the platform's columns as examples, crm.instructors.payout_iban: sensitive
the prompt with the platform's columns as examples, billing.payments.card_number: sensitive

點出平台自身欄位的提示,能答對敏感欄位;本書的提示則答不對。以平台欄位作為範例時,模型答對了 2 of 2 個敏感欄位。本書的提示答對 0 of 2 個,並把這兩個欄位都判為 internal。

整體一致率幾乎沒有變動,分別是 41 of 64 與 40 of 64,所以單一數字絕對看不出差異。個人欄位並未從範例中獲得好處:在那個提示下,0 of 12 的結果同樣是 personal。

「卡號」幾乎逐字對應 card_number,而「姓名」則要模型自行判斷 full_name 屬於個人資料,結果它沒有判斷出來。範例與資料目錄相符的提示,會讓模型在最需要保護的欄位上顯得比實際更好,因此本書的提示所用的範例取自平台資料之外。

實用原則: 只用你有權送出的輸入,請模型起草中繼資料,而且最好使用在自己機器上執行的模型。記錄每個答案所用的模型、其修訂版本、提示與設定,讓審查者能看出每份草稿是由什麼產生的,並且在任何程式讀取答案之前,先解析並驗證每個答案。提示的範例要取自用來量測它的資料表之外。回報與已審查中繼資料的一致率時,要依每種分類分別回報,不可只給單一數字。

測量模型草稿錯在哪裡

最近一次變動於第 2026.11.27 版

已記錄的草稿是本機模型為平台來源資料表所產生的描述與分類,在大多數欄位上與經過審查的分類一致。就治理而言,錯誤的方向更為重要。分類比應有的保護程度更高,只會讓審查者多花一分鐘,並讓分析師看到一個被遮罩的欄位。分類比應有的保護程度更低,則會讓個人資料被政策視為員工資料,或是任何人都可查看的資料。依方向拆分這些錯誤,就能看出模型往哪一邊失誤:

suggestion-mistakes.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Counts the recorded classifications that protect a column less than the review does, and those that protect it more,then lists each kind of miss. The prose's point rests on the two counts and on the `personal columns the model calledinternal` line."""from lib import metadatareviewed = metadata.load()rank = metadata.LEVELS.index                                   # a column's position in LEVELS: a larger rank is more protectedweaker, stronger = [], []for s in metadata.suggestions():    if s.classification not in metadata.LEVELS:                    # an answer nobody can act on goes to a person        continue    column = reviewed[s.table].columns[s.field_id]    if rank(s.classification) < rank(column.classification):        weaker.append((s, column.classification))    elif rank(s.classification) > rank(column.classification):        stronger.append((s, column.classification))print("columns the model protects less than the review:", len(weaker))print("columns the model protects more than the review:", len(stronger))# Each kind of miss, the most protected review first.for misses in (weaker, stronger):    for was, said in sorted({(was, s.classification) for s, was in misses}, key=lambda p: (-rank(p[0]), -rank(p[1]))):        names = [f"{s.table}.{s.column}" for s, w in misses if (w, s.classification) == (was, said)]        print(f"{was} columns the model called {said}:", len(names))        print(f"which, {was} called {said}:", ", ".join(names))
輸出python suggestion-mistakes.py
columns the model protects less than the review: 19
columns the model protects more than the review: 5
sensitive columns the model called internal: 2
which, sensitive called internal: crm.instructors.payout_iban, billing.payments.card_number
personal columns the model called internal: 9
which, personal called internal: crm.learners.full_name, crm.learners.email, crm.learners.phone, crm.learners.date_of_birth, crm.learners.notes, crm.instructors.full_name, crm.instructors.email, learning.events.ip_address, support.tickets.body
personal columns the model called public: 3
which, personal called public: crm.learners.city, crm.learners.street_address, learning.reviews.body
internal columns the model called public: 5
which, internal called public: learning.events.event_type, learning.events.occurred_at, learning.events.user_agent, learning.reviews.posted_at, support.tickets.opened_at
internal columns the model called sensitive: 2
which, internal called sensitive: crm.learners.marketing_consent, crm.learners.ai_training_consent
public columns the model called internal: 3
which, public called internal: learning.courses.course_id, learning.courses.title, learning.courses.price_usd

模型的錯誤集中在最需要保護的欄位上。它把 19 個欄位的保護程度判得比審查結果低,判得較高的有 5 個。所有個人欄位與敏感欄位,都屬於保護程度偏低的那一類。它把 9 個個人欄位標為 internal,其中包括 crm.learners 與 crm.instructors 兩者的姓名與電子郵件、學員的電話號碼、出生日期與備註、事件的 IP 位址,以及支援工單的文字。它把其中 3 個標為 public:學員的城市與街道地址,以及他們的評論文字。提示詞本身對 personal 的定義是以「識別一個人,或說明關於某人的事」開頭,而姓名或電子郵件兩者皆做到。

其他保護程度偏低的欄位是時間戳記、事件類型與使用者代理字串(user agent)。模型把它們標為 public,而審查選擇的是 internal:這是較小的錯誤,但審查者仍然必須將它找出來。另一個方向的錯誤代價較低。模型把 2 個同意旗標標為 sensitive,並把課程目錄的公開 ID、標題與價格標為 internal。

描述則以另一種方式出錯,以下將模型的四則描述與審查後的描述並列,以呈現這一點:

suggestion-descriptions.py Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 通過
"""Prints the recorded description beside the reviewed one for four columns, where the review knows what a name, a typeand five values can't say. The prose's point rests on the pairs themselves, read one after another."""from lib import metadatareviewed = metadata.load()suggested = {(s.table, s.column): s for s in metadata.suggestions()}   # keyed by name, so the four columns below can be named# Four columns where the platform knows more than a name, a type and five values say.for table, column in [("crm.learners", "ai_training_consent"), ("crm.learners", "support_region"),                      ("crm.learners", "notes"), ("billing.payments", "card_number")]:    print(f"{table}.{column}, the model:", suggested[table, column].description)    print(f"{table}.{column}, the review:", reviewed[table].column(column).description)   # looked up by name: the file keys columns by field ID, but `column()` finds one by name
輸出python suggestion-descriptions.py
crm.learners.ai_training_consent, the model: This column indicates whether a learner has consented to receive AI training.
crm.learners.ai_training_consent, the review: Whether the learner agreed to their text training the platform's AI. Unknown for sign-ups before the question was asked.
crm.learners.support_region, the model: The 'support_region' column indicates the regional support area for learners in the online learning platform.
crm.learners.support_region, the review: The support team that serves the learner: americas, emea or apac.
crm.learners.notes, the model: Stores notes from learners about their move, billing address, or preferences.
crm.learners.notes, the review: Free text from support agents. May hold anything, so it is treated as personal.
billing.payments.card_number, the model: Stores the card numbers of customers for payment processing.
billing.payments.card_number, the review: The full card number, from a legacy export. Should not be stored at all.

每一則描述都通順而可信,但四則中有三則漏掉了某些內容,而這些內容只有了解平台如何填寫該欄位的人才寫得出來。ai_training_consent 記錄的是學員的文字是否可用於訓練平台的 AI,而不是學員是否希望進行訓練,空值表示從未詢問過這個問題。notes 如同其值所暗示的,是由支援人員撰寫關於學員的內容,例如一則關於學員雇主發票的備註,而不是由學員自己撰寫。card_number 是一個根本不該存在的舊有欄位,這正是其描述所能說出的最有用的一點。第四則在實質上是正確的:平台為每位學員指定服務其所在國家的支援團隊的 support_region,而模型把它稱為「區域支援範圍」。通順的描述本身無法顯示它是正確的,還是漏掉了什麼,因此審查者必須閱讀每一則。一種產生欄位描述的方法,其作者在單一提示詞的模型中指出了同樣的失敗:這些模型會寫出虛構或不完整的描述(Cai et al., 2026)。

這些是一個小型模型以貪婪解碼對兩個提示詞所給出的答案。這份記錄無法分辨,模型漏掉個人欄位是因為規模太小,還是因為它起草時只有一個名稱、一種型別與五個值可以依據。更大的模型或更好的提示詞也許能答對更多欄位,要找出答案,方法是以同樣的方式對照經審查的檔案進行測量,然後才能放心讓它處理無人審查過的資料表。

實用原則: 讓模型可以調高欄位的分類,但絕不調低:只有審查者能在合併請求(pull request)中調低分類,並在該欄位的描述中寫明原因。將模型的描述視為初稿,交由了解該欄位的人重寫。在讓模型為無人審查過的資料表起草中繼資料之前,先對照你已審查的中繼資料,依各分類與各錯誤方向進行測量,並且在模型或提示詞每次變更時再測量一次。

正式環境中會出什麼問題

最近一次變動於第 2026.11.27 版

中繼資料過時時不會產生任何錯誤,而下列每一種失敗在有人依賴它之前,看起來都是完整無缺的。

  • 擁有者過時。 團隊會被改名、合併或解散,檔案中的擁有者卻比它們存在得更久。團隊不同於個人,能撐過一次人員離開,但檔案仍需要一項檢查,對照貴組織的團隊清單,確認每位擁有者都是現存的團隊。若以 ODCS 保存擁有權資訊,其團隊成員帶有 dateOut 與 replacedByUsername(ODCS v3.2.0),檢查程式可以讀取這些欄位,找出在 owner 角色上已沒有現任成員的團隊。
  • 描述寫的是去年的結構描述。 欄位的意義改變,名稱與型別卻維持不變:amount 開始以負數記錄退款,或 status 多出一個值。任何結構描述檢查都看不出來。程式碼註解也會以同樣方式失效。針對註解的研究發現,偵測器在變更提交之前就看到變更時,比起只看到變更後的程式碼,能更準確地找出不一致之處(Panthaplackel et al., 2021)。對欄位而言,這樣的比較只適用於由 pull request 呈現變更的情況。管線的程式碼會以 diff 呈現變更,但來源系統可以在沒有任何 diff 的情況下改變它寫出的內容。在有 diff 的地方,應讓修改管線邏輯的 pull request 一併顯示該管線所寫出欄位的描述,使審查者能同時閱讀兩者。
  • 以名稱作為鍵值的中繼資料會在改名時過時。 欄位的試算表、儀表板的資料字典,或依欄位名稱撰寫的政策,每次改名都會遺失一個欄位,如欄位 ID 一節所示,而且不會有任何回報。請由以 ID 為鍵值的檔案產生所有以名稱為鍵值的檢視,並在每次結構描述變更時執行 problems。
  • 兩個中繼資料來源互相覆寫。 目錄伺服器的網頁與儲存庫中的檔案都接受編輯,任一方向的同步都會丟失另一方的內容。DataHub 將在其使用者介面中所做的編輯與擷取而來的中繼資料分開保存,使擷取管線不會覆寫使用者輸入的內容(DataHub dataset entity)。請選定一個來源,並將另一個設為唯讀或由程式產生。
  • 描述的編輯觸發了無人需要的審查。 在 PyIceberg 中,每次編輯文件說明都會讓結構描述取得新的 ID,如上方的文件說明編輯所示,因此以結構描述 ID 為依據的審查會無謂地觸發。請改為比較欄位 ID、名稱與型別。
  • 模型起草的分類被整批核准。 一個加入四十個由模型起草之欄位的 pull request,其描述會被閱讀,分類卻只被略讀,已記錄的草稿顯示了這會導致什麼結果。請將每項起草的分類與描述分開列出,並由保護程度最低者排起,讓審查者看出每份草稿會使哪些內容失去保護。
  • 在管線之外建立的資料表逃過了測試。 筆記本或一次性的回補作業會建立一個沒有人會加入檔案的資料表。請對正式環境的目錄定期執行 problems,而不只是在 pull request 中執行。

本章的決定

最近一次變動於第 2026.11.27 版
  • **中介資料是一份經過審查的檔案:**擁有者、領域、描述與分類都放在 governance/metadata.toml,只能透過 pull request 變更。
  • 四種分類,有順序之分:public、internal、personal 與 sensitive;當一個欄位由多個來源欄位組成時,以較嚴格的分類為準。沒有分類的欄位視為 sensitive。
  • **每張資料表只有一個負責團隊,**不是個人,旁邊並列一個領域。
  • **欄位以 Iceberg 欄位 ID 作為索引鍵,**同時在旁邊保留名稱,使重新命名能在審查時被看見。
  • **描述寫入 Iceberg 的 doc,**由 lib.metadata.apply 負責寫入。擁有者與領域則以指標的形式放入資料表屬性,分類仍留在檔案中。
  • 目錄會依據該檔案進行測試:problems 會走訪目錄中的每一張資料表,dropped 則找出對應到已不存在欄位的項目。結構描述的變更是以欄位 ID、名稱與型別來比較,而不是以結構描述 ID 比較。
  • **learner_id 在每張資料表中都是 internal,**其描述會說明它經由 crm.learners 解析。保護措施設在該資料表上,並由存取政策決定誰可以在看得到姓名的情況下將兩者連接。
  • **apply 與 problems 在每次變更時都會執行:**每次合併時,apply 會把該檔案寫入目錄;problems 則在每個 pull request 中執行,並依排程對正式環境的目錄執行。
  • **起草中介資料的模型在平台自己的機器上執行,**因為只要欄位屬於個人資料,提供給模型看的範例值就是個人資料。
  • **由模型起草的中介資料只是提案。**草稿會連同所用的模型一併記錄,並與已審查的檔案相比較來衡量,再由人員改寫。模型可以提高分類,但不得降低分類。若提示詞提到目錄本身的欄位,會使模型的表現看起來比實際更好,因此提示詞中的範例取自平台資料之外。過時的描述會成為未通過的檢查。

本章,於本版次中

版次 2026.11.31範例於 2026年11 月 19 通過測試本章 11 個範例 個範例中有 11 個通過

譯自英文書,版次 2026.11.31。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 2 章,共 10 章: 可測試的目錄

  1. Keep the metadata in a reviewed file
  2. 以欄位 ID 作為欄位中繼資料的鍵
  3. 將描述寫入資料表的結構描述
  4. A description edit is a schema change
  5. Test the catalog against the metadata
  6. 在中繼資料過時之前攔截結構描述變更
  7. 為連接資料而分類假名化識別碼
  8. 以本機模型草擬中繼資料
  9. 提示中的範例可能洩漏答案
  10. 測量模型草稿錯在哪裡
  11. 正式環境中會出什麼問題
  12. 本章的決定

返回頂端所有章節