A catalog you can test

By the end of this chapter you can keep a catalog's metadata as code. Every table gets an owner and a domain, and every column a description and a classification, in a file your team reviews. The file is keyed so a rename can't lose anything, written into the Iceberg catalog, and checked by a test that fails when anything is missing. You'll also have a local model draft descriptions and classifications, and measure how often its drafts are wrong before anyone accepts them.

The platform, the online learning platform this book follows, sells professional courses to adult learners and keeps its warehouse in an Iceberg catalog. Eight source tables sit under the namespaces of the teams that run them, such as crm.learners and support.tickets, and the platform's pipelines build reporting tables from them. Chapter 1 built a rough version of the whole governed platform. This chapter makes its catalog solid, and the catalog has three problems. The pipelines keep creating tables, and nothing says who owns marketing.audience, the list of learners marketing may email, or whether ai.review_corpus, the text the platform's course assistant reads, holds personal data. A column renamed in a schema change can lose its classification without anyone noticing. And every table a pipeline adds needs a description and a classification for each of its columns, written by hand, which is slow enough that teams skip it.

The chapter's examples use five of the book's shared modules. lib.catalog.open_catalog builds a fresh catalog of the platform's tables in a temporary folder, with PyIceberg's SQL catalog on SQLite, which PyIceberg's docs recommend "for exploratory or development purposes" only (PyIceberg configuration). lib.pipelines.run runs one of the platform's pipelines and writes its output to the catalog. lib.warehouse.connect opens DuckDB on the catalog's tables, so a query can name them as the platform does, and lib.policy evaluates the platform's access policies, which chapter 5 writes. And lib.metadata, which this chapter builds up, loads the platform's metadata, writes it into the catalog and checks the catalog against it.

Keep the metadata in a reviewed file

A catalog's metadata is what you know about each table that the data doesn't say: who owns it, which part of the business it belongs to, what it holds, and how each column must be protected. Catalog servers keep this in their own database and edit it in a web page. That works for browsing, but a change made there is invisible to code review, and no test can say whether the catalog is complete. Keeping the metadata in a file in the repository, beside the pipelines that create the tables, makes every change a diff that someone approves, and gives a test something to compare the catalog with.

The platform's file is governance/metadata.toml. For each table it records:

• an owner, the team answerable for the table, never a person;
• a domain, the part of the business the table belongs to;
• a description of the table;
• for each column, its name, a description and a classification.

A classification is one of four values, ordered from least to most protected:

The order matters: when a column is built from several others, as a pipeline's output columns often are, the stricter classification wins, and lib.metadata.stricter returns it. Load the file and count what it says:

1from collections import Counter23from lib import metadata45described = metadata.load()                      # governance/metadata.toml6columns = [column for table in described.values() for column in table.columns.values()]7print("tables described:", len(described))8print("columns described:", len(columns))9counts = Counter(column.classification for column in columns)10for level in metadata.LEVELS:11    print(f"columns classified {level}:", counts[level])12owners = sorted({table.owner for table in described.values()})13print("teams that own tables:", len(owners))14print("owners:", ", ".join(owners))1516learners = described["crm.learners"]17print("crm.learners owner:", learners.owner)18print("crm.learners domain:", learners.domain)19for field_id in (1, 3, 5, 6):20    column = learners.columns[field_id]21    print(f"crm.learners field {field_id}:", column.name, "|", column.classification, "|", column.description)

tables described: 10
columns described: 77
columns classified public: 6
columns classified internal: 57
columns classified personal: 12
columns classified sensitive: 2
teams that own tables: 5
owners: team-analytics, team-billing, team-crm, team-learning, team-support
crm.learners owner: team-crm
crm.learners domain: crm
crm.learners field 1: learner_id | internal | The learner's ID, which the platform's other tables use for the learner. Identifies a person through this table's personal columns.
crm.learners field 3: email | personal | The email the learner signs in with.
crm.learners field 5: country | internal | The learner's country, as an ISO 3166 two-letter code.
crm.learners field 6: city | personal | The learner's city. Identifying together with date of birth.

The file describes 10 tables and 77 columns: the eight source tables and the two reporting tables the platform's analysts rely on. Most columns are internal, 57 of them, because most of a warehouse is IDs, dates and counts. The file classifies 12 as personal and 2 as sensitive, the card number in billing.payments and the payout account in crm.instructors, and every table belongs to one of 5 owning teams. Each column's description says what a reviewer needs in order to check the classification: country is internal because a country alone doesn't single anyone out, while city is personal because, as its description says, it identifies a learner together with their date of birth.

The four values and the one-owner rule are ours, not a standard's. The Open Data Contract Standard (ODCS) records a domain, ownership as a free-text role on a team, and a free-text classification (ODCS v3.2.0), which its JSON schema accepts as any string (ODCS JSON schema). The catalog servers differ too. DataHub types its owners (ownership types), allows an asset "only one Domain at a time" (domains), and classifies columns with glossary terms (business glossary). In OpenMetadata, "either a team or multiple user can be the owner", a rule that "is enforced at the instance level" and is on by default (data ownership), and a classification is a controlled vocabulary of tags (classification), such as PII.Sensitive (PII tags). In Databricks' Unity Catalog every object has an owner, who holds every privilege on it (privileges). Tables and columns carry tags (tags), and a governed tag takes values "only from a predefined set of allowed values" (governed tags). A file like the platform's maps onto each of them. What none of these documents describes is the check this chapter's test makes: a team, never a person, as the owner of every table, and one of the four classifications on every column.

Working rule: Keep owners, domains, descriptions and classifications in a file in the repository, reviewed like the pipelines. Use a closed list of classifications, ordered by how much protection each needs, and one owning team per table. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it; two sources that both accept edits drift apart.

Key column metadata by field ID

The platform's metadata file keys each column by a number, not by its name, because names change. Iceberg gives every column a field ID: "Each field in the tuple is named and has an integer id that is unique in the table schema", and "Renaming an existing field must change the name, but not the field ID" (Iceberg table spec). Readers find a column's values by that ID too, since "Columns in Iceberg data files are selected by field id", so a renamed column still reads the files written before the rename. Metadata keyed by name has nothing to hold on to: after a rename, the old name points at nothing and the new name has no entry.

The platform's CRM team wants crm.learners.email called contact_email, to make room for a work email. The example renames it in a scratch catalog, then looks the column up in two ways: in the platform's metadata file, keyed by field ID, and in a copy keyed by name, the way a spreadsheet of columns usually is:

1from lib import catalog, metadata, policy23platform = catalog.open_catalog(["crm.learners"])4described = metadata.load()5by_id = described["crm.learners"].columns                                    # the book's metadata: keyed by field ID6by_name = {column.name: column.classification for column in by_id.values()}  # a spreadsheet's: keyed by name78learners = platform.load_table("crm.learners")9with learners.update_schema() as update:10    update.rename_column("email", "contact_email")11field = learners.schema().find_field("contact_email")12print("contact_email field ID:", field.field_id)13values = learners.scan(selected_fields=("contact_email",)).to_arrow().column("contact_email")14print("values read under the new name:", f"{len(values) - values.null_count:,} of {len(values):,}")1516print("classification of contact_email, looked up by name:", by_name.get("contact_email"))17print("classification of contact_email, looked up by field ID:", by_id[field.field_id].classification)18names = {f.name for f in learners.schema().fields}19print("name-keyed entries that match no column:", ", ".join(sorted(set(by_name) - names)))20for problem in metadata.problems(platform, described):21    print("check:", problem)22# The policies read classifications by name, generated from the file, which still says email.23decision = policy.decide(policy.request("agent-a01", "crm.learners", ["contact_email"]))24print("a support agent reading contact_email sees it as:", decision["column_masks"].get("contact_email", "shown"))

contact_email field ID: 3
values read under the new name: 5,000 of 5,000
classification of contact_email, looked up by name: None
classification of contact_email, looked up by field ID: personal
name-keyed entries that match no column: email
check: crm.learners.contact_email: renamed from email since it was reviewed
a support agent reading contact_email sees it as: redact

The rename kept field ID 3, and every value came through under the new name: 5,000 of 5,000. The two lookups disagree. Keyed by name, contact_email has no classification, and the entry for email matches no column. Keyed by field ID, the column is still personal. A check built on the name-keyed copy would report an unclassified column and miss that a classified one had gone; the ID-keyed file knows it is the same column.

The ID-keyed file does notice the rename, and lib.metadata.problems, the check this chapter builds on, reports it: contact_email: renamed from email since it was reviewed. The data surviving a rename doesn't make the rename harmless. SQL that names email breaks, and so does any system that knows columns only by name. Access policies are one: the platform's policies, which chapter 5 writes, receive the columns a query reads by name, so they read classifications from a name-keyed view that lib.policy generates from this file. When the check fails, someone updates the name in the file, reads the description again, and the generated view follows. Until then, the view has no entry for contact_email, and the policies treat a column they can't find as sensitive. A support agent, who sees every other personal column of crm.learners, gets contact_email as redact: the rename fails closed, and support notices first.

Working rule: Key column metadata by field ID, with the name beside it for people. Treat a name that no longer matches as a reason to review the column's description, not as a new column. Where a system knows columns only by name, generate its name-keyed view from the ID-keyed file, never maintain it by hand.

Write descriptions into the table's schema

The metadata file is where the platform's metadata is reviewed. People and tools that read a table, though, look in the catalog, so some of the metadata has to be copied there. The question is where in the catalog each fact belongs, and Iceberg answers part of it. Each column in an Iceberg schema "may have an optional comment or doc string", its doc (Iceberg table spec), which travels with the schema. A table also has properties, a map of strings, but the spec says that map "is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata."

So we follow the spec as far as we can. Column descriptions go into doc. Classifications stay in the file, where the policies and checks read them. The owner and the domain go into two table properties, as a pointer for tools that read only the table, so anyone with a table and no repository can find the team to ask. That last choice goes against the spec's stated intent, and we keep it to those two short values. The Iceberg project's own answer is a separate channel: the REST catalog spec of Iceberg 1.12.0 adds labels, ownership or classification that a catalog returns with a table, per table and per field ID, kept out of properties on purpose (REST catalog spec, pull request 15750). PyIceberg 0.12.0's REST client doesn't read them (source).

lib.metadata.apply does the copying: for every table the file describes and the catalog holds, it writes each description into its column's doc, and the owner and domain into the properties, changing only what differs. Apply it to a fresh catalog of the platform's eight source tables:

1from lib import catalog, metadata23platform = catalog.open_catalog()4described = metadata.load()5print("crm.learners schema ID before:", platform.load_table("crm.learners").schema().schema_id)6print("tables changed:", len(metadata.apply(platform, described)))78learners = platform.load_table("crm.learners")9print("crm.learners schema ID after:", learners.schema().schema_id)10print("email's doc, in the Iceberg schema:", learners.schema().find_field("email").doc)11scanned = learners.scan(selected_fields=("email",), limit=1).to_arrow()12print("email's doc, in a scan's Arrow schema:", scanned.schema.field("email").metadata[b"doc"].decode())13print("crm.learners properties:", ", ".join(f"{k}={v}" for k, v in sorted(learners.properties.items())))14print("tables changed by applying again:", len(metadata.apply(platform, described)))

crm.learners schema ID before: 0
tables changed: 8
crm.learners schema ID after: 1
email's doc, in the Iceberg schema: The email the learner signs in with.
email's doc, in a scan's Arrow schema: The email the learner signs in with.
crm.learners properties: domain=crm, owner=team-crm
tables changed by applying again: 0

All 8 tables changed. The description of email is now in the Iceberg schema and in the Arrow schema that a scan returns, as field metadata under doc, so code that reads the table through PyIceberg gets the description with the data, without knowing the file exists. The properties hold owner=team-crm and domain=crm, and applying a second time changes 0 tables, so apply can run on every merge without writing anything when nothing changed. The schema ID of crm.learners went from 0 to 1, although no column's ID, name or type changed, which is the subject of the next subsection.

A description edit is a schema change

Iceberg's spec says that every schema change produces "a new schema that is identified by a unique schema ID, is added to the table's list of schemas, and is set as the table's current schema" (Iceberg table spec). In PyIceberg 0.12.0, editing nothing but a column's doc counts as such a change. Edit the description of learning.reviews.body, after lib.metadata.apply has written the platform's descriptions into the schema:

1from lib import catalog, metadata23platform = catalog.open_catalog(["learning.reviews"])4metadata.apply(platform, metadata.load())5reviews = platform.load_table("learning.reviews")6before = reviews.schema()7print("schema ID before the edit:", before.schema_id)89with reviews.update_schema() as update:                                        # change one description, nothing else10    update.update_column("body", doc="The review's text, as the learner wrote it. May hold names and emails.")11after = reviews.schema()12print("schema ID after the edit:", after.schema_id)13print("schemas the table keeps:", len(reviews.metadata.schemas))14print("schema ID changed:", before.schema_id != after.schema_id)15print("field IDs, names or types changed:", metadata.shape(before) != metadata.shape(after))

schema ID before the edit: 1
schema ID after the edit: 2
schemas the table keeps: 3
schema ID changed: True
field IDs, names or types changed: False

The schema ID went from 1 to 2, and the table keeps 3 schemas: the original, the one apply wrote, and this one. Yet the field IDs, names and types are the same, which lib.metadata.shape compares. A rule such as "classify a table again whenever its schema ID changes" would fire on every description edit, and teams learn to ignore a check that fires for nothing.

Working rule: Write column descriptions into Iceberg's doc, so every reader of the table gets them, and keep classifications in the reviewed file. Put no more than an owner and a domain in table properties, and say in your code why. To decide whether a schema change needs a review, compare field IDs, names and types, never schema IDs.

Test the catalog against the metadata

With the metadata applied to the catalog, the remaining question is whether the file covers everything the catalog holds. A test answers it, and it has to start from the catalog: a check that walks the file finds problems only in tables someone already described. lib.metadata.problems lists every table in every namespace of the catalog, and reports a table with no entry or no owner, a column whose field ID has no classification, a classification that isn't one of the four, and a column renamed since it was reviewed.

On the platform, the gaps come from pipelines. Each pipeline is one create table ... as select ... in pipelines/, and running it creates a table. Run all four of them on a fresh catalog, then check:

1from lib import catalog, metadata, pipelines23platform = catalog.open_catalog()4for name in ("course_performance", "learner_activity", "marketing_audience", "review_corpus"):5    print("pipeline output:", pipelines.run(platform, name))67tables = [f"{namespace}.{name}" for (namespace,) in platform.list_namespaces() for _, name in platform.list_tables(namespace)]8print("tables in the catalog:", len(tables))9found = metadata.problems(platform, metadata.load())10print("problems found:", len(found))11for problem in found:12    print("problem:", problem)

pipeline output: analytics.course_performance
pipeline output: analytics.learner_activity
pipeline output: marketing.audience
pipeline output: ai.review_corpus
tables in the catalog: 12
problems found: 2
problem: ai.review_corpus: no metadata
problem: marketing.audience: no metadata

The catalog holds 12 tables, and the check finds 2 problems: ai.review_corpus: no metadata and marketing.audience: no metadata. The two analytics tables pass, because the file describes them, with field IDs that match the order of their select lists. The two that fail are the ones that matter most. marketing.audience copies learners' names and emails out of crm.learners, and ai.review_corpus copies the free text of reviews, so both hold personal data, and neither has an owner who would answer for it. Until someone classifies them, the platform's access policy, governance/policies/access.rego, treats every column it can't find a classification for as sensitive, so the gap fails closed rather than open. Chapter 4 proposes their classifications from lineage, from the columns each output column was built from.

The test is then one line, assert not metadata.problems(catalog, metadata.load()), run wherever the catalog changes: in the pull request that adds a pipeline, which chapter 10 sets up, and on a schedule against the production catalog, which catches tables created outside any pull request.

Working rule: Walk the catalog, not the metadata file, so a table nobody described shows up. Fail the test on any table without an owner and any column without one of the four classifications, and never default a missing classification to anything but the most protected value. A pipeline that creates a table ships with that table's metadata, in the same pull request.

Catch schema changes before the metadata goes stale

The test against the catalog catches new tables. Tables also change underneath their metadata, through schema evolution, and each kind of change needs its own response. A rename keeps the field ID, so the entry survives, but its description may now be wrong. An added column gets a field ID nobody has reviewed. A dropped column leaves an entry that describes nothing. And dropping a column and adding one with the same name doesn't bring the old column back: the spec says that "adding a new field assigns a new ID for that field" (Iceberg table spec), so it's a different column. Schemas change often enough for this to matter: in a study of ten open-source applications, "80% of the schema changes happened in 20-30% of the tables" (Qiu, Li and Su, 2013). The study is of application databases, not warehouses. If a warehouse's changes concentrate the same way, its busiest tables are the ones whose metadata goes stale first.

Stage two of those changes on crm.learners in a scratch catalog: add a column, preferred_name, with a description but no classification, then drop country and add it back. Run problems, and lib.metadata.dropped, which reports entries whose field ID the table no longer has. Then read the dropped column's values the way Iceberg's time travel does, by naming a snapshot:

1from pyiceberg.types import StringType23from lib import catalog, metadata45platform = catalog.open_catalog(["crm.learners"])6described = metadata.load()7learners = platform.load_table("crm.learners")8written = learners.current_snapshot()                                        # the snapshot the rows were written in9print("country field ID, before it was dropped:", learners.schema().find_field("country").field_id)1011with learners.update_schema() as update:                                     # a column nobody has reviewed12    update.add_column("preferred_name", StringType(), doc="What the learner asks to be called.")13with learners.update_schema() as update:                                     # drop a column, then add it back14    update.delete_column("country")15with learners.update_schema() as update:16    update.add_column("country", StringType())1718country = learners.schema().find_field("country")19print("country field ID, after it was added back:", country.field_id)20values = learners.scan(selected_fields=("country",)).to_arrow().column("country")21print("country values after it was added back:", f"{values.null_count:,} nulls of {len(values):,}")22print("snapshots the schema changes wrote:", len(learners.metadata.snapshots) - 1)23old = learners.scan(snapshot_id=written.snapshot_id, selected_fields=("country",)).to_arrow().column("country")24print("country values, read by time travel to that snapshot:", f"{len(old) - old.null_count:,} of {len(old):,}")25for problem in metadata.problems(platform, described):26    print("problems():", problem)27for problem in metadata.dropped(platform, described):28    print("dropped():", problem)

country field ID, before it was dropped: 5
country field ID, after it was added back: 17
country values after it was added back: 5,000 nulls of 5,000
snapshots the schema changes wrote: 0
country values, read by time travel to that snapshot: 5,000 of 5,000
problems(): crm.learners.preferred_name: not classified
problems(): crm.learners.country: not classified
dropped(): crm.learners.country: field 5 is described but no longer in the table

Both added columns are unclassified, and the check says so. The description given to preferred_name doesn't count: a description is a doc in the schema, and a classification is a reviewed line in the file. country came back as field 17, with 5,000 nulls of 5,000, because a plain scan reads with the current schema, where field 5 no longer exists. The learners' countries haven't gone anywhere. A schema change writes no snapshot (0 here), so the table's snapshot is still the one its rows were written in, and a snapshot records the "ID of the table's current schema when the snapshot was created" (Iceberg table spec). In PyIceberg 0.12.0, a scan that names a snapshot reads with that schema, and it returns 5,000 of 5,000 countries. Every snapshot stays readable this way, each with its own schema, until it expires and the files only it referenced are deleted. Dropping a column deletes nothing: chapter 7 is where the platform erases data for good. To the check, the re-added country is a column like any other new one, and dropped reports that the reviewed entry for the old field now describes nothing. The rename from the field ID section is the third kind of change, and problems reports it the same way, as a column to review.

Working rule: Run both checks on every schema change: problems for columns nobody has classified or renamed ones nobody has re-read, and dropped for entries that describe nothing. Remove a dropped column's entry in the same pull request that drops it. Dropping a personal column doesn't remove its values, which time travel reads until the snapshots that hold them expire and their files are deleted. Never drop and re-add a column to rename it: rename it in place, so it keeps its field ID and its data.

Classify pseudonymous IDs for joining

The metadata file classifies some columns that point at people as internal, and the most common of them is learner_id. It's a number with no meaning of its own, and the platform's analysts and data scientists need it to join enrollments, events, payments and tickets to each other. An ID like this is a pseudonymous identifier: it doesn't name anyone, but it leads to someone through another table. Count the tables that carry it and how many enrollments one join turns into an email, then ask the platform's access policy who can make that join and see names:

1from lib import catalog, metadata, policy, warehouse23described = metadata.load()4carrying = {name: table.column("learner_id") for name, table in described.items() if table.column("learner_id")}5print("tables with a learner_id column:", len(carrying))6print("learner_id classified as:", ", ".join(sorted({column.classification for column in carrying.values()})))7personal = [c.name for c in described["crm.learners"].columns.values() if c.classification == "personal"]8print("personal columns in crm.learners:", ", ".join(personal))910con = warehouse.connect(catalog.open_catalog(["crm.learners", "learning.enrollments"]))11found, total = con.execute("""12    select count(l.email), count(*)13    from learning.enrollments e left join crm.learners l on l.learner_id = e.learner_id14""").fetchall()[0]15print("enrollments whose learner_id leads to an email:", f"{found:,} of {total:,}")1617# Who can put a name on another table's learner_id: users who read that table and see crm.learners' names unmasked.18rego, seeing, naming = policy.interpreter(), [], []19for user in policy.data_document()["platform"]["users"]:20    names = policy.decide(policy.request(user, "crm.learners", ["full_name"]), rego)21    if names["allow"] and "full_name" not in names["column_masks"]:22        seeing.append(user)23        tables = [t for t in carrying if t != "crm.learners"24                  and policy.decide(policy.request(user, t, ["learner_id"]), rego)["allow"]]25        naming += [f"{user} ({', '.join(tables)})"] if tables else []26print("users who see learners' names in crm.learners:", ", ".join(seeing))27print("of them, users who read learner_id in another table:", ", ".join(naming))

tables with a learner_id column: 7
learner_id classified as: internal
personal columns in crm.learners: full_name, email, phone, city, street_address, date_of_birth, notes
enrollments whose learner_id leads to an email: 20,000 of 20,000
users who see learners' names in crm.learners: agent-a01, agent-a05, agent-a09, marketing-1
of them, users who read learner_id in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets)

learner_id appears in 7 tables, classified internal in every one. One join to crm.learners gives every enrollment an email, 20,000 of 20,000, and with it the learner's name, phone number, address and date of birth, which are among the table's personal columns. The EU's General Data Protection Regulation (GDPR) addresses data like this in Recital 26: "Personal data which have undergone pseudonymisation, which could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person" (GDPR, Recital 26). What the recital means for a given platform is a question for its lawyers, and we don't answer it.

What we do decide is how the book's code treats the ID. internal describes what the column reveals in the table that holds it, and each learner_id's description in the metadata file says the rest, as learning.enrollments' does: "The learner who enrolled. Identifies a person when joined with crm.learners." The protection sits on the table that resolves the ID: chapter 5 decides who may read the personal columns of crm.learners, and the example's last lines ask its policy who sees learners' names there: agent-a01, agent-a05, agent-a09, marketing-1. Of them, only the support agents also read a learner_id in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets). They can put a name on every ticket of their region, and that's their job. Marketing sees names but reads no other table that holds the ID.

So the policy, not the ID's classification, decides who can make the join. Classifying support.tickets.learner_id as personal would change nothing for support, the only group that reads the table, because the policy already shows support every personal column. Erasure, in chapter 7, has to reach every table that holds a learner's learner_id, whatever its classification. We call such IDs pseudonymized, never anonymized.

Working rule: Classify a pseudonymous ID by what its own table reveals, so the joins your analysts need keep working, and say in its description what it resolves through. Protect the table that resolves it, and include the ID in every erasure. Who can resolve the ID is an access decision: list the users who may read both the ID and the resolving table's personal columns unmasked, and grant that pair only to those whose job is linking rows to people.

Draft metadata with a local model

The test against the catalog tells you which columns nobody has described, but it doesn't write the descriptions. Writing a description and choosing a classification for every column is the slowest part of keeping a catalog, and the work comes back with every table a pipeline adds. A language model can draft both from what a catalog tool can show it: a column's name, its type and a few of its values. Research supports both the help and its limits. On one benchmark, given instructions and a two-step pipeline, ChatGPT labelled the semantic types of table columns with "F1 scores of over 85% in zero- and one-shot setups" (Korini and Bizer, 2023), while a study of generated column descriptions found that "models unsurprisingly struggle with columns that exhibit inherent ambiguity, highlighting the need for manual expert input" (Wretblad et al., 2024). So a model's draft is a proposal that a person accepts, corrects or rejects in the pull request, and the first job is to measure how wrong the drafts are, so the review knows where to look.

The book's drafts, which the examples call the metadata suggestions, were recorded once by recordings/metadata_suggestions.py, and the examples read that recording; none of them calls a model. The script runs Qwen3-1.7B at a pinned revision. The model is open, under the Apache-2.0 licence, so anyone can download the same weights. At 1.7 billion parameters it runs on a CPU, with no GPU, which keeps the sample values on the machine that runs it. And it's instruction-tuned, with a switch that turns off its "thinking", the reasoning it otherwise writes before it answers. The script decodes greedily, always taking the most likely next token, although the model's card suggests sampling in this mode: "Temperature=0.7, TopP=0.8, TopK=20, and MinP=0" (Qwen3-1.7B). A recording has to come out the same when someone records it again, and a sampled answer is one draw among many.

For each column of the platform's eight source tables, the model saw the table's name, the column's name and Iceberg type, five of the column's values chosen with a fixed seed, and the four classifications with the meanings in the table of classifications. Each meaning comes with two examples, and none of them is a column the platform has: for sensitive, "a government ID number or a health condition". The model was asked for a one-sentence description and one classification, as JSON. The recording's manifest names the model, its revision, a hash of the prompts and settings, and the date, so anyone can record again and compare.

Those five values are personal data whenever the column is: for support.tickets.body, the tickets the model read held learners' names, street addresses, a phone number, a date of birth and a bank account number. A model running on your own machine keeps them there. Sending the same prompt to a hosted model sends the platform's personal data to another company, which needs the same decision as any other transfer of it.

Compare the model's classifications with the reviewed ones in the metadata file:

1from collections import Counter23from lib import metadata45reviewed = metadata.load()                                   # governance/metadata.toml, reviewed by people6suggested = metadata.suggestions()                           # data/recorded/metadata-suggestions7assert all(reviewed[s.table].columns[s.field_id].name == s.column for s in suggested)8print("columns the model was asked about:", len(suggested))9print("answers that parse as the JSON asked for:", sum(s.description is not None for s in suggested))10print("answers with one of the four classifications:", sum(s.classification in metadata.LEVELS for s in suggested))1112pairs = Counter((reviewed[s.table].columns[s.field_id].classification, s.classification) for s in suggested)13agree = sum(n for (was, said), n in pairs.items() if was == said)14print("classifications that agree with the review:", f"{agree} of {len(suggested)} ({agree / len(suggested):.0%})")15for level in metadata.LEVELS:16    said = sorted(((s, n) for (was, s), n in pairs.items() if was == level), key=lambda p: -p[1])17    print(f"reviewed {level}, the model said:", ", ".join(f"{s} {n}" for s, n in said))18for level in ("personal", "sensitive"):19    reviewed_as = sum(n for (was, _), n in pairs.items() if was == level)20    print(f"{level} columns the model called {level}:", f"{pairs[level, level]} of {reviewed_as}")

columns the model was asked about: 64
answers that parse as the JSON asked for: 64
answers with one of the four classifications: 64
classifications that agree with the review: 40 of 64 (62%)
reviewed public, the model said: internal 3, public 3
reviewed internal, the model said: internal 37, public 5, sensitive 2
reviewed personal, the model said: internal 9, public 3
reviewed sensitive, the model said: internal 2
personal columns the model called personal: 0 of 12
sensitive columns the model called sensitive: 0 of 2

All 64 answers are valid JSON with one of the four classifications, which shows that the prompt's format worked and says nothing about whether the answers are right. They agree with the review on 40 of 64 (62%) columns, and that single number hides where the misses are. Most of the agreement is internal columns called internal. Of the platform's personal columns, the model called 0 of 12 personal, and of its sensitive ones, the card number and the payout account, 0 of 2: it called both internal.

Examples in a prompt can give the answer away

A prompt's examples are part of the measurement. The script also records the same model on the same columns under a second prompt, identical but for its examples. It gives examples for personal and sensitive only, and they're the platform's own columns: personal is "a name, contact details, an address or text they wrote", and sensitive is "a card number or a bank account". Those two are billing.payments.card_number and crm.instructors.payout_iban, the platform's only sensitive columns. Compare the two prompts' answers:

1from lib import metadata23reviewed = metadata.load()4prompts = {"the book's prompt": metadata.SUGGESTIONS,5           "the prompt with the platform's columns as examples": metadata.SUGGESTIONS_WITH_PLATFORM_EXAMPLES}6for name, path in prompts.items():7    suggested = [(s, reviewed[s.table].columns[s.field_id].classification) for s in metadata.suggestions(path)]8    agree = sum(s.classification == was for s, was in suggested)9    print(f"{name}, classifications that agree with the review:", f"{agree} of {len(suggested)}")10    for level in ("personal", "sensitive"):11        reviewed_as = [s for s, was in suggested if was == level]12        print(f"{name}, {level} columns called {level}:",13              f"{sum(s.classification == level for s in reviewed_as)} of {len(reviewed_as)}")14    for s, was in suggested:15        if was == "sensitive":16            print(f"{name}, {s.table}.{s.column}:", s.classification)

the book's prompt, classifications that agree with the review: 40 of 64
the book's prompt, personal columns called personal: 0 of 12
the book's prompt, sensitive columns called sensitive: 0 of 2
the book's prompt, crm.instructors.payout_iban: internal
the book's prompt, billing.payments.card_number: internal
the prompt with the platform's columns as examples, classifications that agree with the review: 41 of 64
the prompt with the platform's columns as examples, personal columns called personal: 0 of 12
the prompt with the platform's columns as examples, sensitive columns called sensitive: 2 of 2
the prompt with the platform's columns as examples, crm.instructors.payout_iban: sensitive
the prompt with the platform's columns as examples, billing.payments.card_number: sensitive

Named in the prompt, the sensitive columns come out right: 2 of 2. With examples from elsewhere, the model finds 0 of 2 and calls both internal. Overall agreement hardly moves, 41 of 64 against 40 of 64, so a report of one number would never show the difference. The examples for personal didn't help: 0 of 12 personal columns came out personal under that prompt too. "A card number" names card_number almost word for word, while "a name" leaves the model to decide that full_name is one, and it didn't. A prompt whose examples match the catalog makes a model look better than it is, on the columns that need protection most, so the book's prompt takes its examples from outside the platform's data.

Working rule: Draft metadata with a model only from inputs you're allowed to send it, preferably to a model on your own machines. Record each answer with the model, its revision, the prompt and the settings, so a reviewer can see what produced each draft, and parse and validate every answer before anything reads it. Take the prompt's examples from outside the tables you measure it on. Report agreement with the reviewed metadata per classification, never as one number.

Measure where the model's drafts are wrong

The recorded drafts, a local model's descriptions and classifications for the platform's source tables, agree with the reviewed classifications on most columns. For governance, what matters more is which way they miss. A classification more protected than it should be costs a reviewer a minute and an analyst a masked column. One less protected than it should be leaves personal data where the policies treat it as staff data, or as data anyone may see. Count the misses in each direction:

1from lib import metadata23reviewed = metadata.load()4rank = metadata.LEVELS.index5weaker, stronger = [], []6for s in metadata.suggestions():7    if s.classification not in metadata.LEVELS:                    # an answer nobody can act on goes to a person8        continue9    column = reviewed[s.table].columns[s.field_id]10    if rank(s.classification) < rank(column.classification):11        weaker.append((s, column.classification))12    elif rank(s.classification) > rank(column.classification):13        stronger.append((s, column.classification))14print("columns the model protects less than the review:", len(weaker))15print("columns the model protects more than the review:", len(stronger))1617# Each kind of miss, the most protected review first.18for misses in (weaker, stronger):19    for was, said in sorted({(was, s.classification) for s, was in misses}, key=lambda p: (-rank(p[0]), -rank(p[1]))):20        names = [f"{s.table}.{s.column}" for s, w in misses if (w, s.classification) == (was, said)]21        print(f"{was} columns the model called {said}:", len(names))22        print(f"which, {was} called {said}:", ", ".join(names))

columns the model protects less than the review: 19
columns the model protects more than the review: 5
sensitive columns the model called internal: 2
which, sensitive called internal: crm.instructors.payout_iban, billing.payments.card_number
personal columns the model called internal: 9
which, personal called internal: crm.learners.full_name, crm.learners.email, crm.learners.phone, crm.learners.date_of_birth, crm.learners.notes, crm.instructors.full_name, crm.instructors.email, learning.events.ip_address, support.tickets.body
personal columns the model called public: 3
which, personal called public: crm.learners.city, crm.learners.street_address, learning.reviews.body
internal columns the model called public: 5
which, internal called public: learning.events.event_type, learning.events.occurred_at, learning.events.user_agent, learning.reviews.posted_at, support.tickets.opened_at
internal columns the model called sensitive: 2
which, internal called sensitive: crm.learners.marketing_consent, crm.learners.ai_training_consent
public columns the model called internal: 3
which, public called internal: learning.courses.course_id, learning.courses.title, learning.courses.price_usd

The model protected 19 columns less than the review did, and 5 more. Every personal and sensitive column is among the less protected. The model called 9 personal columns internal, including both tables' names and emails, the learners' phone numbers, dates of birth and notes, the IP addresses of events and the text of support tickets. It called 3 of them public: learners' cities and street addresses, and the text of their reviews. The prompt's own meaning for personal begins "identifies a person or tells something about one", and a name or an email does both. The other 5 less protected columns are three timestamps, an event's type and its user agent, which the model called public where the review chose internal: a smaller mistake, but one a reviewer still has to catch. The misses the other way cost less. The model called the 2 consent flags sensitive, and 3 public columns of the course catalog internal.

Descriptions go wrong differently. Put four of the model's descriptions beside the reviewed ones:

1from lib import metadata23reviewed = metadata.load()4suggested = {(s.table, s.column): s for s in metadata.suggestions()}56# Four columns where the platform knows more than a name, a type and five values say.7for table, column in [("crm.learners", "ai_training_consent"), ("crm.learners", "support_region"),8                      ("crm.learners", "notes"), ("billing.payments", "card_number")]:9    print(f"{table}.{column}, the model:", suggested[table, column].description)10    print(f"{table}.{column}, the review:", reviewed[table].column(column).description)

crm.learners.ai_training_consent, the model: This column indicates whether a learner has consented to receive AI training.
crm.learners.ai_training_consent, the review: Whether the learner agreed to their text training the platform's AI. Unknown for sign-ups before the question was asked.
crm.learners.support_region, the model: The 'support_region' column indicates the regional support area for learners in the online learning platform.
crm.learners.support_region, the review: The support team that serves the learner: americas, emea or apac.
crm.learners.notes, the model: Stores notes from learners about their move, billing address, or preferences.
crm.learners.notes, the review: Free text from support agents. May hold anything, so it is treated as personal.
billing.payments.card_number, the model: Stores the card numbers of customers for payment processing.
billing.payments.card_number, the review: The full card number, from a legacy export. Should not be stored at all.

Every description is fluent and plausible, and three of the four miss something that only someone who knows how the platform fills the column could write. ai_training_consent records whether the learner's text may train the platform's AI, not whether the learner wants training, and a null in it means the question was never asked. notes are written by support agents about learners, as the values hint ("Asked about invoices for their employer."), not by learners. And card_number is a legacy column that shouldn't exist at all, which is the most useful thing its description can say. The fourth is right in substance: the platform gives each learner the support_region of the support team that serves their country, and the model calls it their "regional support area". Nothing in a fluent description says which kind it is, so a reviewer reads every one. The authors of one method for generating column descriptions name the same failure in single-prompt models: "hallucinated or incomplete descriptions" (Cai et al., 2026).

These are one small model's answers, decoded greedily, to two prompts. The recording can't tell whether the 0 of 12 for personal columns comes from the model's size or from drafting with only a name, a type and five values to go on. A larger model or a better prompt may get more columns right, and the way to find out is to measure it the same way, against the reviewed file, before trusting it with tables nobody has reviewed.

Working rule: Let a model raise a column's classification, never lower it: only a reviewer lowers one, in the pull request, with the reason in the column's description. Treat the model's descriptions as first drafts for someone who knows the column to rewrite. Before a model drafts metadata for tables nobody has reviewed, measure it against your reviewed metadata, per classification and per direction of error, and measure again whenever the model or the prompt changes.

What goes wrong in production

A catalog's metadata fails quietly: nothing errors when it goes stale, and every failure below looks like a complete catalog until someone relies on it.

• Owners go stale. Teams are renamed, merged and dissolved, and the owner in the file outlives them. A team, unlike a person, survives one departure, but the file still needs a check that every owner is a team that exists, run against your organization's list of teams. If you keep ownership in ODCS, its team members carry a dateOut and a replacedByUsername (ODCS v3.2.0), which a check can read to find a team with no current member in the owner role.
• Descriptions describe last year's schema. A column's meaning changes while its name and type stay the same: amount starts holding refunds as negative numbers, or status gains a value. No schema check sees it. Code comments fail the same way, and research on them found that a detector catches inconsistencies more accurately when it sees the code change, "before they are committed", than when it sees only the code that results (Panthaplackel et al., 2021). The comparison holds for columns only where the change is in a pull request: a pipeline's code changes in a diff, but a source system can change what it writes with no diff anywhere. Where there is a diff, make the pull request that changes a pipeline's logic show the descriptions of the columns it writes, so the reviewer reads both.
• Metadata keyed by name goes stale on a rename. A spreadsheet of columns, a dashboard's data dictionary, or a policy written against column names loses a column on every rename, as the field ID section shows, and reports nothing. Generate every name-keyed view from the ID-keyed file, and run problems on every schema change.
• Two sources of metadata overwrite each other. A catalog server's web page and a file in a repository both accept edits, and a sync in either direction loses the other's. DataHub keeps edits made in its UI apart from ingested metadata "to avoid accidental overwrites of user-provided data by ingestion pipelines" (DataHub dataset entity). Choose one source, and make the other read-only or generated.
• Description edits trigger reviews nobody needs. In PyIceberg every doc edit gives the schema a new ID, as the doc edit above shows, so a review keyed on schema IDs fires for nothing, and people learn to ignore it. Compare field IDs, names and types instead.
• Drafted classifications are approved in bulk. A pull request that adds forty model-drafted columns gets its descriptions read and its classifications skimmed, and the recorded drafts show where that leads. List each drafted classification separately from the descriptions, least protected first, so the reviewer sees what each draft would leave unprotected.
• A table created outside the pipelines escapes the test. A notebook or a one-off backfill creates a table nobody will add to the file. Run problems on a schedule against the production catalog, not only in pull requests.

What this chapter decided

• Metadata is a reviewed file: owners, domains, descriptions and classifications live in governance/metadata.toml, changed only by pull request.
• Four classifications, ordered: public, internal, personal and sensitive, with the stricter one winning when a column is built from several. A column without one is treated as sensitive.
• One owning team per table, never a person, and a domain beside it.
• Columns are keyed by Iceberg field ID, with the name kept beside it so a rename shows up for review.
• Descriptions go into Iceberg's doc, written by lib.metadata.apply. The owner and domain go into table properties as a pointer, and classifications stay in the file.
• The catalog is tested against the file: problems walks every table in the catalog, and dropped finds entries for columns that are gone. Schema changes are compared by field IDs, names and types, not by schema IDs.
• learner_id is internal in every table, with a description that says it resolves through crm.learners. The protection sits on that table, and the access policy decides who may join the two with names in view.
• apply and problems run on every change: apply writes the file into the catalog on every merge, and problems runs in every pull request and on a schedule against the production catalog.
• The model that drafts metadata runs on the platform's own machines, because the sample values it's shown are personal data whenever the column is.
• Model-drafted metadata is a proposal. Descriptions and classifications drafted by a local model are recorded, measured against the reviewed file and rewritten by a person. A model may raise a classification, never lower it: the recorded model called 0 of 12 personal columns personal, and found the sensitive ones only when the prompt named them.