Lineage from pipelines

閱讀約需 100 分鐘 分鐘,11 節 中共有 11 個範例。

版次 2026.11.31 範例於 2026年11 月 19 通過測試 108 個範例中有 108 個通過 變更內容

由 AI 翻譯。經人工以英文驗證。 我們如何製作書籍

譯自英文書,版次 2026.11.31。 閱讀原文

此部分尚未翻譯,因此以英文呈現。

By the end of this chapter you can capture lineage from your pipelines as OpenLineage run events, with column lineage parsed from their SQL, including the columns that only join, filter and group. You'll use the graph those events describe to say what a column change breaks downstream, and to propose classifications for tables nobody has classified. You'll also have checks for the three ways lineage fails in production: events that never arrive, a job that sends none, and a copy of personal data classified as less than personal.

The online learning platform from chapter 1 builds its reporting tables with four SQL pipelines over eight source tables. Each pipeline is one create table ... as select ... in examples/pipelines/, and the book's shared module lib.pipelines runs it on DuckDB and writes the result back to the catalog:

  • course_performance writes analytics.course_performance: enrollments, completions, ratings and revenue per course.
  • learner_activity writes analytics.learner_activity: each learner's events and last activity.
  • marketing_audience writes marketing.audience: the learners marketing may email, with their name and email.
  • review_corpus writes ai.review_corpus: the review text the platform's course assistant is grounded on.

Chapter 2's metadata check found that two of those outputs, marketing.audience and ai.review_corpus, have no metadata at all: no owner, and no classification for any column. Yet one holds learners' names and emails, and the other holds the free text of reviews, where chapter 3 found names and emails too. Two questions follow, and the catalog can't answer either. Which columns of those tables are personal data? And if the CRM team renames or reformats crm.learners.email, what breaks? Both are lineage questions. Lineage is where each column's data came from, and where it goes.

This chapter adds the book's shared module for lineage, lib.lineage. It runs each pipeline, tells OpenLineage what the run read and wrote, column by column, and reads the result back as a graph that answers both questions.

此部分尚未翻譯,因此以英文呈現。

Send a run event from every pipeline

最近一次變動於第 2026.11.27 版

OpenLineage, an open specification for lineage, describes it with three objects (object model). A job is "a process that consumes or produces Datasets", such as a pipeline. A run is one occurrence of a job, identified by a UUID the client generates, and the spec recommends UUIDv7. A dataset is a table or file, named by a namespace and a name that together identify it. A run reports itself through run events: a START when it begins, then one terminal event, COMPLETE, FAIL or ABORT, and a server treats the events for one run as accumulating (run cycle). Lineage comes mostly from the work itself: a backend learns about datasets mainly from information about jobs.

The Python client, openlineage-python 1.55.0, builds events from RunEvent, Run, Job, InputDataset and OutputDataset in openlineage.client.event_v2, and sends them through a transport (client docs). It ships HTTP and Kafka transports for servers, and a file transport "designed mainly for integration testing", which with append set writes one JSON event per line to a single file (configuration). The client computes no lineage. It serializes what the caller passes in and adds only metadata of its own: a run facet with its version, the job's source location when it runs in a Git checkout, and any environment variables you configure it to collect (emit in client/client.py, openlineage-python). This means the lineage that reaches a server is the lineage your code put in the event.

lib.lineage.run wraps lib.pipelines.run, which runs one of the platform's four SQL pipelines and writes its table to the catalog. It sends START with the tables the SQL reads as inputs, runs the pipeline, and then sends COMPLETE with the output dataset and two facets on it. A facet is OpenLineage's name for a block of metadata attached to a run, a job or a dataset, and these two hold the output's schema and its column lineage, which the next section builds. If the pipeline raises, it sends FAIL instead, and lib.lineage.run_all runs all four pipelines this way, into one events file. The datasets need a namespace, and OpenLineage's naming conventions have no entry for an Iceberg table in a local SQL catalog. The nearest is the local file system, named by its path, which would put a folder on your machine into every event. Hence lib.lineage uses the placeholder iceberg://platform, with the catalog's identifiers, such as crm.learners, as names.

The examples need the same bytes every time, yet an event's time and random run ID change by default. Here lib.lineage stamps every event with the nominal start of the nightly run on the book's "today", and derives the run ID with the client's generate_static_uuid, which, its docstring says, "always produces the same result" for the same time and data: here, the job's name. Run the four pipelines with lineage twice, into two files, and print what the first file's events say, not the raw JSON:

pipeline-run-events.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""Runs the four pipelines with lineage twice, into two files, and reads back what the first file's events say. Theprose rests on the counts of events and run IDs, on the run facet the client adds, and on the same-bytes check, whichthe example asserts."""from openlineage.client.uuid import generate_new_uuid, generate_static_uuidfrom lib import lineagefrom lib.catalog import open_catalogfrom lib.platform import scratchfolder = scratch()catalog = open_catalog()lineage.run_all(catalog, folder / "first.jsonl")lineage.run_all(catalog, folder / "second.jsonl")          # the same nightly run, sent again# lib.lineage fixes the event time and derives the run IDsassert (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes()events = lineage.read_events(folder / "first.jsonl")print("events:", len(events))for event in events:    job, state = event["job"]["name"], event["eventType"]    if state == "START":        print(f"{job} START, reads:", ", ".join(dataset["name"] for dataset in event["inputs"]))    else:        output = event["outputs"][0]        lineage_fields = output["facets"]["columnLineage"]["fields"]        print(f"{job} {state}, writes:", output["name"], f"({len(lineage_fields)} columns with lineage)")print("event times:", ", ".join(sorted({event["eventTime"] for event in events})))print("distinct run IDs:", len({event["run"]["runId"] for event in events}))print("run facets the client added:", ", ".join(events[0]["run"]["facets"]))print("same bytes when sent again:", (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes())# generate_new_uuid draws a random UUID each time; generate_static_uuid derives one from its arguments.print("two random run IDs equal:", generate_new_uuid() == generate_new_uuid())print("two static run IDs equal:", generate_static_uuid(lineage.NIGHTLY, b"marketing_audience")      == generate_static_uuid(lineage.NIGHTLY, b"marketing_audience"))
輸出python pipeline-run-events.py
events: 8
course_performance START, reads: billing.payments, learning.courses, learning.enrollments, learning.reviews
course_performance COMPLETE, writes: analytics.course_performance (7 columns with lineage)
learner_activity START, reads: crm.learners, learning.events
learner_activity COMPLETE, writes: analytics.learner_activity (6 columns with lineage)
marketing_audience START, reads: crm.learners, learning.courses, learning.enrollments
marketing_audience COMPLETE, writes: marketing.audience (5 columns with lineage)
review_corpus START, reads: learning.courses, learning.reviews
review_corpus COMPLETE, writes: ai.review_corpus (5 columns with lineage)
event times: 2026-10-01T02:00:00+00:00
distinct run IDs: 4
run facets the client added: tags
same bytes when sent again: True
two random run IDs equal: False
two static run IDs equal: True

Sending the same nominal run again wrote the same bytes, as random run IDs never would, and the example asserts it. The first file holds 8 events, a START and a COMPLETE for each of 4 runs, all stamped 2026-10-01T02:00:00+00:00. The marketing_audience job's START names the tables its SQL reads, crm.learners, learning.courses and learning.enrollments, and its COMPLETE names marketing.audience, with lineage for each column. Yet the events hold something you didn't put there: a run facet named tags, carrying the client's own version (changelog, 1.38.0), and every facet's _producer URL names that version too (DEFAULT_PRODUCER in client/constants.py). So the bytes change with each client release, though the lineage doesn't, and a test should compare what events say, as this example's summary does.

A production emitter stamps COMPLETE with the time the run finished. A derived run ID is still worth keeping there, made from the scheduled time, the job's name and the attempt number: an event sent twice then lands in the same run instead of creating a second one, and a retried job gets a run of its own. Production events should also carry the output's Iceberg snapshot ID, in the version dataset facet the object model names for it. The code here leaves it out, because PyIceberg 0.12.0 draws each snapshot ID from a random UUID (_generate_snapshot_id in its pyiceberg/table/metadata.py).

Working rule: Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run can happen without them, and name datasets by catalog identifiers under a namespace you choose, never a path. In tests, use the file transport with a fixed event time and static run IDs, and compare what events say, never their bytes. In production, send through a transport to your lineage server, and derive each run ID from the scheduled time, the job and the attempt, so a resent event never creates a duplicate run.

追蹤每個輸出欄位的來源

最近一次變動於第 2026.11.30 版

執行事件會說明一項作業讀取了哪些資料表、寫入了哪些資料表。欄位血緣(column lineage)則說明每個輸出欄位來自哪些輸入欄位,而這正是治理所需要的:一張讀取 crm.learners 的資料表,可能複製了電子郵件,也可能沒有。OpenLineage 把欄位血緣放在輸出端的資料集 facet columnLineage 中:fields 將每個輸出欄位對應到它的 inputFields,另有一份選用的 dataset 清單,記錄影響整個資料集的輸入(facet 結構描述)。由於用戶端不會計算這些內容,必須有工具去剖析每項作業的 SQL。

sqlglot 是一個 SQL 剖析器,它用 sqlglot.lineage.lineage 來做這件事。給定一段陳述式以及它所讀取資料表的結構描述,它會為每個輸出欄位回傳一棵樹,樹的葉節點就是來源欄位;使用 column=None 時,則會一次回傳所有輸出欄位(sqlglot lineage)。它需要結構描述。沒有結構描述時,SELECT * 只會回傳一個名為 * 的輸出,而連接(join)中未加限定的欄位,則會變成沒有資料表的佔位項目。lib.lineage.schema 會依據目錄的 Iceberg 中繼資料建立結構描述,使剖析器看到的是倉儲實際擁有的欄位,所用的版本為 sqlglot 30.33.0。

先從 marketing_audience 開始,它供應行銷部門可以寄送電子郵件的名單。用文字描述,它會挑出同意接收行銷電子郵件且帳號為啟用狀態的學員,並為每位學員加上其最喜愛的課程類別,這個類別由一個 CTE 依學員的選課次數對類別排名而算出。向 sqlglot 查詢該管線各輸出欄位以及 course_performance 兩個欄位的血緣,看看它遺漏了什麼:

sqlglot-misses-row-inputs.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""What sqlglot.lineage says the pipeline outputs come from, and what it leaves out. The prose rests on the CTE-tracedsource of favourite_category, on the sources of the two course performance columns, and on the assertion that neithercolumn the WHERE reads reaches any output."""import sqlglotfrom sqlglot import expfrom lib import lineage, pipelinesfrom lib.catalog import open_catalog# sqlglot resolves each name against the catalog's columns, so lineage.schema builds them from the table metadata.schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",                                      "billing.payments"]))statement = pipelines.sql("marketing_audience")found = lineage.sqlglot_sources(statement, schema)for column, sources in found.items():    print(f"marketing.audience.{column} comes from:", ", ".join(sources))parsed = sqlglot.parse_one(statement, read="duckdb")# the WHERE names columns by alias, as in l.status, so each alias maps to its tabletables = {table.alias_or_name: f"{table.db}.{table.name}" for table in parsed.find_all(exp.Table)}where = parsed.find(exp.Where)print("the pipeline's WHERE:", where.this.sql(dialect="duckdb"))in_lineage = {source for sources in found.values() for source in sources}     # every source column of every outputfor column in where.find_all(exp.Column):    source = f"{tables[column.table]}.{column.name}"    assert source not in in_lineage        # every column the WHERE reads, checked against the lineage above    print(f"{source} in any column's lineage:", source in in_lineage)course_performance = lineage.sqlglot_sources(pipelines.sql("course_performance"), schema)for column in ("completions", "revenue_usd"):    # the two aggregates that carry a FILTER inside them    print(f"analytics.course_performance.{column} comes from:", ", ".join(course_performance[column]))
輸出python sqlglot-misses-row-inputs.py
marketing.audience.learner_id comes from: crm.learners.learner_id
marketing.audience.full_name comes from: crm.learners.full_name
marketing.audience.email comes from: crm.learners.email
marketing.audience.country comes from: crm.learners.country
marketing.audience.favourite_category comes from: learning.courses.category
the pipeline's WHERE: l.marketing_consent AND l.status = 'active'
crm.learners.marketing_consent in any column's lineage: False
crm.learners.status in any column's lineage: False
analytics.course_performance.completions comes from: learning.enrollments.completed_at, learning.enrollments.enrollment_id
analytics.course_performance.revenue_usd comes from: billing.payments.amount, billing.payments.status

這些值是正確的,但決定哪些學員會出現在資料表中的條件卻不見了。email 來自 crm.learners.email,而 favourite_category 經由 CTE 追溯到 learning.courses.category。管線的 WHERE 是 l.marketing_consent AND l.status = 'active',它的兩個欄位 marketing_consent 與 status,都不在任何輸出的血緣中。這個範例對兩者都做了斷言。同意條件的篩選,正是行銷部門可以持有這張資料表的理由,血緣卻完全沒有提到它。選課資料與課程資料的連接鍵也同樣沒有被提到。

第二個問題正好相反:只用來篩選的輸入,被當成值列了出來。course_performance 只加總付款成功的款項,使用的是 filter (where p.status = 'succeeded'),而 sqlglot 說 revenue_usd 來自 billing.payments.amount, billing.payments.status:被加總的欄位,與只用來決定哪些付款要計入的欄位,被一視同仁地列出。completions 以 filter (where e.completed_at is not null) 計算選課數,回傳的結果也是如此,來自 learning.enrollments.completed_at, learning.enrollments.enrollment_id。這兩個問題都源於 sqlglot 建構樹的方式。它收集每個輸出之運算式中的欄位,因此能看到運算式內部的篩選,卻看不到運算式之外的部分,而且它的節點沒有型別,無法說明某個輸入是如何被使用的。

實用原則: 以 sqlglot 的 lineage 取得每個輸出值所計算自的欄位,並且一律傳入目錄的結構描述,該結構描述由資料表中繼資料產生,絕不手動輸入。不要把它的答案當作某個欄位所依賴的全部內容:它會漏掉用於連接、篩選與分組的欄位,也不會說明它列出的輸入中哪些是值。

此部分尚未翻譯,因此以英文呈現。

Add the inputs that decide which rows exist

最近一次變動於第 2026.11.31 版

sqlglot's lineage lists the columns a value is computed from, and OpenLineage has a place for the rest. Each input field carries a list of transformations, each with a type, a subtype and a masking flag (column lineage facet). DIRECT means the output value is derived from the input value, through IDENTITY, TRANSFORMATION or AGGREGATION. INDIRECT means the output value is impacted by an input without being derived from it, through JOIN, GROUP_BY, FILTER, SORT, WINDOW or CONDITIONAL. Inputs that shape every row, such as a WHERE, belong in the facet's dataset list.

For governance, the INDIRECT side matters, since information flows through conditions as well as values. Denning calls the information that flows through conditions "implicit flow": a statement that assigns only when a condition holds still tells you the condition's value (Denning, 1976). A row in marketing.audience tells you that learner consented.

lib.lineage.column_lineage adds two things sqlglot's answer lacks: a type for each value input, and the INDIRECT inputs. It types each value input by where the column sits in the output's expression: a bare column is IDENTITY, a column inside an aggregate is AGGREGATION, and anything else is TRANSFORMATION. A column inside an aggregate's FILTER, a window's PARTITION BY or ORDER BY, or a CASE's condition or operand is INDIRECT, and a hash on the way sets masking. COALESCE's arguments count as values: OpenLineage lists COALESCE under CONDITIONAL, but any of its arguments can become the output's value. Then it walks every SELECT in the statement, CTEs included, and collects the columns of its row clauses, following CTE columns back to the tables they come from:

ROW_CLAUSES = {"joins": "JOIN", "where": "FILTER", "group": "GROUP_BY", "having": "FILTER", "qualify": "FILTER",
               "order": "SORT"}

Run it on the same pipeline and the same two columns:

indirect-column-lineage.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""The book's column lineage for the marketing audience pipeline and two course performance columns: each input,with its OpenLineage type and subtype, then the inputs that shape every row. The prose rests on the source counts."""from lib import lineage, pipelinesfrom lib.catalog import open_catalogschema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",                                      "billing.payments"]))# rows holds the inputs that shape every row, which OpenLineage keeps in the dataset listfields, rows = lineage.column_lineage(pipelines.sql("marketing_audience"), schema)for column, inputs in fields.items():    print(f"marketing.audience.{column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"                                                    for dependency in inputs))print("inputs that shape every row of marketing.audience:")for source in sorted({dependency.source for dependency in rows}):    print(f"  {source}:", "INDIRECT",          ", ".join(dependency.subtype for dependency in rows if dependency.source == source))# the source columns that the output values are computed fromvalues = {dependency.source for inputs in fields.values() for dependency in inputs}print("source columns, values only:", len(values))# A column can both give a value and shape the rows, so the union counts it once.print("source columns, with the rows' inputs:", len(values | {dependency.source for dependency in rows}))fields, _ = lineage.column_lineage(pipelines.sql("course_performance"), schema)   # only these two columns are printedfor column in ("completions", "revenue_usd"):    print(f"analytics.course_performance.{column}:",          ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}" for dependency in fields[column]))
輸出python indirect-column-lineage.py
marketing.audience.learner_id: crm.learners.learner_id DIRECT IDENTITY
marketing.audience.full_name: crm.learners.full_name DIRECT IDENTITY
marketing.audience.email: crm.learners.email DIRECT IDENTITY
marketing.audience.country: crm.learners.country DIRECT IDENTITY
marketing.audience.favourite_category: learning.courses.category DIRECT IDENTITY
inputs that shape every row of marketing.audience:
  crm.learners.learner_id: INDIRECT JOIN
  crm.learners.marketing_consent: INDIRECT FILTER
  crm.learners.status: INDIRECT FILTER
  learning.courses.category: INDIRECT GROUP_BY, JOIN
  learning.courses.course_id: INDIRECT JOIN
  learning.enrollments.course_id: INDIRECT JOIN
  learning.enrollments.learner_id: INDIRECT GROUP_BY, JOIN
source columns, values only: 5
source columns, with the rows' inputs: 10
analytics.course_performance.completions: learning.enrollments.completed_at INDIRECT FILTER, learning.enrollments.enrollment_id DIRECT AGGREGATION
analytics.course_performance.revenue_usd: billing.payments.amount DIRECT AGGREGATION, billing.payments.status INDIRECT FILTER

The values match sqlglot's, each an IDENTITY copy, and the rows' inputs are the new part. Adding them grows what marketing.audience depends on from 5 source columns to 10. Some row inputs are columns no value uses: marketing_consent and status as FILTER, the two course IDs as JOIN, and the enrollments' learner_id as GROUP_BY and JOIN. crm.learners.learner_id and learning.courses.category, which values copy too, also shape rows: the ID as JOIN, the category as GROUP_BY and JOIN. For revenue_usd, the two inputs are now told apart: amount is a DIRECT AGGREGATION, and status an INDIRECT FILTER. So are the two inputs of completions: enrollment_id is counted, and completed_at only filters.

One more detail: the per-field transformationType and transformationDescription are deprecated, replaced by the per-input transformations that lib.lineage writes (changelog, 1.17.1). And OpenLineage 1.53.0 added a Lineage dataset facet for declared lineage, such as a view's, sent on a DatasetEvent, which consumers should prefer where both describe the same relationship (lineage facet). Run events from jobs carry columnLineage, as the platform's do.

Working rule: Record INDIRECT inputs with every job's column lineage: the join keys, filters and grouping columns of every SELECT, CTEs included, in the facet's dataset list, and type each value input with OpenLineage's subtypes. Expect more edges, many of them join keys. Keep them anyway: a filter on a consent column decides who may be emailed, and it reaches the table only as an INDIRECT input.

此部分尚未翻譯,因此以英文呈現。

Fail on statements the parser can't read

最近一次變動於第 2026.11.27 版

The platform's column lineage is only as good as the parse behind it, and a parse can be wrong without failing. sqlglot's lineage accepts more than a SELECT: it finds the query inside a CREATE TABLE ... AS, an INSERT ... SELECT or a CREATE VIEW, and it reads a MERGE too (sqlglot, optimizer/scope.py). Three cases come back looking fine. An INSERT's outputs are named after its SELECT, never after the table's columns. A MERGE comes back as the last query sqlglot reaches in it, as if that query were what the MERGE writes. When USING names a table or a CTE and no subquery follows, sqlglot returns no lineage and no error. When USING is a subquery, or a subquery sits in a WHEN condition or a SET, it returns the columns of the last such subquery, whether the MERGE writes them or not. And a misspelt column gets through when it's unqualified in a query on one table: sqlglot assigns it to that table whether the table has the column or not.

All four of the platform's pipelines are CREATE TABLE ... AS statements. As tables grow, pipelines like these often become incremental: an INSERT that appends a day's revenue to analytics.daily_revenue, or a MERGE that updates analytics.learner_activity in place. Ask sqlglot and lib.lineage.column_lineage about each, the INSERT and the MERGE in two forms, and compare the answers:

insert-and-merge-lineage.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""sqlglot.lineage's answers for an INSERT and for two MERGE statements, beside the book's lineage: sqlglot names theINSERT's outputs after its SELECT, and returns nothing or a subquery's columns for a MERGE, which the book refuses. Theprose rests on the sqlglot outputs and on the refusals at the end."""import pyarrow as pafrom sqlglot.lineage import lineage as sqlglot_lineagefrom lib import lineagefrom lib.catalog import open_catalogcatalog = open_catalog(["billing.payments", "crm.learners"])catalog.create_namespace("analytics")             # the incremental table, as it exists before the day's INSERTcatalog.create_table("analytics.daily_revenue", schema=pa.schema([("day", pa.date32()), ("revenue_usd", pa.float64())]))schema = lineage.schema(catalog)insert = """insert into analytics.daily_revenueselect cast(p.paid_at as date), sum(p.amount) from billing.payments p where p.status = 'succeeded' group by 1"""merges = {"USING a table": """merge into analytics.learner_activity a using crm.learners lon a.learner_id = l.learner_id when matched then update set country = l.country""",          "USING a subquery": """merge into analytics.learner_activity ausing (select l.learner_id, l.country, l.email from crm.learners l) s on a.learner_id = s.learner_idwhen not matched then insert (learner_id, country) values (s.learner_id, s.country)"""}print("analytics.daily_revenue columns:", ", ".join(schema["analytics"]["daily_revenue"]))# The SELECT's expressions have no aliases, so sqlglot names each output after the SELECT, not the table. None asks for# every output column at once.print("sqlglot, INSERT outputs:", ", ".join(sqlglot_lineage(None, insert, schema, dialect="duckdb")))fields, rows = lineage.column_lineage(insert, schema)for column, inputs in fields.items():    print(f"book, INSERT {column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"                                               for dependency in inputs))print("book, INSERT rows shaped by:", ", ".join(f"{dependency.source} {dependency.subtype}" for dependency in rows))for label, merge in merges.items():    outputs = sqlglot_lineage(None, merge, schema, dialect="duckdb")    print(f"sqlglot, MERGE {label}, outputs:", ", ".join(outputs) or "none")# One value too many for daily_revenue's two columns, caught after sqlglot parses it; each MERGE is refused before sqlglot is asked.refused = {"INSERT with one value too many": insert.replace("sum(p.amount)", "sum(p.amount), count(*)"),           **{f"MERGE {label}": merge for label, merge in merges.items()}}for label, statement in refused.items():    try:        lineage.column_lineage(statement, schema)    except lineage.LineageError as error:        print(f"book, {label}:", error)
輸出python insert-and-merge-lineage.py
analytics.daily_revenue columns: day, revenue_usd
sqlglot, INSERT outputs: paid_at, _col_1
book, INSERT day: billing.payments.paid_at DIRECT TRANSFORMATION
book, INSERT revenue_usd: billing.payments.amount DIRECT AGGREGATION
book, INSERT rows shaped by: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER
sqlglot, MERGE USING a table, outputs: none
sqlglot, MERGE USING a subquery, outputs: learner_id, country, email
book, INSERT with one value too many: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values
book, MERGE USING a table: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
book, MERGE USING a subquery: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE

The table analytics.daily_revenue has the columns day, revenue_usd, yet sqlglot names the INSERT's outputs paid_at, _col_1: the first after the column inside the cast, the second a generated name for an expression with no alias.

An INSERT fills columns by position, so lib.lineage matches by position too, taking the names from the catalog: the columns the INSERT lists, or else all of the table's. So day comes from billing.payments.paid_at DIRECT TRANSFORMATION, and the INSERT's grouping and its filter on status shape every row: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER. When the SELECT has more or fewer values than the columns written, lib.lineage raises: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values.

For the MERGE whose USING names a table, sqlglot returns none, and no error. For the one whose USING is a subquery, it returns learner_id, country, email, though that MERGE writes only learner_id and country. A check for empty lineage catches the first and passes the second, so lib.lineage doesn't judge what sqlglot returns for a MERGE. Before it calls sqlglot, it refuses every statement that isn't a CREATE ... AS or an INSERT ... SELECT: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE.

A misspelt column, the third case, depends on how the column is written. The platform's pipelines qualify every column with its table's alias, as in l.email, and sqlglot checks a qualified column against the schema itself:

misspelt-column-lineage.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""A misspelt column, emial for email, in a query on one table: sqlglot raises when the column is qualified by itsalias, and traces it to the table when it isn't. The prose rests on the sqlglot answer for each form, and on the book'scheck against the schema."""from sqlglot.errors import OptimizeErrorfrom lib import lineagefrom lib.catalog import open_catalogschema = lineage.schema(open_catalog(["crm.learners"]))    # the schema lets sqlglot check each column name# the alias l is right, and only the column name is misspeltqualified = "create table marketing.audience as select l.learner_id, l.emial from crm.learners l"# no alias, so nothing names the table the column comes fromunqualified = "create table marketing.audience as select learner_id, emial from crm.learners"try:    lineage.sqlglot_sources(qualified, schema)except OptimizeError as error:    print("sqlglot, qualified l.emial:", f"{type(error).__name__}: {error}")print("sqlglot, unqualified emial comes from:", ", ".join(lineage.sqlglot_sources(unqualified, schema)["emial"]))try:    lineage.column_lineage(unqualified, schema)except lineage.LineageError as error:    print("book, unqualified emial:", error)
輸出python misspelt-column-lineage.py
sqlglot, qualified l.emial: OptimizeError: Unknown column: emial
sqlglot, unqualified emial comes from: crm.learners.emial
book, unqualified emial: crm.learners has no column emial

A qualified typo stops sqlglot with OptimizeError: Unknown column: emial. An unqualified one, in a query on one table, comes back as crm.learners.emial: a confident answer about a column that doesn't exist. In a join, a misspelt unqualified column belongs to none of the tables, and sqlglot returns a placeholder with no table (sqlglot, lineage.py), which lib.lineage rejects as a source it can't place. For the single-table case, lib.lineage checks every source against the schema and raises crm.learners has no column emial.

lib.lineage.record, which run uses, reads the lineage before the job runs, so a pipeline whose lineage can't be read fails before it writes anything. That is strict, and deliberate: a job with empty lineage would vanish from every impact report and classification check in this chapter, and one with wrong lineage would mislead them. Run the same check in CI on every pull request that changes a pipeline, so the failure shows up in review rather than at night.

Working rule: Treat a statement your parser can't read correctly as an error, never as empty or partial lineage. Refuse every kind of statement it gets wrong, such as MERGE, before you ask it, and fail on empty lineage, a source the parser can't place and a column the schema doesn't have. Match an INSERT's columns by position against the table's columns in the catalog, never the SELECT's names, and fail when the counts differ. For a MERGE, write a SELECT of the columns it writes, and record the job's lineage from that, as an INSERT ... SELECT into the MERGE's table, until your parser handles MERGE itself.

找出欄位變更會破壞什麼

最近一次變動於第 2026.11.27 版

每個 COMPLETE 事件都帶有欄位血緣,因此平台擁有一張圖:欄位是節點,每個輸入欄位則是一條邊,指向它所餵入的欄位或資料表。影響分析(impact analysis)要問的是,對某個欄位的變更可能破壞什麼。它從該欄位出發,沿著每一條向外的邊走,並從每條邊抵達之處繼續走下去,不限深度。這不需要特殊方法,就是圖的可達性(reachability)問題。只有一條規則是血緣所特有的:指向某個資料表的 INDIRECT 邊,代表該表的每一列都可能改變,因此走訪會從該表的所有欄位繼續進行。

lib.lineage.graph 如同血緣伺服器一樣,只憑事件檔建立這張圖,而不依據 SQL。因此,凡是會送出事件的作業,不論以何種語言撰寫,都會納入;不送出事件的作業則一律不在其中。lib.lineage.downstream 負責進行走訪。

以平台團隊可能做的三項變更為例。CRM 團隊將所有電子郵件轉為小寫,或重新命名 crm.learners.email。新的註冊流程改變了 crm.learners.marketing_consent 對舊帳號的意義。學習團隊重新命名 learning.courses 中的某個類別。請先猜猜看:同意狀態的變更可以影響多少個欄位?

column-impact.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""What a change to one source column can break, walked over the column graph that the nightly run's events describe.The prose rests on the reach counts printed for each column: through values, and in all."""from lib import lineagefrom lib.catalog import open_catalogfrom lib.platform import scratchevents_file = scratch() / "events.jsonl"lineage.run_all(open_catalog(), events_file)graph = lineage.graph(lineage.read_events(events_file))    # built from the events alone, as a lineage server wouldprint("edges in the column graph:", len(graph.edges))for column in ("crm.learners.email", "crm.learners.marketing_consent", "learning.courses.category"):    reached = lineage.downstream(graph, column)    print(f"{column} reaches:")    # dict.fromkeys drops repeated targets and keeps the order they were found in    for target in dict.fromkeys(edge.target for edge in reached):        how = ", ".join(f"{edge.type} {edge.subtype}" for edge in reached if edge.target == target)        # a table's name has one dot and a column's two, so a table target means every row of it        print(f"  {target}{', every row' if target.count('.') == 1 else ''}:", how)    values = [edge for edge in reached if edge.type == "DIRECT"]    # only DIRECT edges copy values    print(f"{column}, columns reached through values:", len(lineage.columns_reached(graph, values)))    print(f"{column}, columns reached in all:", len(lineage.columns_reached(graph, reached)))
輸出python column-impact.py
edges in the column graph: 51
crm.learners.email reaches:
  marketing.audience.email: DIRECT IDENTITY
crm.learners.email, columns reached through values: 1
crm.learners.email, columns reached in all: 1
crm.learners.marketing_consent reaches:
  marketing.audience, every row: INDIRECT FILTER
crm.learners.marketing_consent, columns reached through values: 0
crm.learners.marketing_consent, columns reached in all: 5
learning.courses.category reaches:
  analytics.course_performance, every row: INDIRECT GROUP_BY
  analytics.course_performance.category: DIRECT IDENTITY
  marketing.audience, every row: INDIRECT GROUP_BY, INDIRECT JOIN
  marketing.audience.favourite_category: DIRECT IDENTITY
learning.courses.category, columns reached through values: 2
learning.courses.category, columns reached in all: 12

同意狀態的結果出人意料。對它的變更透過值影響 0 個欄位,但透過它的篩選條件,卻影響 marketing.audience 的全部 5 個欄位,因為它改變了誰會收到電子郵件。只看值的血緣工具,會回報同意狀態的變更沒有任何影響。

對 email 的變更影響 1 個欄位,即 marketing.audience.email,因此重新命名它會破壞行銷管線,而不會破壞其他任何送出血緣的作業。類別的重新命名透過值影響 2 個欄位,總共影響 12 個,因為類別也用於對 analytics.course_performance 分組,並用來為每位學習者排出最喜愛的項目。

同樣的走訪會用在後面兩章:第 7 章沿著下游追蹤學習者的欄位,找出刪除請求必須涵蓋的每一個衍生資料表;第 10 章則為每個變更結構描述的 pull request 撰寫影響報告。

實用原則: 在由事件建立的圖上進行影響分析,並納入 INDIRECT 邊,然後回報兩份清單:值可能改變的欄位,以及列可能改變的資料表。僅用於篩選與連接的欄位若有變更,應視為對其所篩選之資料表的每個欄位都有變更,並將報告送交其所影響之每個資料表的負責人。

此部分尚未翻譯,因此以英文呈現。

Propose classifications from lineage

最近一次變動於第 2026.11.27 版

A derived column holds data from its inputs, so its classification can be derived from theirs. The classic rule comes from Denning's lattice model of information flow: the class of a value computed from several inputs is the least upper bound, the "join", of their classes (Denning, 1976). The platform's classifications form a line, public, internal, personal and sensitive, so the join is the stricter of them, which stricter in lib.metadata, the book's metadata module, returns. Which inputs count is a judgement. We count INDIRECT inputs too, following Denning's implicit flows, because a table filtered on consent reveals each learner's consent. That choice lives in governance/lineage.toml, so a team can change it in one reviewed line.

Two more choices follow from the same caution. OpenLineage's masking flag says only that a value was obfuscated, such as by a hash (column lineage facet). Whether a hashed identifier is still personal data is a legal question: chapter 2 quotes the GDPR recital on pseudonymized data and leaves the answer to lawyers. So the book's rule never lowers a classification because of masking. Catalog servers leave the choice to configuration too. Apache Atlas propagates classifications along lineage, and can block that on an edge, for example where a view masks a PII column (Atlas). DataHub propagates glossary terms along column lineage, a feature its 1.7.0 documentation calls a public beta in DataHub Cloud (DataHub). A proposal is a floor, not a verdict: like the drafts a local model writes in chapter 2, it's reviewed before it goes into governance/metadata.toml, the platform's file of owners and classifications.

lib.lineage.propose proposes a classification for every column a job wrote, and lib.metadata.problems lists the tables that need one. Check the rule on the two tables people did classify, analytics.course_performance and analytics.learner_activity, before you trust it on marketing.audience and ai.review_corpus, the two with no metadata:

propagate-classifications.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""Proposes classifications from lineage for the columns with no declared one, then checks the rule against the columnspeople did classify. The prose rests on the personal count, on the reproduced counts, and on the check that no declared class is lower."""from lib import lineage, metadatafrom lib.catalog import open_catalogfrom lib.platform import scratchcatalog = open_catalog()events_file = scratch() / "events.jsonl"lineage.run_all(catalog, events_file)for problem in metadata.problems(catalog, metadata.load()):    print("metadata problem:", problem)graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()   # known: governance/metadata.tomlprint("propagated through:", " and ".join(lineage.rules()["propagate_through"]))      # from governance/lineage.tomlproposed = lineage.propose(graph, known)for column, (classification, decided_by) in proposed.items():    if column not in known:        first = decided_by[0]      # lib sorts the deciding edges DIRECT first, so this is a value when one decides        print(f"{column}: {classification}, from {first.source} ({first.type} {first.subtype})")print("personal columns proposed:", sum(1 for column, (classification, _) in proposed.items()                                        if column not in known and classification == "personal"))classified = [column for column in proposed if column in known]print("declared classifications checked:", len(classified))print("classes declared for them:", ", ".join(sorted({known[column] for column in classified})))for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):    proposals = lineage.propose(graph, known, through)    agree = sum(1 for column in classified if proposals[column][0] == known[column])    print(f"declared classifications reproduced, through {' and '.join(through)}:", f"{agree} of {len(classified)}")# proposals is the DIRECT-only run from the loop abovemissed = [column for column in classified if proposals[column][0] != known[column]]print("missed with DIRECT inputs only:", len(missed))for column in missed:    print(f"  {column}: declared {known[column]}, DIRECT inputs propose {proposals[column][0]}")print("declared less protected than lineage proposes:", len(lineage.understated(proposed, known)))assert not lineage.understated(proposed, known)
輸出python propagate-classifications.py
metadata problem: ai.review_corpus: no metadata
metadata problem: marketing.audience: no metadata
propagated through: DIRECT and INDIRECT
marketing.audience.learner_id: internal, from crm.learners.learner_id (DIRECT IDENTITY)
marketing.audience.full_name: personal, from crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email: personal, from crm.learners.email (DIRECT IDENTITY)
marketing.audience.country: internal, from crm.learners.country (DIRECT IDENTITY)
marketing.audience.favourite_category: internal, from crm.learners.learner_id (INDIRECT JOIN)
ai.review_corpus.review_id: internal, from learning.reviews.review_id (DIRECT IDENTITY)
ai.review_corpus.course_id: internal, from learning.reviews.course_id (DIRECT IDENTITY)
ai.review_corpus.course_title: internal, from learning.reviews.course_id (INDIRECT JOIN)
ai.review_corpus.rating: internal, from learning.reviews.rating (DIRECT IDENTITY)
ai.review_corpus.body: personal, from learning.reviews.body (DIRECT IDENTITY)
personal columns proposed: 3
declared classifications checked: 13
classes declared for them: internal
declared classifications reproduced, through DIRECT and INDIRECT: 13 of 13
declared classifications reproduced, through DIRECT: 10 of 13
missed with DIRECT inputs only: 3
  analytics.course_performance.course_id: declared internal, DIRECT inputs propose public
  analytics.course_performance.title: declared internal, DIRECT inputs propose public
  analytics.course_performance.category: declared internal, DIRECT inputs propose public
declared less protected than lineage proposes: 0

The metadata check from chapter 2 still reports the two tables with no metadata, and lineage proposes a classification for each of their columns. It proposes 3 personal columns: full_name and email in marketing.audience, each an IDENTITY copy of a personal column in crm.learners, and body in ai.review_corpus, a copy of the reviews' free text. The rest come out internal, including favourite_category, for which the example prints internal, from crm.learners.learner_id (INDIRECT JOIN). Its values are public course-catalog category names, but the rows they sit in are chosen by a join on learner IDs and a consent filter, so those are internal.

Counting INDIRECT inputs reproduces the declared classifications on the columns people did classify, 13 of 13, and all of them are internal.

DIRECT inputs alone reproduce 10 of 13, and miss 3: course_id, title and category of analytics.course_performance. A DIRECT-only rule calls them public, as in the course catalog, while the analytics team chose internal. So the rule proposes no less than people chose here, and lineage finds no declared classification lower than its proposal. That is weak evidence: a rule that proposed internal for every column would score as well. And it can't show what the rule does with a personal filter, since none of the platform's filters or joins reads a personal column: even marketing_consent is classified internal.

So build a table whose filter does. For a birthday offer, marketing wants the learners who consented and were born this month, with their ID and country, and nothing else:

personal-filter-propagates.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""A birthday-offer table whose only personal input is a filter on date of birth. The prose rests on the learner count,and on the proposals counting DIRECT inputs alone and counting DIRECT and INDIRECT inputs."""from lib import lineagefrom lib.catalog import open_catalogfrom lib.platform import scratchfrom lib.warehouse import connectcatalog = open_catalog(["crm.learners"])# The consenting learners born in the month of the nightly run, on the book's "today".select = ("select l.learner_id, l.country from crm.learners l "          f"where l.marketing_consent and month(l.date_of_birth) = {lineage.NIGHTLY.month}")def offers() -> None:    rows = connect(catalog).sql(select).to_arrow_table()    catalog.create_namespace_if_not_exists("marketing")    catalog.create_table("marketing.birthday_offers", schema=rows.schema).append(rows)events_file = scratch() / "events.jsonl"# record derives the lineage from this statement, and reads it before offers() runslineage.record(lineage.client(events_file), "birthday_offers", f"create table marketing.birthday_offers as {select}",               catalog, offers)offers_table = catalog.load_table("marketing.birthday_offers").scan().to_arrow()print("learners in marketing.birthday_offers:", f"{len(offers_table):,}")graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()print("crm.learners.date_of_birth declared:", known["crm.learners.date_of_birth"])for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):    for column, (classification, decided_by) in lineage.propose(graph, known, through).items():        label = f"{column}, through {' and '.join(through)}"        print(f"{label}:", classification)        print(f"{label}, decided by:", f"{decided_by[0].source} ({decided_by[0].type} {decided_by[0].subtype})")
輸出python personal-filter-propagates.py
learners in marketing.birthday_offers: 165
crm.learners.date_of_birth declared: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.country, through DIRECT and INDIRECT: personal
marketing.birthday_offers.country, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.learner_id, through DIRECT: internal
marketing.birthday_offers.learner_id, through DIRECT, decided by: crm.learners.learner_id (DIRECT IDENTITY)
marketing.birthday_offers.country, through DIRECT: internal
marketing.birthday_offers.country, through DIRECT, decided by: crm.learners.country (DIRECT IDENTITY)

Neither column copies anything personal, but every row says its learner was born in the offer's month, part of a date of birth. The table holds 165 learners, and governance/metadata.toml classifies the date of birth personal. Counting DIRECT inputs alone proposes internal for learner_id and internal for country.

Add INDIRECT inputs, and it proposes personal for learner_id and personal for country, both decided by crm.learners.date_of_birth (INDIRECT FILTER). That is the case INDIRECT inputs are for, and it shows the cost too: one personal filter makes every column of a table personal.

OpenLineage can carry classifications in the events themselves, with the Tags dataset facet, whose field key "identifies the field in a dataset if a tag applies to one" (tags facet, schema). We keep them in governance/metadata.toml instead, the one source that the platform's access policies also read, through lib.policy, the book's policy module (chapter 5).

Working rule: Propose each derived column's classification as the stricter of its inputs', counting INDIRECT inputs, and send the proposals to the table's owner for review before they become metadata. Never let lineage lower a classification someone declared, and never treat masking as declassification. Count DIRECT inputs only if your pipelines often join or filter on personal columns, such as matching on email, and every output would otherwise come out personal; then review INDIRECT personal inputs by hand.

正式環境中會出什麼問題

血緣(lineage)的失敗是悄無聲息的:遺漏的邊(edge)看起來與沒有任何相依關係的欄位完全相同,因此所有建立在這張圖上的檢查,從影響分析到建議的分類,都會照常通過。常見的失敗有三種,每一種都有對應的檢查。

永遠不會送達的事件

最近一次變動於第 2026.11.27 版

用戶端會從環境中讀取 OPENLINEAGE_DISABLED,當它的值為 true 時,事件就會被丟棄(設定)。在用戶端的原始碼中,這項檢查最先執行(openlineage-python,client/client.py):在它查看你明確傳入的傳輸機制之前,用戶端就已經選了一個什麼都不做的傳輸機制。它唯一的提示是一行 INFO 記錄:「OpenLineage is disabled. No events will be emitted.」Python 的根記錄器預設層級為 WARNING,因此這一行永遠不會顯示(logging HOWTO)。只要在除錯後把這個變數遺留在共用的環境檔中,就足以造成這種情況。請設定該變數,以明確指定的檔案傳輸機制執行 review_corpus,並檢查檔案中收到了什麼:

lineage-disabled-silently.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""With OPENLINEAGE_DISABLED=true in the environment, an explicit file transport writes nothing and raises nothing. Theprose rests on the transport the client chose and on the job the check finds without a COMPLETE event."""import osfrom lib import lineagefrom lib.catalog import open_catalogfrom lib.platform import scratchos.environ["OPENLINEAGE_DISABLED"] = "true"        # say, left in a shared environment file after a debugging sessionevents_file = scratch() / "events.jsonl"client = lineage.client(events_file)                # a file transport, passed in explicitlycatalog = open_catalog(["learning.courses", "learning.reviews"])output = lineage.run(catalog, "review_corpus", client)print("transport the client chose:", client.transport.kind)      # OPENLINEAGE_DISABLED wins over the transport passed inprint("pipeline wrote:", output)print("rows written:", f"{len(catalog.load_table(output).scan().to_arrow()):,}")assert not events_file.exists()         # the noop transport never creates the fileprint("events file exists:", events_file.exists())missing = lineage.missing_runs(["review_corpus"], lineage.read_events(events_file))   # ran, but sent no COMPLETEprint("jobs that ran with no COMPLETE event:", ", ".join(missing))
輸出python lineage-disabled-silently.py
transport the client chose: noop
pipeline wrote: ai.review_corpus
rows written: 3,000
events file exists: False
jobs that ran with no COMPLETE event: review_corpus

設定該變數後,管線仍會執行,但檔案中沒有任何內容。用戶端選擇了 noop 傳輸機制,而不是程式碼傳入的檔案傳輸機制;管線將 3,000 列資料寫入 ai.review_corpus,沒有任何錯誤,也沒有產生事件檔。能找出這個問題的檢查,是比對排程器執行過的作業與送出 COMPLETE 事件的作業:lib.lineage.missing_runs 會將 review_corpus 列為已執行但沒有 COMPLETE 事件。

**實用原則:**每次排程執行之後,都要比對已執行的作業與 COMPLETE 事件已送達的作業,只要有任何差異就發出警示。不要仰賴事件發送端自行回報它的沉默。

沒有人加上儀測的作業

最近一次變動於第 2026.11.27 版

血緣只涵蓋會送出事件的作業。平台的行銷團隊每週執行一支 Python 指令碼,把 marketing.audience 複製到 marketing.campaign_list,這是其電子郵件工具所讀取的資料表。沒有人把這支指令碼接上 OpenLineage:

uninstrumented-job.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""A Python step copies marketing.audience for the email tool and sends no lineage. The prose rests on the walk from theemail column, which misses the copy, and on the catalog check against the lineage, which finds it."""from lib import lineagefrom lib.catalog import open_catalogfrom lib.platform import scratchfrom lib.warehouse import connectcatalog = open_catalog()events_file = scratch() / "events.jsonl"lineage.run_all(catalog, events_file)# The marketing team's weekly export, written straight to the catalog: no lineage event.export = connect(catalog).sql("select learner_id, full_name, email from marketing.audience").to_arrow_table()catalog.create_table("marketing.campaign_list", schema=export.schema).append(export)print("rows exported:", f"{len(export):,}")events = lineage.read_events(events_file)reached = lineage.downstream(lineage.graph(events), "crm.learners.email")print("crm.learners.email reaches:", ", ".join(edge.target for edge in reached))# the catalog's tables, less the platform's sources and the outputs the events nameprint("tables no job's lineage accounts for:", ", ".join(lineage.unaccounted(catalog, events)))
輸出python uninstrumented-job.py
rows exported: 1,829
crm.learners.email reaches: marketing.audience.email
tables no job's lineage accounts for: marketing.campaign_list

不送出事件的匯出作業,在影響分析中是看不見的。這支指令碼複製了 1,829 位學習者的姓名與電子郵件,但 crm.learners.email 在血緣中仍然只會到達 marketing.audience.email,因此依這份血緣進行的刪除作業會漏掉這份複本。不過,資料目錄知道這張資料表存在,所以 lib.lineage.unaccounted 會將資料目錄中的資料表與血緣相互比對:每一張資料表都必須是平台的來源之一,或是血緣所知某個作業的輸出。它找出的結果是 marketing.campaign_list。

實用原則: 定期將資料目錄與血緣互相核對,並把沒有任何作業的血緣能夠說明的資料表視為事故:找出該作業,並讓它送出事件。

此部分尚未翻譯,因此以英文呈現。

Personal data in a table classified internal

最近一次變動於第 2026.11.27 版

Instrumenting the marketing team's export fixes the graph, and exposes a second problem. When the export's owner created marketing.campaign_list, they classified every column internal, because only the marketing team reads it. That's a statement about who reads a table, not what it holds. Wrap the script's work in lib.lineage.record, with the SELECT it runs as its lineage, and compare the declared classifications with lineage's proposal:

personal-copy-tagged-internal.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通過
"""The same export, now recorded with lineage, into a table its owner tagged internal. The prose rests on the walk tothe copy and on the check for understated classifications."""from lib import lineagefrom lib.catalog import open_catalogfrom lib.platform import scratchfrom lib.warehouse import connectcatalog = open_catalog()events_file = scratch() / "events.jsonl"lineage.run_all(catalog, events_file)select = "select a.learner_id, a.full_name, a.email from marketing.audience a"def export() -> None:    rows = connect(catalog).sql(select).to_arrow_table()    catalog.create_table("marketing.campaign_list", schema=rows.schema).append(rows)# record reads this statement's lineage before export() runsstatement = f"create table marketing.campaign_list as {select}"lineage.record(lineage.client(events_file), "campaign_export", statement, catalog, export)# Its owner tagged every column internal: "only the marketing team reads it".known = lineage.declared() | {f"marketing.campaign_list.{c}": "internal" for c in ("learner_id", "full_name", "email")}graph = lineage.graph(lineage.read_events(events_file))print("crm.learners.email reaches:", ", ".join(edge.target for edge in lineage.downstream(graph, "crm.learners.email")))for problem in lineage.understated(lineage.propose(graph, known), known):    print("understated:", problem)
輸出python personal-copy-tagged-internal.py
crm.learners.email reaches: marketing.audience.email, marketing.campaign_list.email
understated: marketing.campaign_list.full_name: declared internal, lineage proposes personal
understated: marketing.campaign_list.email: declared internal, lineage proposes personal

Once the export sends lineage, the walk reaches its copy: crm.learners.email reaches marketing.audience.email, marketing.campaign_list.email, two hops, the second through the export. And lib.lineage.understated reports full_name and email in marketing.campaign_list as declared internal while lineage proposes personal: copies of personal data that any policy reading the declared classification would treat as internal.

Working rule: Run lib.lineage.understated, or your own check like it, on every lineage update, and fail when a declared classification is lower than lineage proposes. Classify by what a table holds, never by who reads it, and fix such a finding by raising the classification, not by dropping the edge.

本章的決定

最近一次變動於第 2026.11.27 版
  • 每次管線執行都會送出 OpenLineage 執行事件, 先是 START,接著是 COMPLETE 或 FAIL,由執行該管線的程式碼(lib.lineage.run)送出,並附上輸出的結構描述與欄位血緣。資料集以目錄識別碼命名,置於預留位置命名空間 iceberg://platform 之下。範例會固定事件時間,並由名義上的執行推導出執行 ID,且比較的是事件所表達的內容,而不是其位元組。
  • 欄位血緣採用 sqlglot 的值血緣,並標示為 OpenLineage 子類型,再加上 INDIRECT 輸入: 即每個 SELECT 的 join、篩選、分組與排序欄位(含 CTE),列於該 facet 的 dataset 清單中。
  • 無法正確讀取的血緣一律視為錯誤, 絕不留空,也不會只給出部分結果。lib.lineage 只讀取 CREATE ... AS 與 INSERT ... SELECT,其他所有陳述式一律拒絕,包括 MERGE,不論其 USING 指向什麼。血緣為空、來源無法定位或欄位未知時,作業會在執行前停止。INSERT 的欄位來自目錄中該資料表的結構描述,依位置對應,數量不一致即為錯誤。
  • 影響分析會走訪由事件建立的圖, 包含 INDIRECT 邊,並回報值可能改變的欄位,以及資料列可能改變的資料表。
  • 分類會傳遞,取所有輸入中較嚴格者, 包含 DIRECT 與 INDIRECT,依 governance/lineage.toml 的設定,因此由個人篩選條件所選出的資料表,會被建議為個人資料。血緣絕不會降低已宣告的分類,遮罩也絕不會解除分類。針對兩個未分類資料表的建議(full_name、email 與 body 為個人資料,其他每個欄位為內部資料),會先交給各自的負責人審查,才會納入 governance/metadata.toml。
  • 三項檢查用來保護血緣本身: 執行過卻沒有 COMPLETE 事件的作業、沒有任何作業的血緣能解釋其來源的資料表,以及已宣告的分類低於血緣所建議的分類。血緣圖只會和它的事件一樣完整,而缺少的邊看起來與沒有任何相依的欄位完全相同。

本章,於本版次中

版次 2026.11.31範例於 2026年11 月 19 通過測試本章 11 個範例 個範例中有 11 個通過

譯自英文書,版次 2026.11.31。

引用本節

格式

各種格式皆引用英文原書,本譯本即依循該原書。

每種格式都會標示本版。其連結會開啟目前版次,節標題會說明該節內文最後一次變動的時間。存取日期與擷取日期即為你開啟此頁的當日。

本頁內容

第 4 章,共 10 章: Lineage from pipelines

  1. Send a run event from every pipeline
  2. 追蹤每個輸出欄位的來源
  3. Add the inputs that decide which rows exist
  4. Fail on statements the parser can't read
  5. 找出欄位變更會破壞什麼
  6. Propose classifications from lineage
  7. 正式環境中會出什麼問題
  8. 永遠不會送達的事件
  9. 沒有人加上儀測的作業
  10. Personal data in a table classified internal
  11. 本章的決定

返回頂端所有章節