Lineage from pipelines
阅读约需 101分钟 分钟,11节中含11个示例。
这部分尚未翻译,因此以英文显示。
By the end of this chapter you can capture lineage from your pipelines as OpenLineage run events, with column lineage parsed from their SQL, including the columns that only join, filter and group. You'll use the graph those events describe to say what a column change breaks downstream, and to propose classifications for tables nobody has classified. You'll also have checks for the three ways lineage fails in production: events that never arrive, a job that sends none, and a copy of personal data classified as less than personal.
The online learning platform from chapter 1 builds its reporting tables with four SQL pipelines over eight source tables. Each pipeline is one create table ... as select ... in examples/pipelines/, and the book's shared module lib.pipelines runs it on DuckDB and writes the result back to the catalog:
course_performancewritesanalytics.course_performance: enrollments, completions, ratings and revenue per course.learner_activitywritesanalytics.learner_activity: each learner's events and last activity.marketing_audiencewritesmarketing.audience: the learners marketing may email, with their name and email.review_corpuswritesai.review_corpus: the review text the platform's course assistant is grounded on.
Chapter 2's metadata check found that two of those outputs, marketing.audience and ai.review_corpus, have no metadata at all: no owner, and no classification for any column. Yet one holds learners' names and emails, and the other holds the free text of reviews, where chapter 3 found names and emails too. Two questions follow, and the catalog can't answer either. Which columns of those tables are personal data? And if the CRM team renames or reformats crm.learners.email, what breaks? Both are lineage questions. Lineage is where each column's data came from, and where it goes.
This chapter adds the book's shared module for lineage, lib.lineage. It runs each pipeline, tells OpenLineage what the run read and wrote, column by column, and reads the result back as a graph that answers both questions.
这部分尚未翻译,因此以英文显示。
Send a run event from every pipeline
OpenLineage, an open specification for lineage, describes it with three objects (object model). A job is "a process that consumes or produces Datasets", such as a pipeline. A run is one occurrence of a job, identified by a UUID the client generates, and the spec recommends UUIDv7. A dataset is a table or file, named by a namespace and a name that together identify it. A run reports itself through run events: a START when it begins, then one terminal event, COMPLETE, FAIL or ABORT, and a server treats the events for one run as accumulating (run cycle). Lineage comes mostly from the work itself: a backend learns about datasets mainly from information about jobs.
The Python client, openlineage-python 1.55.0, builds events from RunEvent, Run, Job, InputDataset and OutputDataset in openlineage.client.event_v2, and sends them through a transport (client docs). It ships HTTP and Kafka transports for servers, and a file transport "designed mainly for integration testing", which with append set writes one JSON event per line to a single file (configuration). The client computes no lineage. It serializes what the caller passes in and adds only metadata of its own: a run facet with its version, the job's source location when it runs in a Git checkout, and any environment variables you configure it to collect (emit in client/client.py, openlineage-python). This means the lineage that reaches a server is the lineage your code put in the event.
lib.lineage.run wraps lib.pipelines.run, which runs one of the platform's four SQL pipelines and writes its table to the catalog. It sends START with the tables the SQL reads as inputs, runs the pipeline, and then sends COMPLETE with the output dataset and two facets on it. A facet is OpenLineage's name for a block of metadata attached to a run, a job or a dataset, and these two hold the output's schema and its column lineage, which the next section builds. If the pipeline raises, it sends FAIL instead, and lib.lineage.run_all runs all four pipelines this way, into one events file. The datasets need a namespace, and OpenLineage's naming conventions have no entry for an Iceberg table in a local SQL catalog. The nearest is the local file system, named by its path, which would put a folder on your machine into every event. Hence lib.lineage uses the placeholder iceberg://platform, with the catalog's identifiers, such as crm.learners, as names.
The examples need the same bytes every time, yet an event's time and random run ID change by default. Here lib.lineage stamps every event with the nominal start of the nightly run on the book's "today", and derives the run ID with the client's generate_static_uuid, which, its docstring says, "always produces the same result" for the same time and data: here, the job's name. Run the four pipelines with lineage twice, into two files, and print what the first file's events say, not the raw JSON:
1"""Runs the four pipelines with lineage twice, into two files, and reads back what the first file's events say. The2prose rests on the counts of events and run IDs, on the run facet the client adds, and on the same-bytes check, which3the example asserts."""45from openlineage.client.uuid import generate_new_uuid, generate_static_uuid67from lib import lineage8from lib.catalog import open_catalog9from lib.platform import scratch1011folder = scratch()12catalog = open_catalog()13lineage.run_all(catalog, folder / "first.jsonl")14lineage.run_all(catalog, folder / "second.jsonl") # the same nightly run, sent again15# lib.lineage fixes the event time and derives the run IDs16assert (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes()17events = lineage.read_events(folder / "first.jsonl")1819print("events:", len(events))20for event in events:21 job, state = event["job"]["name"], event["eventType"]22 if state == "START":23 print(f"{job} START, reads:", ", ".join(dataset["name"] for dataset in event["inputs"]))24 else:25 output = event["outputs"][0]26 lineage_fields = output["facets"]["columnLineage"]["fields"]27 print(f"{job} {state}, writes:", output["name"], f"({len(lineage_fields)} columns with lineage)")2829print("event times:", ", ".join(sorted({event["eventTime"] for event in events})))30print("distinct run IDs:", len({event["run"]["runId"] for event in events}))31print("run facets the client added:", ", ".join(events[0]["run"]["facets"]))32print("same bytes when sent again:", (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes())33# generate_new_uuid draws a random UUID each time; generate_static_uuid derives one from its arguments.34print("two random run IDs equal:", generate_new_uuid() == generate_new_uuid())35print("two static run IDs equal:", generate_static_uuid(lineage.NIGHTLY, b"marketing_audience")36 == generate_static_uuid(lineage.NIGHTLY, b"marketing_audience"))
python pipeline-run-events.pyevents: 8
course_performance START, reads: billing.payments, learning.courses, learning.enrollments, learning.reviews
course_performance COMPLETE, writes: analytics.course_performance (7 columns with lineage)
learner_activity START, reads: crm.learners, learning.events
learner_activity COMPLETE, writes: analytics.learner_activity (6 columns with lineage)
marketing_audience START, reads: crm.learners, learning.courses, learning.enrollments
marketing_audience COMPLETE, writes: marketing.audience (5 columns with lineage)
review_corpus START, reads: learning.courses, learning.reviews
review_corpus COMPLETE, writes: ai.review_corpus (5 columns with lineage)
event times: 2026-10-01T02:00:00+00:00
distinct run IDs: 4
run facets the client added: tags
same bytes when sent again: True
two random run IDs equal: False
two static run IDs equal: True
Sending the same nominal run again wrote the same bytes, as random run IDs never would, and the example asserts it. The first file holds 8 events, a START and a COMPLETE for each of 4 runs, all stamped 2026-10-01T02:00:00+00:00. The marketing_audience job's START names the tables its SQL reads, crm.learners, learning.courses and learning.enrollments, and its COMPLETE names marketing.audience, with lineage for each column. Yet the events hold something you didn't put there: a run facet named tags, carrying the client's own version (changelog, 1.38.0), and every facet's _producer URL names that version too (DEFAULT_PRODUCER in client/constants.py). So the bytes change with each client release, though the lineage doesn't, and a test should compare what events say, as this example's summary does.
A production emitter stamps COMPLETE with the time the run finished. A derived run ID is still worth keeping there, made from the scheduled time, the job's name and the attempt number: an event sent twice then lands in the same run instead of creating a second one, and a retried job gets a run of its own. Production events should also carry the output's Iceberg snapshot ID, in the version dataset facet the object model names for it. The code here leaves it out, because PyIceberg 0.12.0 draws each snapshot ID from a random UUID (_generate_snapshot_id in its pyiceberg/table/metadata.py).
Working rule: Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run can happen without them, and name datasets by catalog identifiers under a namespace you choose, never a path. In tests, use the file transport with a fixed event time and static run IDs, and compare what events say, never their bytes. In production, send through a transport to your lineage server, and derive each run ID from the scheduled time, the job and the attempt, so a resent event never creates a duplicate run.
将每个输出列追溯到其来源
运行事件说明了一个作业读取和写入了哪些表。列级血缘则说明每个输出列来自哪些输入列,而这正是治理所需要的:一张读取 crm.learners 的表,可能复制了其中的电子邮件,也可能没有。OpenLineage 把列级血缘放在输出数据集的一个数据集 facet columnLineage 中:fields 将每个输出列映射到它的 inputFields,另有一个可选的 dataset 列表,用于存放影响整个数据集的输入(facet 模式)。由于客户端不计算其中任何内容,就需要有工具来解析每个作业的 SQL。
SQL 解析器 sqlglot 用 sqlglot.lineage.lineage 完成这项工作。给定一条语句和它所读取的表的模式,它会为每个输出列返回一棵树,树的叶子是源列;使用 column=None 时,它会一次返回所有输出列(sqlglot lineage)。它需要模式。没有模式时,SELECT * 只会返回一个名为 * 的输出列,而连接中未限定表名的列会变成没有表的占位符。lib.lineage.schema 根据目录中的 Iceberg 元数据构建模式,因此解析器看到的是仓库中实际存在的列,所用的 sqlglot 版本为 30.33.0。
先看 marketing_audience,它为营销部门可以发送邮件的名单提供数据。用文字描述,它挑选同意接收营销邮件且账户处于活跃状态的学习者,并为每人加上其最喜欢的课程类别,该类别由一个 CTE 按学习者的报名记录对类别排序得出。向 sqlglot 查询该流水线各输出的血缘,以及两个 course_performance 列的血缘,看看它遗漏了什么:
1"""What sqlglot.lineage says the pipeline outputs come from, and what it leaves out. The prose rests on the CTE-traced2source of favourite_category, on the sources of the two course performance columns, and on the assertion that neither3column the WHERE reads reaches any output."""4import sqlglot5from sqlglot import exp67from lib import lineage, pipelines8from lib.catalog import open_catalog910# sqlglot resolves each name against the catalog's columns, so lineage.schema builds them from the table metadata.11schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",12 "billing.payments"]))13statement = pipelines.sql("marketing_audience")14found = lineage.sqlglot_sources(statement, schema)15for column, sources in found.items():16 print(f"marketing.audience.{column} comes from:", ", ".join(sources))1718parsed = sqlglot.parse_one(statement, read="duckdb")19# the WHERE names columns by alias, as in l.status, so each alias maps to its table20tables = {table.alias_or_name: f"{table.db}.{table.name}" for table in parsed.find_all(exp.Table)}21where = parsed.find(exp.Where)22print("the pipeline's WHERE:", where.this.sql(dialect="duckdb"))23in_lineage = {source for sources in found.values() for source in sources} # every source column of every output24for column in where.find_all(exp.Column):25 source = f"{tables[column.table]}.{column.name}"26 assert source not in in_lineage # every column the WHERE reads, checked against the lineage above27 print(f"{source} in any column's lineage:", source in in_lineage)2829course_performance = lineage.sqlglot_sources(pipelines.sql("course_performance"), schema)30for column in ("completions", "revenue_usd"): # the two aggregates that carry a FILTER inside them31 print(f"analytics.course_performance.{column} comes from:", ", ".join(course_performance[column]))
python sqlglot-misses-row-inputs.pymarketing.audience.learner_id comes from: crm.learners.learner_id
marketing.audience.full_name comes from: crm.learners.full_name
marketing.audience.email comes from: crm.learners.email
marketing.audience.country comes from: crm.learners.country
marketing.audience.favourite_category comes from: learning.courses.category
the pipeline's WHERE: l.marketing_consent AND l.status = 'active'
crm.learners.marketing_consent in any column's lineage: False
crm.learners.status in any column's lineage: False
analytics.course_performance.completions comes from: learning.enrollments.completed_at, learning.enrollments.enrollment_id
analytics.course_performance.revenue_usd comes from: billing.payments.amount, billing.payments.status
这些值是正确的,但决定哪些学习者进入该表的条件却缺失了。email 来自 crm.learners.email,favourite_category 则经由 CTE 追溯到 learning.courses.category。流水线的 WHERE 是 l.marketing_consent AND l.status = 'active',而它涉及的两列 marketing_consent 和 status 都不在任何输出的血缘中。该示例对这两列都做了断言。同意接收营销邮件的过滤条件,正是营销部门可以持有这张表的原因,而血缘中对此只字未提。来自报名表和课程表的连接键同样没有出现。
第二个问题恰好相反:只起过滤作用的输入被当作值列了出来。course_performance 只对成功的付款求和,使用的是 filter (where p.status = 'succeeded'),而 sqlglot 称 revenue_usd 来自 billing.payments.amount, billing.payments.status:被求和的列和仅决定哪些付款计入的列被同等列出。completions 用 filter (where e.completed_at is not null) 统计报名数,返回的结果也是如此,来自 learning.enrollments.completed_at, learning.enrollments.enrollment_id。这两个问题都源于 sqlglot 构建树的方式。它收集每个输出表达式中的列,因此能看到表达式内部的过滤,看不到表达式之外的过滤,而且它的节点不带类型,无法说明某个输入是如何被使用的。
工作准则: 用 sqlglot 的 lineage 获取每个输出的值所依据的列,并且始终传入目录的模式,该模式由表元数据生成,绝不手工输入。不要把它的结果当作某一列所依赖的全部内容:它会遗漏用于连接、过滤和分组的列,也不会说明它的哪些输入是值。
这部分尚未翻译,因此以英文显示。
Add the inputs that decide which rows exist
sqlglot's lineage lists the columns a value is computed from, and OpenLineage has a place for the rest. Each input field carries a list of transformations, each with a type, a subtype and a masking flag (column lineage facet). DIRECT means the output value is derived from the input value, through IDENTITY, TRANSFORMATION or AGGREGATION. INDIRECT means the output value is impacted by an input without being derived from it, through JOIN, GROUP_BY, FILTER, SORT, WINDOW or CONDITIONAL. Inputs that shape every row, such as a WHERE, belong in the facet's dataset list.
For governance, the INDIRECT side matters, since information flows through conditions as well as values. Denning calls the information that flows through conditions "implicit flow": a statement that assigns only when a condition holds still tells you the condition's value (Denning, 1976). A row in marketing.audience tells you that learner consented.
lib.lineage.column_lineage adds two things sqlglot's answer lacks: a type for each value input, and the INDIRECT inputs. It types each value input by where the column sits in the output's expression: a bare column is IDENTITY, a column inside an aggregate is AGGREGATION, and anything else is TRANSFORMATION. A column inside an aggregate's FILTER, a window's PARTITION BY or ORDER BY, or a CASE's condition or operand is INDIRECT, and a hash on the way sets masking. COALESCE's arguments count as values: OpenLineage lists COALESCE under CONDITIONAL, but any of its arguments can become the output's value. Then it walks every SELECT in the statement, CTEs included, and collects the columns of its row clauses, following CTE columns back to the tables they come from:
ROW_CLAUSES = {"joins": "JOIN", "where": "FILTER", "group": "GROUP_BY", "having": "FILTER", "qualify": "FILTER",
"order": "SORT"}
Run it on the same pipeline and the same two columns:
1"""The book's column lineage for the marketing audience pipeline and two course performance columns: each input,2with its OpenLineage type and subtype, then the inputs that shape every row. The prose rests on the source counts."""3from lib import lineage, pipelines4from lib.catalog import open_catalog56schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",7 "billing.payments"]))8# rows holds the inputs that shape every row, which OpenLineage keeps in the dataset list9fields, rows = lineage.column_lineage(pipelines.sql("marketing_audience"), schema)10for column, inputs in fields.items():11 print(f"marketing.audience.{column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"12 for dependency in inputs))13print("inputs that shape every row of marketing.audience:")14for source in sorted({dependency.source for dependency in rows}):15 print(f" {source}:", "INDIRECT",16 ", ".join(dependency.subtype for dependency in rows if dependency.source == source))1718# the source columns that the output values are computed from19values = {dependency.source for inputs in fields.values() for dependency in inputs}20print("source columns, values only:", len(values))21# A column can both give a value and shape the rows, so the union counts it once.22print("source columns, with the rows' inputs:", len(values | {dependency.source for dependency in rows}))2324fields, _ = lineage.column_lineage(pipelines.sql("course_performance"), schema) # only these two columns are printed25for column in ("completions", "revenue_usd"):26 print(f"analytics.course_performance.{column}:",27 ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}" for dependency in fields[column]))
python indirect-column-lineage.pymarketing.audience.learner_id: crm.learners.learner_id DIRECT IDENTITY
marketing.audience.full_name: crm.learners.full_name DIRECT IDENTITY
marketing.audience.email: crm.learners.email DIRECT IDENTITY
marketing.audience.country: crm.learners.country DIRECT IDENTITY
marketing.audience.favourite_category: learning.courses.category DIRECT IDENTITY
inputs that shape every row of marketing.audience:
crm.learners.learner_id: INDIRECT JOIN
crm.learners.marketing_consent: INDIRECT FILTER
crm.learners.status: INDIRECT FILTER
learning.courses.category: INDIRECT GROUP_BY, JOIN
learning.courses.course_id: INDIRECT JOIN
learning.enrollments.course_id: INDIRECT JOIN
learning.enrollments.learner_id: INDIRECT GROUP_BY, JOIN
source columns, values only: 5
source columns, with the rows' inputs: 10
analytics.course_performance.completions: learning.enrollments.completed_at INDIRECT FILTER, learning.enrollments.enrollment_id DIRECT AGGREGATION
analytics.course_performance.revenue_usd: billing.payments.amount DIRECT AGGREGATION, billing.payments.status INDIRECT FILTER
The values match sqlglot's, each an IDENTITY copy, and the rows' inputs are the new part. Adding them grows what marketing.audience depends on from 5 source columns to 10. Some row inputs are columns no value uses: marketing_consent and status as FILTER, the two course IDs as JOIN, and the enrollments' learner_id as GROUP_BY and JOIN. crm.learners.learner_id and learning.courses.category, which values copy too, also shape rows: the ID as JOIN, the category as GROUP_BY and JOIN. For revenue_usd, the two inputs are now told apart: amount is a DIRECT AGGREGATION, and status an INDIRECT FILTER. So are the two inputs of completions: enrollment_id is counted, and completed_at only filters.
One more detail: the per-field transformationType and transformationDescription are deprecated, replaced by the per-input transformations that lib.lineage writes (changelog, 1.17.1). And OpenLineage 1.53.0 added a Lineage dataset facet for declared lineage, such as a view's, sent on a DatasetEvent, which consumers should prefer where both describe the same relationship (lineage facet). Run events from jobs carry columnLineage, as the platform's do.
Working rule: Record INDIRECT inputs with every job's column lineage: the join keys, filters and grouping columns of every SELECT, CTEs included, in the facet's dataset list, and type each value input with OpenLineage's subtypes. Expect more edges, many of them join keys. Keep them anyway: a filter on a consent column decides who may be emailed, and it reaches the table only as an INDIRECT input.
这部分尚未翻译,因此以英文显示。
Fail on statements the parser can't read
The platform's column lineage is only as good as the parse behind it, and a parse can be wrong without failing. sqlglot's lineage accepts more than a SELECT: it finds the query inside a CREATE TABLE ... AS, an INSERT ... SELECT or a CREATE VIEW, and it reads a MERGE too (sqlglot, optimizer/scope.py). Three cases come back looking fine. An INSERT's outputs are named after its SELECT, never after the table's columns. A MERGE comes back as the last query sqlglot reaches in it, as if that query were what the MERGE writes. When USING names a table or a CTE and no subquery follows, sqlglot returns no lineage and no error. When USING is a subquery, or a subquery sits in a WHEN condition or a SET, it returns the columns of the last such subquery, whether the MERGE writes them or not. And a misspelt column gets through when it's unqualified in a query on one table: sqlglot assigns it to that table whether the table has the column or not.
All four of the platform's pipelines are CREATE TABLE ... AS statements. As tables grow, pipelines like these often become incremental: an INSERT that appends a day's revenue to analytics.daily_revenue, or a MERGE that updates analytics.learner_activity in place. Ask sqlglot and lib.lineage.column_lineage about each, the INSERT and the MERGE in two forms, and compare the answers:
1"""sqlglot.lineage's answers for an INSERT and for two MERGE statements, beside the book's lineage: sqlglot names the2INSERT's outputs after its SELECT, and returns nothing or a subquery's columns for a MERGE, which the book refuses. The3prose rests on the sqlglot outputs and on the refusals at the end."""4import pyarrow as pa5from sqlglot.lineage import lineage as sqlglot_lineage67from lib import lineage8from lib.catalog import open_catalog910catalog = open_catalog(["billing.payments", "crm.learners"])11catalog.create_namespace("analytics") # the incremental table, as it exists before the day's INSERT12catalog.create_table("analytics.daily_revenue", schema=pa.schema([("day", pa.date32()), ("revenue_usd", pa.float64())]))13schema = lineage.schema(catalog)14insert = """insert into analytics.daily_revenue15select cast(p.paid_at as date), sum(p.amount) from billing.payments p where p.status = 'succeeded' group by 1"""16merges = {"USING a table": """merge into analytics.learner_activity a using crm.learners l17on a.learner_id = l.learner_id when matched then update set country = l.country""",18 "USING a subquery": """merge into analytics.learner_activity a19using (select l.learner_id, l.country, l.email from crm.learners l) s on a.learner_id = s.learner_id20when not matched then insert (learner_id, country) values (s.learner_id, s.country)"""}2122print("analytics.daily_revenue columns:", ", ".join(schema["analytics"]["daily_revenue"]))23# The SELECT's expressions have no aliases, so sqlglot names each output after the SELECT, not the table. None asks for24# every output column at once.25print("sqlglot, INSERT outputs:", ", ".join(sqlglot_lineage(None, insert, schema, dialect="duckdb")))26fields, rows = lineage.column_lineage(insert, schema)27for column, inputs in fields.items():28 print(f"book, INSERT {column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"29 for dependency in inputs))30print("book, INSERT rows shaped by:", ", ".join(f"{dependency.source} {dependency.subtype}" for dependency in rows))31for label, merge in merges.items():32 outputs = sqlglot_lineage(None, merge, schema, dialect="duckdb")33 print(f"sqlglot, MERGE {label}, outputs:", ", ".join(outputs) or "none")3435# One value too many for daily_revenue's two columns, caught after sqlglot parses it; each MERGE is refused before sqlglot is asked.36refused = {"INSERT with one value too many": insert.replace("sum(p.amount)", "sum(p.amount), count(*)"),37 **{f"MERGE {label}": merge for label, merge in merges.items()}}38for label, statement in refused.items():39 try:40 lineage.column_lineage(statement, schema)41 except lineage.LineageError as error:42 print(f"book, {label}:", error)
python insert-and-merge-lineage.pyanalytics.daily_revenue columns: day, revenue_usd
sqlglot, INSERT outputs: paid_at, _col_1
book, INSERT day: billing.payments.paid_at DIRECT TRANSFORMATION
book, INSERT revenue_usd: billing.payments.amount DIRECT AGGREGATION
book, INSERT rows shaped by: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER
sqlglot, MERGE USING a table, outputs: none
sqlglot, MERGE USING a subquery, outputs: learner_id, country, email
book, INSERT with one value too many: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values
book, MERGE USING a table: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
book, MERGE USING a subquery: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
The table analytics.daily_revenue has the columns day, revenue_usd, yet sqlglot names the INSERT's outputs paid_at, _col_1: the first after the column inside the cast, the second a generated name for an expression with no alias.
An INSERT fills columns by position, so lib.lineage matches by position too, taking the names from the catalog: the columns the INSERT lists, or else all of the table's. So day comes from billing.payments.paid_at DIRECT TRANSFORMATION, and the INSERT's grouping and its filter on status shape every row: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER. When the SELECT has more or fewer values than the columns written, lib.lineage raises: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values.
For the MERGE whose USING names a table, sqlglot returns none, and no error. For the one whose USING is a subquery, it returns learner_id, country, email, though that MERGE writes only learner_id and country. A check for empty lineage catches the first and passes the second, so lib.lineage doesn't judge what sqlglot returns for a MERGE. Before it calls sqlglot, it refuses every statement that isn't a CREATE ... AS or an INSERT ... SELECT: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE.
A misspelt column, the third case, depends on how the column is written. The platform's pipelines qualify every column with its table's alias, as in l.email, and sqlglot checks a qualified column against the schema itself:
1"""A misspelt column, emial for email, in a query on one table: sqlglot raises when the column is qualified by its2alias, and traces it to the table when it isn't. The prose rests on the sqlglot answer for each form, and on the book's3check against the schema."""4from sqlglot.errors import OptimizeError56from lib import lineage7from lib.catalog import open_catalog89schema = lineage.schema(open_catalog(["crm.learners"])) # the schema lets sqlglot check each column name10# the alias l is right, and only the column name is misspelt11qualified = "create table marketing.audience as select l.learner_id, l.emial from crm.learners l"12# no alias, so nothing names the table the column comes from13unqualified = "create table marketing.audience as select learner_id, emial from crm.learners"1415try:16 lineage.sqlglot_sources(qualified, schema)17except OptimizeError as error:18 print("sqlglot, qualified l.emial:", f"{type(error).__name__}: {error}")19print("sqlglot, unqualified emial comes from:", ", ".join(lineage.sqlglot_sources(unqualified, schema)["emial"]))20try:21 lineage.column_lineage(unqualified, schema)22except lineage.LineageError as error:23 print("book, unqualified emial:", error)
python misspelt-column-lineage.pysqlglot, qualified l.emial: OptimizeError: Unknown column: emial
sqlglot, unqualified emial comes from: crm.learners.emial
book, unqualified emial: crm.learners has no column emial
A qualified typo stops sqlglot with OptimizeError: Unknown column: emial. An unqualified one, in a query on one table, comes back as crm.learners.emial: a confident answer about a column that doesn't exist. In a join, a misspelt unqualified column belongs to none of the tables, and sqlglot returns a placeholder with no table (sqlglot, lineage.py), which lib.lineage rejects as a source it can't place. For the single-table case, lib.lineage checks every source against the schema and raises crm.learners has no column emial.
lib.lineage.record, which run uses, reads the lineage before the job runs, so a pipeline whose lineage can't be read fails before it writes anything. That is strict, and deliberate: a job with empty lineage would vanish from every impact report and classification check in this chapter, and one with wrong lineage would mislead them. Run the same check in CI on every pull request that changes a pipeline, so the failure shows up in review rather than at night.
Working rule: Treat a statement your parser can't read correctly as an error, never as empty or partial lineage. Refuse every kind of statement it gets wrong, such as MERGE, before you ask it, and fail on empty lineage, a source the parser can't place and a column the schema doesn't have. Match an INSERT's columns by position against the table's columns in the catalog, never the SELECT's names, and fail when the counts differ. For a MERGE, write a SELECT of the columns it writes, and record the job's lineage from that, as an INSERT ... SELECT into the MERGE's table, until your parser handles MERGE itself.
找出列变更会破坏什么
有了每个 COMPLETE 事件中的列级血缘,平台就得到了一张图:列是节点,每个输入字段是一条边,指向它所供给的列或表。影响分析要回答的问题是:对某一列的变更会破坏什么。它从该列出发,沿着从它延伸出去的每一条边前进,再从每条边的落点继续,深度不限。这里不需要任何特殊方法,它就是图的可达性问题。只有一条规则是血缘所特有的:指向某个表的 INDIRECT 边意味着该表的每一行都可能改变,因此遍历要从该表的所有列继续进行。
lib.lineage.graph 仅依据事件文件构建这张图,就像血缘服务器那样,而不是依据 SQL。因此,凡是发送事件的作业,无论用什么语言编写,图中都包含在内;不发送事件的作业则一概不含。lib.lineage.downstream 负责执行遍历。
设想平台团队可能做出的三项变更。CRM 团队把每个电子邮件地址转为小写,或者重命名 crm.learners.email。新的注册流程改变了 crm.learners.marketing_consent 对旧账户的含义。学习团队重命名 learning.courses 中的某个类别。先猜一猜:同意状态的变更可以波及多少列?
1"""What a change to one source column can break, walked over the column graph that the nightly run's events describe.2The prose rests on the reach counts printed for each column: through values, and in all."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch78events_file = scratch() / "events.jsonl"9lineage.run_all(open_catalog(), events_file)10graph = lineage.graph(lineage.read_events(events_file)) # built from the events alone, as a lineage server would11print("edges in the column graph:", len(graph.edges))1213for column in ("crm.learners.email", "crm.learners.marketing_consent", "learning.courses.category"):14 reached = lineage.downstream(graph, column)15 print(f"{column} reaches:")16 # dict.fromkeys drops repeated targets and keeps the order they were found in17 for target in dict.fromkeys(edge.target for edge in reached):18 how = ", ".join(f"{edge.type} {edge.subtype}" for edge in reached if edge.target == target)19 # a table's name has one dot and a column's two, so a table target means every row of it20 print(f" {target}{', every row' if target.count('.') == 1 else ''}:", how)21 values = [edge for edge in reached if edge.type == "DIRECT"] # only DIRECT edges copy values22 print(f"{column}, columns reached through values:", len(lineage.columns_reached(graph, values)))23 print(f"{column}, columns reached in all:", len(lineage.columns_reached(graph, reached)))
python column-impact.pyedges in the column graph: 51
crm.learners.email reaches:
marketing.audience.email: DIRECT IDENTITY
crm.learners.email, columns reached through values: 1
crm.learners.email, columns reached in all: 1
crm.learners.marketing_consent reaches:
marketing.audience, every row: INDIRECT FILTER
crm.learners.marketing_consent, columns reached through values: 0
crm.learners.marketing_consent, columns reached in all: 5
learning.courses.category reaches:
analytics.course_performance, every row: INDIRECT GROUP_BY
analytics.course_performance.category: DIRECT IDENTITY
marketing.audience, every row: INDIRECT GROUP_BY, INDIRECT JOIN
marketing.audience.favourite_category: DIRECT IDENTITY
learning.courses.category, columns reached through values: 2
learning.courses.category, columns reached in all: 12
同意状态的结果出人意料。对它的变更通过值波及 0 列,但通过其过滤条件,它波及 marketing.audience 的全部 5 列,因为它改变了谁会收到邮件。只看值的血缘工具会报告,同意状态的变更没有任何影响。
对 email 的变更波及 1 列,即 marketing.audience.email,因此重命名它会破坏营销流水线,而不会破坏其他任何发送血缘的作业。类别重命名通过值波及 2 列,总共波及 12 列,因为类别还用于对 analytics.course_performance 分组,并为每位学习者的最爱排名。
同样的遍历还服务于后面的两章:第 7 章沿下游追踪学习者的列,找出删除请求必须覆盖的每一张衍生表;第 10 章为每个修改模式的拉取请求撰写影响报告。
**工作准则:**在由事件构建的图上运行影响分析,包含 INDIRECT 边,并报告两份清单:值可能改变的列,以及行可能改变的表。对于仅用于过滤和连接的列,其变更应视为对其所过滤的各表的每一列的变更,并把报告发送给它所波及的每张表的负责人。
这部分尚未翻译,因此以英文显示。
Propose classifications from lineage
A derived column holds data from its inputs, so its classification can be derived from theirs. The classic rule comes from Denning's lattice model of information flow: the class of a value computed from several inputs is the least upper bound, the "join", of their classes (Denning, 1976). The platform's classifications form a line, public, internal, personal and sensitive, so the join is the stricter of them, which stricter in lib.metadata, the book's metadata module, returns. Which inputs count is a judgement. We count INDIRECT inputs too, following Denning's implicit flows, because a table filtered on consent reveals each learner's consent. That choice lives in governance/lineage.toml, so a team can change it in one reviewed line.
Two more choices follow from the same caution. OpenLineage's masking flag says only that a value was obfuscated, such as by a hash (column lineage facet). Whether a hashed identifier is still personal data is a legal question: chapter 2 quotes the GDPR recital on pseudonymized data and leaves the answer to lawyers. So the book's rule never lowers a classification because of masking. Catalog servers leave the choice to configuration too. Apache Atlas propagates classifications along lineage, and can block that on an edge, for example where a view masks a PII column (Atlas). DataHub propagates glossary terms along column lineage, a feature its 1.7.0 documentation calls a public beta in DataHub Cloud (DataHub). A proposal is a floor, not a verdict: like the drafts a local model writes in chapter 2, it's reviewed before it goes into governance/metadata.toml, the platform's file of owners and classifications.
lib.lineage.propose proposes a classification for every column a job wrote, and lib.metadata.problems lists the tables that need one. Check the rule on the two tables people did classify, analytics.course_performance and analytics.learner_activity, before you trust it on marketing.audience and ai.review_corpus, the two with no metadata:
1"""Proposes classifications from lineage for the columns with no declared one, then checks the rule against the columns2people did classify. The prose rests on the personal count, on the reproduced counts, and on the check that no declared class is lower."""34from lib import lineage, metadata5from lib.catalog import open_catalog6from lib.platform import scratch78catalog = open_catalog()9events_file = scratch() / "events.jsonl"10lineage.run_all(catalog, events_file)11for problem in metadata.problems(catalog, metadata.load()):12 print("metadata problem:", problem)1314graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared() # known: governance/metadata.toml15print("propagated through:", " and ".join(lineage.rules()["propagate_through"])) # from governance/lineage.toml16proposed = lineage.propose(graph, known)17for column, (classification, decided_by) in proposed.items():18 if column not in known:19 first = decided_by[0] # lib sorts the deciding edges DIRECT first, so this is a value when one decides20 print(f"{column}: {classification}, from {first.source} ({first.type} {first.subtype})")21print("personal columns proposed:", sum(1 for column, (classification, _) in proposed.items()22 if column not in known and classification == "personal"))2324classified = [column for column in proposed if column in known]25print("declared classifications checked:", len(classified))26print("classes declared for them:", ", ".join(sorted({known[column] for column in classified})))27for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):28 proposals = lineage.propose(graph, known, through)29 agree = sum(1 for column in classified if proposals[column][0] == known[column])30 print(f"declared classifications reproduced, through {' and '.join(through)}:", f"{agree} of {len(classified)}")31# proposals is the DIRECT-only run from the loop above32missed = [column for column in classified if proposals[column][0] != known[column]]33print("missed with DIRECT inputs only:", len(missed))34for column in missed:35 print(f" {column}: declared {known[column]}, DIRECT inputs propose {proposals[column][0]}")36print("declared less protected than lineage proposes:", len(lineage.understated(proposed, known)))37assert not lineage.understated(proposed, known)
python propagate-classifications.pymetadata problem: ai.review_corpus: no metadata
metadata problem: marketing.audience: no metadata
propagated through: DIRECT and INDIRECT
marketing.audience.learner_id: internal, from crm.learners.learner_id (DIRECT IDENTITY)
marketing.audience.full_name: personal, from crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email: personal, from crm.learners.email (DIRECT IDENTITY)
marketing.audience.country: internal, from crm.learners.country (DIRECT IDENTITY)
marketing.audience.favourite_category: internal, from crm.learners.learner_id (INDIRECT JOIN)
ai.review_corpus.review_id: internal, from learning.reviews.review_id (DIRECT IDENTITY)
ai.review_corpus.course_id: internal, from learning.reviews.course_id (DIRECT IDENTITY)
ai.review_corpus.course_title: internal, from learning.reviews.course_id (INDIRECT JOIN)
ai.review_corpus.rating: internal, from learning.reviews.rating (DIRECT IDENTITY)
ai.review_corpus.body: personal, from learning.reviews.body (DIRECT IDENTITY)
personal columns proposed: 3
declared classifications checked: 13
classes declared for them: internal
declared classifications reproduced, through DIRECT and INDIRECT: 13 of 13
declared classifications reproduced, through DIRECT: 10 of 13
missed with DIRECT inputs only: 3
analytics.course_performance.course_id: declared internal, DIRECT inputs propose public
analytics.course_performance.title: declared internal, DIRECT inputs propose public
analytics.course_performance.category: declared internal, DIRECT inputs propose public
declared less protected than lineage proposes: 0
The metadata check from chapter 2 still reports the two tables with no metadata, and lineage proposes a classification for each of their columns. It proposes 3 personal columns: full_name and email in marketing.audience, each an IDENTITY copy of a personal column in crm.learners, and body in ai.review_corpus, a copy of the reviews' free text. The rest come out internal, including favourite_category, for which the example prints internal, from crm.learners.learner_id (INDIRECT JOIN). Its values are public course-catalog category names, but the rows they sit in are chosen by a join on learner IDs and a consent filter, so those are internal.
Counting INDIRECT inputs reproduces the declared classifications on the columns people did classify, 13 of 13, and all of them are internal.
DIRECT inputs alone reproduce 10 of 13, and miss 3: course_id, title and category of analytics.course_performance. A DIRECT-only rule calls them public, as in the course catalog, while the analytics team chose internal. So the rule proposes no less than people chose here, and lineage finds no declared classification lower than its proposal. That is weak evidence: a rule that proposed internal for every column would score as well. And it can't show what the rule does with a personal filter, since none of the platform's filters or joins reads a personal column: even marketing_consent is classified internal.
So build a table whose filter does. For a birthday offer, marketing wants the learners who consented and were born this month, with their ID and country, and nothing else:
1"""A birthday-offer table whose only personal input is a filter on date of birth. The prose rests on the learner count,2and on the proposals counting DIRECT inputs alone and counting DIRECT and INDIRECT inputs."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch7from lib.warehouse import connect89catalog = open_catalog(["crm.learners"])10# The consenting learners born in the month of the nightly run, on the book's "today".11select = ("select l.learner_id, l.country from crm.learners l "12 f"where l.marketing_consent and month(l.date_of_birth) = {lineage.NIGHTLY.month}")131415def offers() -> None:16 rows = connect(catalog).sql(select).to_arrow_table()17 catalog.create_namespace_if_not_exists("marketing")18 catalog.create_table("marketing.birthday_offers", schema=rows.schema).append(rows)192021events_file = scratch() / "events.jsonl"22# record derives the lineage from this statement, and reads it before offers() runs23lineage.record(lineage.client(events_file), "birthday_offers", f"create table marketing.birthday_offers as {select}",24 catalog, offers)25offers_table = catalog.load_table("marketing.birthday_offers").scan().to_arrow()26print("learners in marketing.birthday_offers:", f"{len(offers_table):,}")2728graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()29print("crm.learners.date_of_birth declared:", known["crm.learners.date_of_birth"])30for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):31 for column, (classification, decided_by) in lineage.propose(graph, known, through).items():32 label = f"{column}, through {' and '.join(through)}"33 print(f"{label}:", classification)34 print(f"{label}, decided by:", f"{decided_by[0].source} ({decided_by[0].type} {decided_by[0].subtype})")
python personal-filter-propagates.pylearners in marketing.birthday_offers: 165
crm.learners.date_of_birth declared: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.country, through DIRECT and INDIRECT: personal
marketing.birthday_offers.country, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.learner_id, through DIRECT: internal
marketing.birthday_offers.learner_id, through DIRECT, decided by: crm.learners.learner_id (DIRECT IDENTITY)
marketing.birthday_offers.country, through DIRECT: internal
marketing.birthday_offers.country, through DIRECT, decided by: crm.learners.country (DIRECT IDENTITY)
Neither column copies anything personal, but every row says its learner was born in the offer's month, part of a date of birth. The table holds 165 learners, and governance/metadata.toml classifies the date of birth personal. Counting DIRECT inputs alone proposes internal for learner_id and internal for country.
Add INDIRECT inputs, and it proposes personal for learner_id and personal for country, both decided by crm.learners.date_of_birth (INDIRECT FILTER). That is the case INDIRECT inputs are for, and it shows the cost too: one personal filter makes every column of a table personal.
OpenLineage can carry classifications in the events themselves, with the Tags dataset facet, whose field key "identifies the field in a dataset if a tag applies to one" (tags facet, schema). We keep them in governance/metadata.toml instead, the one source that the platform's access policies also read, through lib.policy, the book's policy module (chapter 5).
Working rule: Propose each derived column's classification as the stricter of its inputs', counting INDIRECT inputs, and send the proposals to the table's owner for review before they become metadata. Never let lineage lower a classification someone declared, and never treat masking as declassification. Count DIRECT inputs only if your pipelines often join or filter on personal columns, such as matching on email, and every output would otherwise come out personal; then review INDIRECT personal inputs by hand.
生产环境中的常见故障
血缘关系的失效是无声的:缺失的一条边,看起来与没有任何依赖的列毫无区别,于是所有基于该图的检查,从影响分析到建议的分类,都会照常通过。以下三种故障较为常见,每一种都有对应的检查方法。
这部分尚未翻译,因此以英文显示。
Events that never arrive
The client reads OPENLINEAGE_DISABLED from the environment, and when it's true, events are dropped (configuration). In the client's source, that check comes first (openlineage-python, client/client.py): before it looks at the transport you passed explicitly, the client picks a transport that does nothing. Its only notice is an INFO log line: "OpenLineage is disabled. No events will be emitted." Python's root logger is set to WARNING by default, so the line never shows (logging HOWTO). A variable left in a shared environment file after debugging is enough. Set the variable, run review_corpus with an explicit file transport, and check what reaches the file:
1"""With OPENLINEAGE_DISABLED=true in the environment, an explicit file transport writes nothing and raises nothing. The2prose rests on the transport the client chose and on the job the check finds without a COMPLETE event."""3import os45from lib import lineage6from lib.catalog import open_catalog7from lib.platform import scratch89os.environ["OPENLINEAGE_DISABLED"] = "true" # say, left in a shared environment file after a debugging session1011events_file = scratch() / "events.jsonl"12client = lineage.client(events_file) # a file transport, passed in explicitly13catalog = open_catalog(["learning.courses", "learning.reviews"])14output = lineage.run(catalog, "review_corpus", client)1516print("transport the client chose:", client.transport.kind) # OPENLINEAGE_DISABLED wins over the transport passed in17print("pipeline wrote:", output)18print("rows written:", f"{len(catalog.load_table(output).scan().to_arrow()):,}")19assert not events_file.exists() # the noop transport never creates the file20print("events file exists:", events_file.exists())21missing = lineage.missing_runs(["review_corpus"], lineage.read_events(events_file)) # ran, but sent no COMPLETE22print("jobs that ran with no COMPLETE event:", ", ".join(missing))
python lineage-disabled-silently.pytransport the client chose: noop
pipeline wrote: ai.review_corpus
rows written: 3,000
events file exists: False
jobs that ran with no COMPLETE event: review_corpus
With the variable set, the pipeline still runs, but nothing reaches the file. The client chose the noop transport over the file transport the code passed in, and the pipeline wrote 3,000 rows to ai.review_corpus without an error and without an events file. The check that catches it compares the jobs the scheduler ran with the jobs that sent a COMPLETE event: lib.lineage.missing_runs lists review_corpus as having run without one.
Working rule: After every scheduled run, compare the jobs that ran with the jobs whose COMPLETE events arrived, and alert on any difference. Don't rely on the emitter to report its own silence.
无人接入埋点的作业
血缘只覆盖会发送血缘事件的作业。平台的市场团队每周运行一个 Python 脚本,把 marketing.audience 复制到 marketing.campaign_list,后者是其邮件工具读取的一张表。没有人把这个脚本接入 OpenLineage:
1"""A Python step copies marketing.audience for the email tool and sends no lineage. The prose rests on the walk from the2email column, which misses the copy, and on the catalog check against the lineage, which finds it."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch7from lib.warehouse import connect89catalog = open_catalog()10events_file = scratch() / "events.jsonl"11lineage.run_all(catalog, events_file)1213# The marketing team's weekly export, written straight to the catalog: no lineage event.14export = connect(catalog).sql("select learner_id, full_name, email from marketing.audience").to_arrow_table()15catalog.create_table("marketing.campaign_list", schema=export.schema).append(export)16print("rows exported:", f"{len(export):,}")1718events = lineage.read_events(events_file)19reached = lineage.downstream(lineage.graph(events), "crm.learners.email")20print("crm.learners.email reaches:", ", ".join(edge.target for edge in reached))21# the catalog's tables, less the platform's sources and the outputs the events name22print("tables no job's lineage accounts for:", ", ".join(lineage.unaccounted(catalog, events)))
python uninstrumented-job.pyrows exported: 1,829
crm.learners.email reaches: marketing.audience.email
tables no job's lineage accounts for: marketing.campaign_list
不发送事件的导出作业,在影响分析中是不可见的。该脚本复制了 1,829 名学习者的姓名和电子邮件地址,而 crm.learners.email 仍然只流向 marketing.audience.email,因此依据这份血缘执行的删除会遗漏这份副本。不过,目录知道这张表的存在,所以 lib.lineage.unaccounted 会把目录中的表与血缘进行比对:每张表都必须是平台的数据源之一,或是血缘所知某个作业的输出。它找到的结果是 marketing.campaign_list。
工作准则: 按计划定期用血缘核对目录,并把没有任何作业的血缘能够解释的表视为事件:找出对应的作业,并让它发送事件。
这部分尚未翻译,因此以英文显示。
Personal data in a table classified internal
Instrumenting the marketing team's export fixes the graph, and exposes a second problem. When the export's owner created marketing.campaign_list, they classified every column internal, because only the marketing team reads it. That's a statement about who reads a table, not what it holds. Wrap the script's work in lib.lineage.record, with the SELECT it runs as its lineage, and compare the declared classifications with lineage's proposal:
1"""The same export, now recorded with lineage, into a table its owner tagged internal. The prose rests on the walk to2the copy and on the check for understated classifications."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch7from lib.warehouse import connect89catalog = open_catalog()10events_file = scratch() / "events.jsonl"11lineage.run_all(catalog, events_file)1213select = "select a.learner_id, a.full_name, a.email from marketing.audience a"141516def export() -> None:17 rows = connect(catalog).sql(select).to_arrow_table()18 catalog.create_table("marketing.campaign_list", schema=rows.schema).append(rows)192021# record reads this statement's lineage before export() runs22statement = f"create table marketing.campaign_list as {select}"23lineage.record(lineage.client(events_file), "campaign_export", statement, catalog, export)2425# Its owner tagged every column internal: "only the marketing team reads it".26known = lineage.declared() | {f"marketing.campaign_list.{c}": "internal" for c in ("learner_id", "full_name", "email")}27graph = lineage.graph(lineage.read_events(events_file))28print("crm.learners.email reaches:", ", ".join(edge.target for edge in lineage.downstream(graph, "crm.learners.email")))29for problem in lineage.understated(lineage.propose(graph, known), known):30 print("understated:", problem)
python personal-copy-tagged-internal.pycrm.learners.email reaches: marketing.audience.email, marketing.campaign_list.email
understated: marketing.campaign_list.full_name: declared internal, lineage proposes personal
understated: marketing.campaign_list.email: declared internal, lineage proposes personal
Once the export sends lineage, the walk reaches its copy: crm.learners.email reaches marketing.audience.email, marketing.campaign_list.email, two hops, the second through the export. And lib.lineage.understated reports full_name and email in marketing.campaign_list as declared internal while lineage proposes personal: copies of personal data that any policy reading the declared classification would treat as internal.
Working rule: Run lib.lineage.understated, or your own check like it, on every lineage update, and fail when a declared classification is lower than lineage proposes. Classify by what a table holds, never by who reads it, and fix such a finding by raising the classification, not by dropping the edge.
本章的决定
- 每次流水线运行都会发出 OpenLineage 运行事件, 先是 START,随后是 COMPLETE 或 FAIL,事件由运行该流水线的代码(
lib.lineage.run)发出,并带有输出的模式和列级血缘。数据集以目录标识符命名,置于占位命名空间iceberg://platform之下。示例固定事件时间,并根据名义运行推导运行 ID,比较时只比较事件所表达的内容,而不比较其字节。 - 列级血缘采用 sqlglot 的值血缘,类型为 OpenLineage 子类型,再加上 INDIRECT 输入: 每条
SELECT(包括 CTE)的连接列、过滤列、分组列和排序列,都列在该 facet 的dataset列表中。 - 无法被正确读取的血缘属于错误, 绝不能为空或不完整。
lib.lineage只读取CREATE ... AS和INSERT ... SELECT,拒绝其他所有语句,包括MERGE,无论其USING指向什么。血缘为空、来源无法定位或列未知,都会在作业运行之前将其终止。INSERT的列来自目录中该表的模式,按位置匹配,数量不一致即为错误。 - 影响分析遍历由事件构建的图, 包括 INDIRECT 边,并报告值可能改变的列和行可能改变的表。
- 分类按所有输入中较严格者传播, 包括 DIRECT 和 INDIRECT,依
governance/lineage.toml中的设置而定,因此由个人信息过滤条件选出的表,会被建议归为个人信息。血缘绝不会降低已声明的分类,脱敏也绝不会使其降级。对两张未分类表的建议是:full_name、email和body为个人信息,其余各列为内部信息;这些建议在写入governance/metadata.toml之前,要交给各自的所有者审阅。 - 三项检查保护血缘本身: 运行过但没有 COMPLETE 事件的作业、没有任何作业的血缘能够解释的表,以及声明的分类低于血缘所建议分类的情况。血缘图的完整程度只取决于它的事件,而缺失的边看上去与没有任何依赖的列并无二致。
本章,在本版次中
版次 2026.11.31示例于 2026年11月19日 通过本章 11个示例 个示例中有 11 个通过
译自英文版图书,版次 2026.11.31。