Lineage from pipelines
읽는 데 약 67분분 걸린다, 11절에 11개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.31판을 번역한 것이다. 원문 읽기
이 페이지의 내용
- Send a run event from every pipeline
- 각 출력 열의 출처를 추적한다
- Add the inputs that decide which rows exist
- Fail on statements the parser can't read
- 컬럼 변경이 무엇을 깨뜨리는지 찾기
- Propose classifications from lineage
- 운영 환경에서 무엇이 잘못되는가
- Events that never arrive
- 아무도 계측하지 않은 작업
- Personal data in a table classified internal
- 이 장에서 결정한 것
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
By the end of this chapter you can capture lineage from your pipelines as OpenLineage run events, with column lineage parsed from their SQL, including the columns that only join, filter and group. You'll use the graph those events describe to say what a column change breaks downstream, and to propose classifications for tables nobody has classified. You'll also have checks for the three ways lineage fails in production: events that never arrive, a job that sends none, and a copy of personal data classified as less than personal.
The online learning platform from chapter 1 builds its reporting tables with four SQL pipelines over eight source tables. Each pipeline is one create table ... as select ... in examples/pipelines/, and the book's shared module lib.pipelines runs it on DuckDB and writes the result back to the catalog:
course_performancewritesanalytics.course_performance: enrollments, completions, ratings and revenue per course.learner_activitywritesanalytics.learner_activity: each learner's events and last activity.marketing_audiencewritesmarketing.audience: the learners marketing may email, with their name and email.review_corpuswritesai.review_corpus: the review text the platform's course assistant is grounded on.
Chapter 2's metadata check found that two of those outputs, marketing.audience and ai.review_corpus, have no metadata at all: no owner, and no classification for any column. Yet one holds learners' names and emails, and the other holds the free text of reviews, where chapter 3 found names and emails too. Two questions follow, and the catalog can't answer either. Which columns of those tables are personal data? And if the CRM team renames or reformats crm.learners.email, what breaks? Both are lineage questions. Lineage is where each column's data came from, and where it goes.
This chapter adds the book's shared module for lineage, lib.lineage. It runs each pipeline, tells OpenLineage what the run read and wrote, column by column, and reads the result back as a graph that answers both questions.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Send a run event from every pipeline
OpenLineage, an open specification for lineage, describes it with three objects (object model). A job is "a process that consumes or produces Datasets", such as a pipeline. A run is one occurrence of a job, identified by a UUID the client generates, and the spec recommends UUIDv7. A dataset is a table or file, named by a namespace and a name that together identify it. A run reports itself through run events: a START when it begins, then one terminal event, COMPLETE, FAIL or ABORT, and a server treats the events for one run as accumulating (run cycle). Lineage comes mostly from the work itself: a backend learns about datasets mainly from information about jobs.
The Python client, openlineage-python 1.55.0, builds events from RunEvent, Run, Job, InputDataset and OutputDataset in openlineage.client.event_v2, and sends them through a transport (client docs). It ships HTTP and Kafka transports for servers, and a file transport "designed mainly for integration testing", which with append set writes one JSON event per line to a single file (configuration). The client computes no lineage. It serializes what the caller passes in and adds only metadata of its own: a run facet with its version, the job's source location when it runs in a Git checkout, and any environment variables you configure it to collect (emit in client/client.py, openlineage-python). This means the lineage that reaches a server is the lineage your code put in the event.
lib.lineage.run wraps lib.pipelines.run, which runs one of the platform's four SQL pipelines and writes its table to the catalog. It sends START with the tables the SQL reads as inputs, runs the pipeline, and then sends COMPLETE with the output dataset and two facets on it. A facet is OpenLineage's name for a block of metadata attached to a run, a job or a dataset, and these two hold the output's schema and its column lineage, which the next section builds. If the pipeline raises, it sends FAIL instead, and lib.lineage.run_all runs all four pipelines this way, into one events file. The datasets need a namespace, and OpenLineage's naming conventions have no entry for an Iceberg table in a local SQL catalog. The nearest is the local file system, named by its path, which would put a folder on your machine into every event. Hence lib.lineage uses the placeholder iceberg://platform, with the catalog's identifiers, such as crm.learners, as names.
The examples need the same bytes every time, yet an event's time and random run ID change by default. Here lib.lineage stamps every event with the nominal start of the nightly run on the book's "today", and derives the run ID with the client's generate_static_uuid, which, its docstring says, "always produces the same result" for the same time and data: here, the job's name. Run the four pipelines with lineage twice, into two files, and print what the first file's events say, not the raw JSON:
1"""Runs the four pipelines with lineage twice, into two files, and reads back what the first file's events say. The2prose rests on the counts of events and run IDs, on the run facet the client adds, and on the same-bytes check, which3the example asserts."""45from openlineage.client.uuid import generate_new_uuid, generate_static_uuid67from lib import lineage8from lib.catalog import open_catalog9from lib.platform import scratch1011folder = scratch()12catalog = open_catalog()13lineage.run_all(catalog, folder / "first.jsonl")14lineage.run_all(catalog, folder / "second.jsonl") # the same nightly run, sent again15# lib.lineage fixes the event time and derives the run IDs16assert (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes()17events = lineage.read_events(folder / "first.jsonl")1819print("events:", len(events))20for event in events:21 job, state = event["job"]["name"], event["eventType"]22 if state == "START":23 print(f"{job} START, reads:", ", ".join(dataset["name"] for dataset in event["inputs"]))24 else:25 output = event["outputs"][0]26 lineage_fields = output["facets"]["columnLineage"]["fields"]27 print(f"{job} {state}, writes:", output["name"], f"({len(lineage_fields)} columns with lineage)")2829print("event times:", ", ".join(sorted({event["eventTime"] for event in events})))30print("distinct run IDs:", len({event["run"]["runId"] for event in events}))31print("run facets the client added:", ", ".join(events[0]["run"]["facets"]))32print("same bytes when sent again:", (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes())33# generate_new_uuid draws a random UUID each time; generate_static_uuid derives one from its arguments.34print("two random run IDs equal:", generate_new_uuid() == generate_new_uuid())35print("two static run IDs equal:", generate_static_uuid(lineage.NIGHTLY, b"marketing_audience")36 == generate_static_uuid(lineage.NIGHTLY, b"marketing_audience"))
python pipeline-run-events.pyevents: 8
course_performance START, reads: billing.payments, learning.courses, learning.enrollments, learning.reviews
course_performance COMPLETE, writes: analytics.course_performance (7 columns with lineage)
learner_activity START, reads: crm.learners, learning.events
learner_activity COMPLETE, writes: analytics.learner_activity (6 columns with lineage)
marketing_audience START, reads: crm.learners, learning.courses, learning.enrollments
marketing_audience COMPLETE, writes: marketing.audience (5 columns with lineage)
review_corpus START, reads: learning.courses, learning.reviews
review_corpus COMPLETE, writes: ai.review_corpus (5 columns with lineage)
event times: 2026-10-01T02:00:00+00:00
distinct run IDs: 4
run facets the client added: tags
same bytes when sent again: True
two random run IDs equal: False
two static run IDs equal: True
Sending the same nominal run again wrote the same bytes, as random run IDs never would, and the example asserts it. The first file holds 8 events, a START and a COMPLETE for each of 4 runs, all stamped 2026-10-01T02:00:00+00:00. The marketing_audience job's START names the tables its SQL reads, crm.learners, learning.courses and learning.enrollments, and its COMPLETE names marketing.audience, with lineage for each column. Yet the events hold something you didn't put there: a run facet named tags, carrying the client's own version (changelog, 1.38.0), and every facet's _producer URL names that version too (DEFAULT_PRODUCER in client/constants.py). So the bytes change with each client release, though the lineage doesn't, and a test should compare what events say, as this example's summary does.
A production emitter stamps COMPLETE with the time the run finished. A derived run ID is still worth keeping there, made from the scheduled time, the job's name and the attempt number: an event sent twice then lands in the same run instead of creating a second one, and a retried job gets a run of its own. Production events should also carry the output's Iceberg snapshot ID, in the version dataset facet the object model names for it. The code here leaves it out, because PyIceberg 0.12.0 draws each snapshot ID from a random UUID (_generate_snapshot_id in its pyiceberg/table/metadata.py).
Working rule: Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run can happen without them, and name datasets by catalog identifiers under a namespace you choose, never a path. In tests, use the file transport with a fixed event time and static run IDs, and compare what events say, never their bytes. In production, send through a transport to your lineage server, and derive each run ID from the scheduled time, the job and the attempt, so a resent event never creates a duplicate run.
각 출력 열의 출처를 추적한다
실행 이벤트는 작업이 어떤 테이블을 읽고 썼는지 알려 준다. 열 단위 리니지는 각 출력 열이 어떤 입력 열에서 왔는지 알려 주며, 거버넌스에 필요한 것은 바로 이 정보이다. crm.learners을(를) 읽는 테이블이 이메일을 복사했을 수도 있고 그렇지 않을 수도 있기 때문이다. OpenLineage는 열 단위 리니지를 출력 쪽 데이터셋 패싯인 columnLineage에 담는다. fields는 각 출력 열을 해당 inputFields에 대응시키고, 선택 사항인 dataset 목록은 데이터셋 전체에 영향을 주는 입력을 담는다(패싯 스키마). 클라이언트는 이 중 어느 것도 계산하지 않으므로, 누군가가 각 작업의 SQL을 파싱해야 한다.
SQL 파서인 sqlglot은 sqlglot.lineage.lineage로 이 일을 한다. 구문과 그 구문이 읽는 테이블의 스키마를 주면, 각 출력 열에 대해 잎이 원본 열인 트리를 반환하며, column=None를 쓰면 모든 출력 열을 한꺼번에 반환한다(sqlglot lineage). 이때 스키마가 필요하다. 스키마가 없으면 SELECT *은(는) *라는 이름의 출력 하나로 반환되고, 조인에서 테이블 한정자가 없는 열은 테이블이 없는 자리 표시자로 반환된다. lib.lineage.schema은(는) 카탈로그의 Iceberg 메타데이터로 스키마를 만들므로, 파서는 웨어하우스에 실제로 있는 열을 보게 되며, 이때 sqlglot 버전은 30.33.0이다.
마케팅팀이 이메일을 보낼 수 있는 목록에 데이터를 공급하는 marketing_audience에서 시작한다. 말로 풀면, 이 테이블은 마케팅 이메일에 동의했고 계정이 활성 상태인 학습자를 고르고, 각 학습자가 가장 좋아하는 강좌 카테고리를 덧붙인다. 이 카테고리는 학습자의 수강 등록 수로 카테고리 순위를 매기는 CTE가 계산한다. sqlglot이 무엇을 빠뜨리는지 보기 위해, 파이프라인의 출력과 course_performance의 두 열에 대한 리니지를 sqlglot에 요청한다.
1"""What sqlglot.lineage says the pipeline outputs come from, and what it leaves out. The prose rests on the CTE-traced2source of favourite_category, on the sources of the two course performance columns, and on the assertion that neither3column the WHERE reads reaches any output."""4import sqlglot5from sqlglot import exp67from lib import lineage, pipelines8from lib.catalog import open_catalog910# sqlglot resolves each name against the catalog's columns, so lineage.schema builds them from the table metadata.11schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",12 "billing.payments"]))13statement = pipelines.sql("marketing_audience")14found = lineage.sqlglot_sources(statement, schema)15for column, sources in found.items():16 print(f"marketing.audience.{column} comes from:", ", ".join(sources))1718parsed = sqlglot.parse_one(statement, read="duckdb")19# the WHERE names columns by alias, as in l.status, so each alias maps to its table20tables = {table.alias_or_name: f"{table.db}.{table.name}" for table in parsed.find_all(exp.Table)}21where = parsed.find(exp.Where)22print("the pipeline's WHERE:", where.this.sql(dialect="duckdb"))23in_lineage = {source for sources in found.values() for source in sources} # every source column of every output24for column in where.find_all(exp.Column):25 source = f"{tables[column.table]}.{column.name}"26 assert source not in in_lineage # every column the WHERE reads, checked against the lineage above27 print(f"{source} in any column's lineage:", source in in_lineage)2829course_performance = lineage.sqlglot_sources(pipelines.sql("course_performance"), schema)30for column in ("completions", "revenue_usd"): # the two aggregates that carry a FILTER inside them31 print(f"analytics.course_performance.{column} comes from:", ", ".join(course_performance[column]))
python sqlglot-misses-row-inputs.pymarketing.audience.learner_id comes from: crm.learners.learner_id
marketing.audience.full_name comes from: crm.learners.full_name
marketing.audience.email comes from: crm.learners.email
marketing.audience.country comes from: crm.learners.country
marketing.audience.favourite_category comes from: learning.courses.category
the pipeline's WHERE: l.marketing_consent AND l.status = 'active'
crm.learners.marketing_consent in any column's lineage: False
crm.learners.status in any column's lineage: False
analytics.course_performance.completions comes from: learning.enrollments.completed_at, learning.enrollments.enrollment_id
analytics.course_performance.revenue_usd comes from: billing.payments.amount, billing.payments.status
값은 올바르지만, 어떤 학습자가 테이블에 들어가는지를 결정하는 요소가 빠져 있다. email은(는) crm.learners.email에서 오고, favourite_category은(는) CTE를 거쳐 learning.courses.category까지 추적된다. 파이프라인의 WHERE은(는) l.marketing_consent AND l.status = 'active'이며, 그 열인 marketing_consent과(와) status 중 어느 것도 어떤 출력의 리니지에도 들어 있지 않다. 예제는 두 열 모두에 대해 단언한다. 동의 필터는 마케팅팀이 이 테이블을 보유해도 되는 이유 그 자체인데, 리니지는 이를 언급하지 않는다. 수강 등록과 강좌의 조인 키도 언급하지 않는다.
두 번째 문제는 그 반대이다. 필터로만 쓰이는 입력이 값인 것처럼 나열된다. course_performance은(는) filter (where p.status = 'succeeded')로 성공한 결제만 합산하는데, sqlglot은 revenue_usd이(가) billing.payments.amount, billing.payments.status에서 온다고 말한다. 합산되는 열과 어떤 결제를 셀지만 결정하는 열이 똑같이 나열되는 것이다. completions은(는) filter (where e.completed_at is not null)로 수강 등록을 세며, learning.enrollments.completed_at, learning.enrollments.enrollment_id에서 온다는 같은 결과가 나온다. 두 문제는 모두 sqlglot이 트리를 만드는 방식에서 비롯된다. sqlglot은 각 출력의 표현식 안에 있는 열을 수집하므로 표현식 안에 있는 필터만 볼 뿐 그 바깥의 것은 보지 못하고, 노드에는 입력이 어떻게 쓰였는지 알려 주는 유형이 없다.
작업 규칙: 각 출력의 값이 계산되는 열을 알아내는 데는 sqlglot의 lineage를 사용하고, 스키마는 직접 입력하지 말고 항상 테이블 메타데이터에서 생성한 카탈로그의 스키마를 전달한다. 그 결과를 열이 의존하는 모든 것으로 읽어서는 안 된다. 조인, 필터, 그룹에 쓰이는 열은 빠져 있고, 입력 중 어느 것이 값인지도 알려 주지 않는다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Add the inputs that decide which rows exist
sqlglot's lineage lists the columns a value is computed from, and OpenLineage has a place for the rest. Each input field carries a list of transformations, each with a type, a subtype and a masking flag (column lineage facet). DIRECT means the output value is derived from the input value, through IDENTITY, TRANSFORMATION or AGGREGATION. INDIRECT means the output value is impacted by an input without being derived from it, through JOIN, GROUP_BY, FILTER, SORT, WINDOW or CONDITIONAL. Inputs that shape every row, such as a WHERE, belong in the facet's dataset list.
For governance, the INDIRECT side matters, since information flows through conditions as well as values. Denning calls the information that flows through conditions "implicit flow": a statement that assigns only when a condition holds still tells you the condition's value (Denning, 1976). A row in marketing.audience tells you that learner consented.
lib.lineage.column_lineage adds two things sqlglot's answer lacks: a type for each value input, and the INDIRECT inputs. It types each value input by where the column sits in the output's expression: a bare column is IDENTITY, a column inside an aggregate is AGGREGATION, and anything else is TRANSFORMATION. A column inside an aggregate's FILTER, a window's PARTITION BY or ORDER BY, or a CASE's condition or operand is INDIRECT, and a hash on the way sets masking. COALESCE's arguments count as values: OpenLineage lists COALESCE under CONDITIONAL, but any of its arguments can become the output's value. Then it walks every SELECT in the statement, CTEs included, and collects the columns of its row clauses, following CTE columns back to the tables they come from:
ROW_CLAUSES = {"joins": "JOIN", "where": "FILTER", "group": "GROUP_BY", "having": "FILTER", "qualify": "FILTER",
"order": "SORT"}
Run it on the same pipeline and the same two columns:
1"""The book's column lineage for the marketing audience pipeline and two course performance columns: each input,2with its OpenLineage type and subtype, then the inputs that shape every row. The prose rests on the source counts."""3from lib import lineage, pipelines4from lib.catalog import open_catalog56schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",7 "billing.payments"]))8# rows holds the inputs that shape every row, which OpenLineage keeps in the dataset list9fields, rows = lineage.column_lineage(pipelines.sql("marketing_audience"), schema)10for column, inputs in fields.items():11 print(f"marketing.audience.{column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"12 for dependency in inputs))13print("inputs that shape every row of marketing.audience:")14for source in sorted({dependency.source for dependency in rows}):15 print(f" {source}:", "INDIRECT",16 ", ".join(dependency.subtype for dependency in rows if dependency.source == source))1718# the source columns that the output values are computed from19values = {dependency.source for inputs in fields.values() for dependency in inputs}20print("source columns, values only:", len(values))21# A column can both give a value and shape the rows, so the union counts it once.22print("source columns, with the rows' inputs:", len(values | {dependency.source for dependency in rows}))2324fields, _ = lineage.column_lineage(pipelines.sql("course_performance"), schema) # only these two columns are printed25for column in ("completions", "revenue_usd"):26 print(f"analytics.course_performance.{column}:",27 ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}" for dependency in fields[column]))
python indirect-column-lineage.pymarketing.audience.learner_id: crm.learners.learner_id DIRECT IDENTITY
marketing.audience.full_name: crm.learners.full_name DIRECT IDENTITY
marketing.audience.email: crm.learners.email DIRECT IDENTITY
marketing.audience.country: crm.learners.country DIRECT IDENTITY
marketing.audience.favourite_category: learning.courses.category DIRECT IDENTITY
inputs that shape every row of marketing.audience:
crm.learners.learner_id: INDIRECT JOIN
crm.learners.marketing_consent: INDIRECT FILTER
crm.learners.status: INDIRECT FILTER
learning.courses.category: INDIRECT GROUP_BY, JOIN
learning.courses.course_id: INDIRECT JOIN
learning.enrollments.course_id: INDIRECT JOIN
learning.enrollments.learner_id: INDIRECT GROUP_BY, JOIN
source columns, values only: 5
source columns, with the rows' inputs: 10
analytics.course_performance.completions: learning.enrollments.completed_at INDIRECT FILTER, learning.enrollments.enrollment_id DIRECT AGGREGATION
analytics.course_performance.revenue_usd: billing.payments.amount DIRECT AGGREGATION, billing.payments.status INDIRECT FILTER
The values match sqlglot's, each an IDENTITY copy, and the rows' inputs are the new part. Adding them grows what marketing.audience depends on from 5 source columns to 10. Some row inputs are columns no value uses: marketing_consent and status as FILTER, the two course IDs as JOIN, and the enrollments' learner_id as GROUP_BY and JOIN. crm.learners.learner_id and learning.courses.category, which values copy too, also shape rows: the ID as JOIN, the category as GROUP_BY and JOIN. For revenue_usd, the two inputs are now told apart: amount is a DIRECT AGGREGATION, and status an INDIRECT FILTER. So are the two inputs of completions: enrollment_id is counted, and completed_at only filters.
One more detail: the per-field transformationType and transformationDescription are deprecated, replaced by the per-input transformations that lib.lineage writes (changelog, 1.17.1). And OpenLineage 1.53.0 added a Lineage dataset facet for declared lineage, such as a view's, sent on a DatasetEvent, which consumers should prefer where both describe the same relationship (lineage facet). Run events from jobs carry columnLineage, as the platform's do.
Working rule: Record INDIRECT inputs with every job's column lineage: the join keys, filters and grouping columns of every SELECT, CTEs included, in the facet's dataset list, and type each value input with OpenLineage's subtypes. Expect more edges, many of them join keys. Keep them anyway: a filter on a consent column decides who may be emailed, and it reaches the table only as an INDIRECT input.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Fail on statements the parser can't read
The platform's column lineage is only as good as the parse behind it, and a parse can be wrong without failing. sqlglot's lineage accepts more than a SELECT: it finds the query inside a CREATE TABLE ... AS, an INSERT ... SELECT or a CREATE VIEW, and it reads a MERGE too (sqlglot, optimizer/scope.py). Three cases come back looking fine. An INSERT's outputs are named after its SELECT, never after the table's columns. A MERGE comes back as the last query sqlglot reaches in it, as if that query were what the MERGE writes. When USING names a table or a CTE and no subquery follows, sqlglot returns no lineage and no error. When USING is a subquery, or a subquery sits in a WHEN condition or a SET, it returns the columns of the last such subquery, whether the MERGE writes them or not. And a misspelt column gets through when it's unqualified in a query on one table: sqlglot assigns it to that table whether the table has the column or not.
All four of the platform's pipelines are CREATE TABLE ... AS statements. As tables grow, pipelines like these often become incremental: an INSERT that appends a day's revenue to analytics.daily_revenue, or a MERGE that updates analytics.learner_activity in place. Ask sqlglot and lib.lineage.column_lineage about each, the INSERT and the MERGE in two forms, and compare the answers:
1"""sqlglot.lineage's answers for an INSERT and for two MERGE statements, beside the book's lineage: sqlglot names the2INSERT's outputs after its SELECT, and returns nothing or a subquery's columns for a MERGE, which the book refuses. The3prose rests on the sqlglot outputs and on the refusals at the end."""4import pyarrow as pa5from sqlglot.lineage import lineage as sqlglot_lineage67from lib import lineage8from lib.catalog import open_catalog910catalog = open_catalog(["billing.payments", "crm.learners"])11catalog.create_namespace("analytics") # the incremental table, as it exists before the day's INSERT12catalog.create_table("analytics.daily_revenue", schema=pa.schema([("day", pa.date32()), ("revenue_usd", pa.float64())]))13schema = lineage.schema(catalog)14insert = """insert into analytics.daily_revenue15select cast(p.paid_at as date), sum(p.amount) from billing.payments p where p.status = 'succeeded' group by 1"""16merges = {"USING a table": """merge into analytics.learner_activity a using crm.learners l17on a.learner_id = l.learner_id when matched then update set country = l.country""",18 "USING a subquery": """merge into analytics.learner_activity a19using (select l.learner_id, l.country, l.email from crm.learners l) s on a.learner_id = s.learner_id20when not matched then insert (learner_id, country) values (s.learner_id, s.country)"""}2122print("analytics.daily_revenue columns:", ", ".join(schema["analytics"]["daily_revenue"]))23# The SELECT's expressions have no aliases, so sqlglot names each output after the SELECT, not the table. None asks for24# every output column at once.25print("sqlglot, INSERT outputs:", ", ".join(sqlglot_lineage(None, insert, schema, dialect="duckdb")))26fields, rows = lineage.column_lineage(insert, schema)27for column, inputs in fields.items():28 print(f"book, INSERT {column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"29 for dependency in inputs))30print("book, INSERT rows shaped by:", ", ".join(f"{dependency.source} {dependency.subtype}" for dependency in rows))31for label, merge in merges.items():32 outputs = sqlglot_lineage(None, merge, schema, dialect="duckdb")33 print(f"sqlglot, MERGE {label}, outputs:", ", ".join(outputs) or "none")3435# One value too many for daily_revenue's two columns, caught after sqlglot parses it; each MERGE is refused before sqlglot is asked.36refused = {"INSERT with one value too many": insert.replace("sum(p.amount)", "sum(p.amount), count(*)"),37 **{f"MERGE {label}": merge for label, merge in merges.items()}}38for label, statement in refused.items():39 try:40 lineage.column_lineage(statement, schema)41 except lineage.LineageError as error:42 print(f"book, {label}:", error)
python insert-and-merge-lineage.pyanalytics.daily_revenue columns: day, revenue_usd
sqlglot, INSERT outputs: paid_at, _col_1
book, INSERT day: billing.payments.paid_at DIRECT TRANSFORMATION
book, INSERT revenue_usd: billing.payments.amount DIRECT AGGREGATION
book, INSERT rows shaped by: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER
sqlglot, MERGE USING a table, outputs: none
sqlglot, MERGE USING a subquery, outputs: learner_id, country, email
book, INSERT with one value too many: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values
book, MERGE USING a table: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
book, MERGE USING a subquery: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
The table analytics.daily_revenue has the columns day, revenue_usd, yet sqlglot names the INSERT's outputs paid_at, _col_1: the first after the column inside the cast, the second a generated name for an expression with no alias.
An INSERT fills columns by position, so lib.lineage matches by position too, taking the names from the catalog: the columns the INSERT lists, or else all of the table's. So day comes from billing.payments.paid_at DIRECT TRANSFORMATION, and the INSERT's grouping and its filter on status shape every row: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER. When the SELECT has more or fewer values than the columns written, lib.lineage raises: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values.
For the MERGE whose USING names a table, sqlglot returns none, and no error. For the one whose USING is a subquery, it returns learner_id, country, email, though that MERGE writes only learner_id and country. A check for empty lineage catches the first and passes the second, so lib.lineage doesn't judge what sqlglot returns for a MERGE. Before it calls sqlglot, it refuses every statement that isn't a CREATE ... AS or an INSERT ... SELECT: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE.
A misspelt column, the third case, depends on how the column is written. The platform's pipelines qualify every column with its table's alias, as in l.email, and sqlglot checks a qualified column against the schema itself:
1"""A misspelt column, emial for email, in a query on one table: sqlglot raises when the column is qualified by its2alias, and traces it to the table when it isn't. The prose rests on the sqlglot answer for each form, and on the book's3check against the schema."""4from sqlglot.errors import OptimizeError56from lib import lineage7from lib.catalog import open_catalog89schema = lineage.schema(open_catalog(["crm.learners"])) # the schema lets sqlglot check each column name10# the alias l is right, and only the column name is misspelt11qualified = "create table marketing.audience as select l.learner_id, l.emial from crm.learners l"12# no alias, so nothing names the table the column comes from13unqualified = "create table marketing.audience as select learner_id, emial from crm.learners"1415try:16 lineage.sqlglot_sources(qualified, schema)17except OptimizeError as error:18 print("sqlglot, qualified l.emial:", f"{type(error).__name__}: {error}")19print("sqlglot, unqualified emial comes from:", ", ".join(lineage.sqlglot_sources(unqualified, schema)["emial"]))20try:21 lineage.column_lineage(unqualified, schema)22except lineage.LineageError as error:23 print("book, unqualified emial:", error)
python misspelt-column-lineage.pysqlglot, qualified l.emial: OptimizeError: Unknown column: emial
sqlglot, unqualified emial comes from: crm.learners.emial
book, unqualified emial: crm.learners has no column emial
A qualified typo stops sqlglot with OptimizeError: Unknown column: emial. An unqualified one, in a query on one table, comes back as crm.learners.emial: a confident answer about a column that doesn't exist. In a join, a misspelt unqualified column belongs to none of the tables, and sqlglot returns a placeholder with no table (sqlglot, lineage.py), which lib.lineage rejects as a source it can't place. For the single-table case, lib.lineage checks every source against the schema and raises crm.learners has no column emial.
lib.lineage.record, which run uses, reads the lineage before the job runs, so a pipeline whose lineage can't be read fails before it writes anything. That is strict, and deliberate: a job with empty lineage would vanish from every impact report and classification check in this chapter, and one with wrong lineage would mislead them. Run the same check in CI on every pull request that changes a pipeline, so the failure shows up in review rather than at night.
Working rule: Treat a statement your parser can't read correctly as an error, never as empty or partial lineage. Refuse every kind of statement it gets wrong, such as MERGE, before you ask it, and fail on empty lineage, a source the parser can't place and a column the schema doesn't have. Match an INSERT's columns by position against the table's columns in the catalog, never the SELECT's names, and fail when the counts differ. For a MERGE, write a SELECT of the columns it writes, and record the job's lineage from that, as an INSERT ... SELECT into the MERGE's table, until your parser handles MERGE itself.
컬럼 변경이 무엇을 깨뜨리는지 찾기
모든 COMPLETE 이벤트에 컬럼 리니지가 있으므로 플랫폼에는 그래프가 생긴다. 컬럼은 노드이고, 각 입력 필드는 그 필드가 값을 공급하는 컬럼이나 테이블로 향하는 에지이다. 영향 분석(impact analysis)은 컬럼 하나를 바꾸면 무엇이 깨질 수 있는지를 묻는다. 이 분석은 해당 컬럼에서 출발해 나가는 모든 에지를 따라가고, 각 에지가 닿은 곳에서 다시 깊이에 상관없이 계속 따라간다. 특별한 방법은 필요 없으며, 이는 그래프 도달 가능성 문제이다. 리니지에만 해당하는 규칙이 하나 있다. 테이블로 들어가는 INDIRECT 에지는 그 테이블의 모든 행이 바뀔 수 있다는 뜻이므로, 탐색은 그 테이블의 모든 컬럼에서 계속된다.
lib.lineage.graph는 SQL이 아니라 이벤트 파일만으로 그래프를 만든다. 리니지 서버가 하는 방식과 같다. 따라서 이벤트를 보내는 작업은 어떤 언어로 작성되었든 모두 포함되고, 이벤트를 보내지 않는 작업은 포함되지 않는다. 탐색은 lib.lineage.downstream가 수행한다.
플랫폼의 팀들이 할 만한 변경 세 가지를 생각해 보자. CRM 팀이 모든 이메일을 소문자로 바꾸거나 crm.learners.email의 이름을 바꾼다. 새 가입 흐름이 오래된 계정에 대해 crm.learners.marketing_consent의 의미를 바꾼다. 학습 팀이 learning.courses의 카테고리 이름을 바꾼다. 먼저 추측해 보자. 동의 값 변경은 몇 개의 컬럼에 영향을 줄 수 있을까?
1"""What a change to one source column can break, walked over the column graph that the nightly run's events describe.2The prose rests on the reach counts printed for each column: through values, and in all."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch78events_file = scratch() / "events.jsonl"9lineage.run_all(open_catalog(), events_file)10graph = lineage.graph(lineage.read_events(events_file)) # built from the events alone, as a lineage server would11print("edges in the column graph:", len(graph.edges))1213for column in ("crm.learners.email", "crm.learners.marketing_consent", "learning.courses.category"):14 reached = lineage.downstream(graph, column)15 print(f"{column} reaches:")16 # dict.fromkeys drops repeated targets and keeps the order they were found in17 for target in dict.fromkeys(edge.target for edge in reached):18 how = ", ".join(f"{edge.type} {edge.subtype}" for edge in reached if edge.target == target)19 # a table's name has one dot and a column's two, so a table target means every row of it20 print(f" {target}{', every row' if target.count('.') == 1 else ''}:", how)21 values = [edge for edge in reached if edge.type == "DIRECT"] # only DIRECT edges copy values22 print(f"{column}, columns reached through values:", len(lineage.columns_reached(graph, values)))23 print(f"{column}, columns reached in all:", len(lineage.columns_reached(graph, reached)))
python column-impact.pyedges in the column graph: 51
crm.learners.email reaches:
marketing.audience.email: DIRECT IDENTITY
crm.learners.email, columns reached through values: 1
crm.learners.email, columns reached in all: 1
crm.learners.marketing_consent reaches:
marketing.audience, every row: INDIRECT FILTER
crm.learners.marketing_consent, columns reached through values: 0
crm.learners.marketing_consent, columns reached in all: 5
learning.courses.category reaches:
analytics.course_performance, every row: INDIRECT GROUP_BY
analytics.course_performance.category: DIRECT IDENTITY
marketing.audience, every row: INDIRECT GROUP_BY, INDIRECT JOIN
marketing.audience.favourite_category: DIRECT IDENTITY
learning.courses.category, columns reached through values: 2
learning.courses.category, columns reached in all: 12
의외인 것은 동의 컬럼이다. 이 컬럼의 변경은 값을 통해서는 0개의 컬럼에 닿는다. 그러나 필터를 통해서는 marketing.audience의 모든 컬럼, 즉 5개에 닿는다. 누구에게 이메일을 보낼지가 바뀌기 때문이다. 값만 추적하는 리니지 도구는 동의 컬럼의 변경에 영향이 없다고 보고할 것이다.
email의 변경은 1개의 컬럼 marketing.audience.email에 닿는다. 따라서 이름을 바꾸면 마케팅 파이프라인이 깨지지만, 리니지를 보내는 다른 작업은 깨지지 않는다. 카테고리 이름 변경은 값을 통해 2개의 컬럼에, 전체로는 12개의 컬럼에 닿는다. 카테고리가 analytics.course_performance을 그룹화하고 각 학습자가 가장 좋아하는 항목의 순위를 매기는 데에도 쓰이기 때문이다.
같은 탐색이 이후 두 개의 장에서도 쓰인다. 7장은 학습자의 컬럼을 하류로 따라가 삭제 요청이 도달해야 하는 모든 파생 테이블을 찾고, 10장은 스키마를 바꾸는 모든 풀 리퀘스트에 대해 영향 보고서를 작성한다.
작업 규칙: 이벤트로 만든 그래프에서 INDIRECT 에지를 포함해 영향 분석을 실행하고, 값이 바뀔 수 있는 컬럼과 행이 바뀔 수 있는 테이블의 두 목록을 보고한다. 필터와 조인에만 쓰이는 컬럼의 변경은 그 컬럼이 필터링하는 테이블의 모든 컬럼에 대한 변경으로 취급하고, 보고서는 영향이 닿는 각 테이블의 소유자에게 보낸다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Propose classifications from lineage
A derived column holds data from its inputs, so its classification can be derived from theirs. The classic rule comes from Denning's lattice model of information flow: the class of a value computed from several inputs is the least upper bound, the "join", of their classes (Denning, 1976). The platform's classifications form a line, public, internal, personal and sensitive, so the join is the stricter of them, which stricter in lib.metadata, the book's metadata module, returns. Which inputs count is a judgement. We count INDIRECT inputs too, following Denning's implicit flows, because a table filtered on consent reveals each learner's consent. That choice lives in governance/lineage.toml, so a team can change it in one reviewed line.
Two more choices follow from the same caution. OpenLineage's masking flag says only that a value was obfuscated, such as by a hash (column lineage facet). Whether a hashed identifier is still personal data is a legal question: chapter 2 quotes the GDPR recital on pseudonymized data and leaves the answer to lawyers. So the book's rule never lowers a classification because of masking. Catalog servers leave the choice to configuration too. Apache Atlas propagates classifications along lineage, and can block that on an edge, for example where a view masks a PII column (Atlas). DataHub propagates glossary terms along column lineage, a feature its 1.7.0 documentation calls a public beta in DataHub Cloud (DataHub). A proposal is a floor, not a verdict: like the drafts a local model writes in chapter 2, it's reviewed before it goes into governance/metadata.toml, the platform's file of owners and classifications.
lib.lineage.propose proposes a classification for every column a job wrote, and lib.metadata.problems lists the tables that need one. Check the rule on the two tables people did classify, analytics.course_performance and analytics.learner_activity, before you trust it on marketing.audience and ai.review_corpus, the two with no metadata:
1"""Proposes classifications from lineage for the columns with no declared one, then checks the rule against the columns2people did classify. The prose rests on the personal count, on the reproduced counts, and on the check that no declared class is lower."""34from lib import lineage, metadata5from lib.catalog import open_catalog6from lib.platform import scratch78catalog = open_catalog()9events_file = scratch() / "events.jsonl"10lineage.run_all(catalog, events_file)11for problem in metadata.problems(catalog, metadata.load()):12 print("metadata problem:", problem)1314graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared() # known: governance/metadata.toml15print("propagated through:", " and ".join(lineage.rules()["propagate_through"])) # from governance/lineage.toml16proposed = lineage.propose(graph, known)17for column, (classification, decided_by) in proposed.items():18 if column not in known:19 first = decided_by[0] # lib sorts the deciding edges DIRECT first, so this is a value when one decides20 print(f"{column}: {classification}, from {first.source} ({first.type} {first.subtype})")21print("personal columns proposed:", sum(1 for column, (classification, _) in proposed.items()22 if column not in known and classification == "personal"))2324classified = [column for column in proposed if column in known]25print("declared classifications checked:", len(classified))26print("classes declared for them:", ", ".join(sorted({known[column] for column in classified})))27for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):28 proposals = lineage.propose(graph, known, through)29 agree = sum(1 for column in classified if proposals[column][0] == known[column])30 print(f"declared classifications reproduced, through {' and '.join(through)}:", f"{agree} of {len(classified)}")31# proposals is the DIRECT-only run from the loop above32missed = [column for column in classified if proposals[column][0] != known[column]]33print("missed with DIRECT inputs only:", len(missed))34for column in missed:35 print(f" {column}: declared {known[column]}, DIRECT inputs propose {proposals[column][0]}")36print("declared less protected than lineage proposes:", len(lineage.understated(proposed, known)))37assert not lineage.understated(proposed, known)
python propagate-classifications.pymetadata problem: ai.review_corpus: no metadata
metadata problem: marketing.audience: no metadata
propagated through: DIRECT and INDIRECT
marketing.audience.learner_id: internal, from crm.learners.learner_id (DIRECT IDENTITY)
marketing.audience.full_name: personal, from crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email: personal, from crm.learners.email (DIRECT IDENTITY)
marketing.audience.country: internal, from crm.learners.country (DIRECT IDENTITY)
marketing.audience.favourite_category: internal, from crm.learners.learner_id (INDIRECT JOIN)
ai.review_corpus.review_id: internal, from learning.reviews.review_id (DIRECT IDENTITY)
ai.review_corpus.course_id: internal, from learning.reviews.course_id (DIRECT IDENTITY)
ai.review_corpus.course_title: internal, from learning.reviews.course_id (INDIRECT JOIN)
ai.review_corpus.rating: internal, from learning.reviews.rating (DIRECT IDENTITY)
ai.review_corpus.body: personal, from learning.reviews.body (DIRECT IDENTITY)
personal columns proposed: 3
declared classifications checked: 13
classes declared for them: internal
declared classifications reproduced, through DIRECT and INDIRECT: 13 of 13
declared classifications reproduced, through DIRECT: 10 of 13
missed with DIRECT inputs only: 3
analytics.course_performance.course_id: declared internal, DIRECT inputs propose public
analytics.course_performance.title: declared internal, DIRECT inputs propose public
analytics.course_performance.category: declared internal, DIRECT inputs propose public
declared less protected than lineage proposes: 0
The metadata check from chapter 2 still reports the two tables with no metadata, and lineage proposes a classification for each of their columns. It proposes 3 personal columns: full_name and email in marketing.audience, each an IDENTITY copy of a personal column in crm.learners, and body in ai.review_corpus, a copy of the reviews' free text. The rest come out internal, including favourite_category, for which the example prints internal, from crm.learners.learner_id (INDIRECT JOIN). Its values are public course-catalog category names, but the rows they sit in are chosen by a join on learner IDs and a consent filter, so those are internal.
Counting INDIRECT inputs reproduces the declared classifications on the columns people did classify, 13 of 13, and all of them are internal.
DIRECT inputs alone reproduce 10 of 13, and miss 3: course_id, title and category of analytics.course_performance. A DIRECT-only rule calls them public, as in the course catalog, while the analytics team chose internal. So the rule proposes no less than people chose here, and lineage finds no declared classification lower than its proposal. That is weak evidence: a rule that proposed internal for every column would score as well. And it can't show what the rule does with a personal filter, since none of the platform's filters or joins reads a personal column: even marketing_consent is classified internal.
So build a table whose filter does. For a birthday offer, marketing wants the learners who consented and were born this month, with their ID and country, and nothing else:
1"""A birthday-offer table whose only personal input is a filter on date of birth. The prose rests on the learner count,2and on the proposals counting DIRECT inputs alone and counting DIRECT and INDIRECT inputs."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch7from lib.warehouse import connect89catalog = open_catalog(["crm.learners"])10# The consenting learners born in the month of the nightly run, on the book's "today".11select = ("select l.learner_id, l.country from crm.learners l "12 f"where l.marketing_consent and month(l.date_of_birth) = {lineage.NIGHTLY.month}")131415def offers() -> None:16 rows = connect(catalog).sql(select).to_arrow_table()17 catalog.create_namespace_if_not_exists("marketing")18 catalog.create_table("marketing.birthday_offers", schema=rows.schema).append(rows)192021events_file = scratch() / "events.jsonl"22# record derives the lineage from this statement, and reads it before offers() runs23lineage.record(lineage.client(events_file), "birthday_offers", f"create table marketing.birthday_offers as {select}",24 catalog, offers)25offers_table = catalog.load_table("marketing.birthday_offers").scan().to_arrow()26print("learners in marketing.birthday_offers:", f"{len(offers_table):,}")2728graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()29print("crm.learners.date_of_birth declared:", known["crm.learners.date_of_birth"])30for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):31 for column, (classification, decided_by) in lineage.propose(graph, known, through).items():32 label = f"{column}, through {' and '.join(through)}"33 print(f"{label}:", classification)34 print(f"{label}, decided by:", f"{decided_by[0].source} ({decided_by[0].type} {decided_by[0].subtype})")
python personal-filter-propagates.pylearners in marketing.birthday_offers: 165
crm.learners.date_of_birth declared: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.country, through DIRECT and INDIRECT: personal
marketing.birthday_offers.country, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.learner_id, through DIRECT: internal
marketing.birthday_offers.learner_id, through DIRECT, decided by: crm.learners.learner_id (DIRECT IDENTITY)
marketing.birthday_offers.country, through DIRECT: internal
marketing.birthday_offers.country, through DIRECT, decided by: crm.learners.country (DIRECT IDENTITY)
Neither column copies anything personal, but every row says its learner was born in the offer's month, part of a date of birth. The table holds 165 learners, and governance/metadata.toml classifies the date of birth personal. Counting DIRECT inputs alone proposes internal for learner_id and internal for country.
Add INDIRECT inputs, and it proposes personal for learner_id and personal for country, both decided by crm.learners.date_of_birth (INDIRECT FILTER). That is the case INDIRECT inputs are for, and it shows the cost too: one personal filter makes every column of a table personal.
OpenLineage can carry classifications in the events themselves, with the Tags dataset facet, whose field key "identifies the field in a dataset if a tag applies to one" (tags facet, schema). We keep them in governance/metadata.toml instead, the one source that the platform's access policies also read, through lib.policy, the book's policy module (chapter 5).
Working rule: Propose each derived column's classification as the stricter of its inputs', counting INDIRECT inputs, and send the proposals to the table's owner for review before they become metadata. Never let lineage lower a classification someone declared, and never treat masking as declassification. Count DIRECT inputs only if your pipelines often join or filter on personal columns, such as matching on email, and every output would otherwise come out personal; then review INDIRECT personal inputs by hand.
운영 환경에서 무엇이 잘못되는가
계보는 조용히 실패한다. 누락된 에지는 의존성이 없는 열과 똑같아 보이며, 영향 분석부터 제안된 분류에 이르기까지 그래프를 바탕으로 한 모든 검사가 그대로 통과한다. 흔한 실패는 세 가지이며, 각각에 대응하는 검사가 있다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Events that never arrive
The client reads OPENLINEAGE_DISABLED from the environment, and when it's true, events are dropped (configuration). In the client's source, that check comes first (openlineage-python, client/client.py): before it looks at the transport you passed explicitly, the client picks a transport that does nothing. Its only notice is an INFO log line: "OpenLineage is disabled. No events will be emitted." Python's root logger is set to WARNING by default, so the line never shows (logging HOWTO). A variable left in a shared environment file after debugging is enough. Set the variable, run review_corpus with an explicit file transport, and check what reaches the file:
1"""With OPENLINEAGE_DISABLED=true in the environment, an explicit file transport writes nothing and raises nothing. The2prose rests on the transport the client chose and on the job the check finds without a COMPLETE event."""3import os45from lib import lineage6from lib.catalog import open_catalog7from lib.platform import scratch89os.environ["OPENLINEAGE_DISABLED"] = "true" # say, left in a shared environment file after a debugging session1011events_file = scratch() / "events.jsonl"12client = lineage.client(events_file) # a file transport, passed in explicitly13catalog = open_catalog(["learning.courses", "learning.reviews"])14output = lineage.run(catalog, "review_corpus", client)1516print("transport the client chose:", client.transport.kind) # OPENLINEAGE_DISABLED wins over the transport passed in17print("pipeline wrote:", output)18print("rows written:", f"{len(catalog.load_table(output).scan().to_arrow()):,}")19assert not events_file.exists() # the noop transport never creates the file20print("events file exists:", events_file.exists())21missing = lineage.missing_runs(["review_corpus"], lineage.read_events(events_file)) # ran, but sent no COMPLETE22print("jobs that ran with no COMPLETE event:", ", ".join(missing))
python lineage-disabled-silently.pytransport the client chose: noop
pipeline wrote: ai.review_corpus
rows written: 3,000
events file exists: False
jobs that ran with no COMPLETE event: review_corpus
With the variable set, the pipeline still runs, but nothing reaches the file. The client chose the noop transport over the file transport the code passed in, and the pipeline wrote 3,000 rows to ai.review_corpus without an error and without an events file. The check that catches it compares the jobs the scheduler ran with the jobs that sent a COMPLETE event: lib.lineage.missing_runs lists review_corpus as having run without one.
Working rule: After every scheduled run, compare the jobs that ran with the jobs whose COMPLETE events arrived, and alert on any difference. Don't rely on the emitter to report its own silence.
아무도 계측하지 않은 작업
리니지는 이벤트를 보내는 작업만 다룬다. 플랫폼의 마케팅 팀은 매주 Python 스크립트를 실행하여 marketing.audience의 내용을 이메일 도구가 읽는 테이블인 marketing.campaign_list로 복사한다. 이 스크립트는 OpenLineage에 연결되어 있지 않다.
1"""A Python step copies marketing.audience for the email tool and sends no lineage. The prose rests on the walk from the2email column, which misses the copy, and on the catalog check against the lineage, which finds it."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch7from lib.warehouse import connect89catalog = open_catalog()10events_file = scratch() / "events.jsonl"11lineage.run_all(catalog, events_file)1213# The marketing team's weekly export, written straight to the catalog: no lineage event.14export = connect(catalog).sql("select learner_id, full_name, email from marketing.audience").to_arrow_table()15catalog.create_table("marketing.campaign_list", schema=export.schema).append(export)16print("rows exported:", f"{len(export):,}")1718events = lineage.read_events(events_file)19reached = lineage.downstream(lineage.graph(events), "crm.learners.email")20print("crm.learners.email reaches:", ", ".join(edge.target for edge in reached))21# the catalog's tables, less the platform's sources and the outputs the events name22print("tables no job's lineage accounts for:", ", ".join(lineage.unaccounted(catalog, events)))
python uninstrumented-job.pyrows exported: 1,829
crm.learners.email reaches: marketing.audience.email
tables no job's lineage accounts for: marketing.campaign_list
이벤트를 보내지 않는 내보내기는 영향 분석에서 보이지 않는다. 이 스크립트는 학습자 1,829명의 이름과 이메일을 복사했지만, crm.learners.email은(는) 여전히 marketing.audience.email에만 도달하는 것으로 나타난다. 따라서 이 리니지를 기준으로 삭제를 진행하면 복사본을 놓치게 된다. 그러나 카탈로그는 해당 테이블이 존재한다는 사실을 알고 있으므로, lib.lineage.unaccounted은(는) 카탈로그의 테이블을 리니지와 비교한다. 모든 테이블은 플랫폼의 소스이거나 리니지가 알고 있는 작업의 출력이어야 한다. 이 검사는 marketing.campaign_list을(를) 찾아낸다.
작업 규칙: 카탈로그를 리니지와 정기적으로 대조하고, 어떤 작업의 리니지로도 설명되지 않는 테이블은 인시던트로 취급한다. 해당 작업을 찾아 이벤트를 보내도록 만든다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Personal data in a table classified internal
Instrumenting the marketing team's export fixes the graph, and exposes a second problem. When the export's owner created marketing.campaign_list, they classified every column internal, because only the marketing team reads it. That's a statement about who reads a table, not what it holds. Wrap the script's work in lib.lineage.record, with the SELECT it runs as its lineage, and compare the declared classifications with lineage's proposal:
1"""The same export, now recorded with lineage, into a table its owner tagged internal. The prose rests on the walk to2the copy and on the check for understated classifications."""34from lib import lineage5from lib.catalog import open_catalog6from lib.platform import scratch7from lib.warehouse import connect89catalog = open_catalog()10events_file = scratch() / "events.jsonl"11lineage.run_all(catalog, events_file)1213select = "select a.learner_id, a.full_name, a.email from marketing.audience a"141516def export() -> None:17 rows = connect(catalog).sql(select).to_arrow_table()18 catalog.create_table("marketing.campaign_list", schema=rows.schema).append(rows)192021# record reads this statement's lineage before export() runs22statement = f"create table marketing.campaign_list as {select}"23lineage.record(lineage.client(events_file), "campaign_export", statement, catalog, export)2425# Its owner tagged every column internal: "only the marketing team reads it".26known = lineage.declared() | {f"marketing.campaign_list.{c}": "internal" for c in ("learner_id", "full_name", "email")}27graph = lineage.graph(lineage.read_events(events_file))28print("crm.learners.email reaches:", ", ".join(edge.target for edge in lineage.downstream(graph, "crm.learners.email")))29for problem in lineage.understated(lineage.propose(graph, known), known):30 print("understated:", problem)
python personal-copy-tagged-internal.pycrm.learners.email reaches: marketing.audience.email, marketing.campaign_list.email
understated: marketing.campaign_list.full_name: declared internal, lineage proposes personal
understated: marketing.campaign_list.email: declared internal, lineage proposes personal
Once the export sends lineage, the walk reaches its copy: crm.learners.email reaches marketing.audience.email, marketing.campaign_list.email, two hops, the second through the export. And lib.lineage.understated reports full_name and email in marketing.campaign_list as declared internal while lineage proposes personal: copies of personal data that any policy reading the declared classification would treat as internal.
Working rule: Run lib.lineage.understated, or your own check like it, on every lineage update, and fail when a declared classification is lower than lineage proposes. Classify by what a table holds, never by who reads it, and fix such a finding by raising the classification, not by dropping the edge.
이 장에서 결정한 것
- 모든 파이프라인 실행은 OpenLineage 실행 이벤트를 보낸다. START 이벤트를 보내고 이어서 COMPLETE 또는 FAIL 이벤트를 보내는 주체는 파이프라인을 실행하는 코드(
lib.lineage.run)이며, 이벤트에는 출력의 스키마와 컬럼 리니지가 담긴다. 데이터셋 이름은 자리 표시용 네임스페이스iceberg://platform아래의 카탈로그 식별자로 짓는다. 예제는 이벤트 시각을 고정하고 명목상의 실행에서 실행 ID를 도출하며, 이벤트의 바이트가 아니라 이벤트가 말하는 내용을 비교한다. - 컬럼 리니지는 sqlglot의 값 리니지를 OpenLineage 하위 유형으로 분류한 것에 INDIRECT 입력을 더한 것이다. CTE를 포함한 모든
SELECT의 조인, 필터, 그룹화, 정렬 컬럼이 패싯의dataset목록에 들어간다. - 올바르게 읽을 수 없는 리니지는 오류다. 비어 있거나 일부만 있는 리니지를 결코 내놓지 않는다.
lib.lineage는CREATE ... AS와INSERT ... SELECT만 읽고, 그 밖의 모든 구문은MERGE를 포함해USING가 무엇을 지정하든 거부한다. 리니지가 비어 있거나, 소스를 특정할 수 없거나, 알 수 없는 컬럼이 있으면 작업은 실행되기 전에 중단된다.INSERT의 컬럼은 카탈로그에 있는 테이블의 스키마에서 가져와 위치로 대응시키며, 개수가 다르면 오류다. - 영향 분석은 이벤트로 만든 그래프를 따라간다. INDIRECT 엣지도 포함하며, 값이 바뀔 수 있는 컬럼과 행이 바뀔 수 있는 테이블을 보고한다.
- 분류는 모든 입력 가운데 더 엄격한 쪽으로 전파된다. DIRECT와 INDIRECT 모두
governance/lineage.toml에 설정한 대로 적용하므로, 개인 정보 컬럼에 대한 필터로 선택된 테이블은 개인 정보로 제안된다. 리니지는 선언된 분류를 결코 낮추지 않으며, 마스킹도 분류를 해제하지 않는다. 분류되지 않은 두 테이블에 대한 제안, 즉full_name,email,body는 개인 정보로, 나머지 모든 컬럼은 내부로 하는 제안은governance/metadata.toml에 들어가기 전에 검토를 위해 각 소유자에게 전달된다. - 세 가지 점검이 리니지 자체를 지킨다. COMPLETE 이벤트 없이 실행된 작업, 어떤 작업의 리니지로도 설명되지 않는 테이블, 리니지가 제안하는 것보다 낮게 선언된 분류가 그 대상이다. 리니지 그래프는 이벤트만큼만 완전하며, 빠진 엣지는 의존하는 것이 없는 컬럼과 똑같이 보인다.
이 판의 이 장
2026.11.31판예제 통과 2026년 11월 19이 장에서 11개 예제개 중 11개가 통과한다
영어판 2026.11.31판을 번역한 것이다.