保留期限、同意与删除

阅读约需 98分钟 分钟,13节中含11个示例。

版次 2026.11.31 示例于 2026年11月19日 通过 108个示例中有108个通过 变化内容

AI 翻译。由人工以英文核验。 我们如何制作图书

译自英文版图书,版次 2026.11.31。 阅读原文

学完本章,你将能把保留规则作为经过评审的配置来维护,让撤回同意在下一次查询时就生效,并把一个人的数据从它流经的每张表中删除。你会针对传入的某一天应用这些规则,通过血缘找到这些表,并且只保留法务团队已记录例外的数据。你还会清除行删除之后 Iceberg 表中留下的内容,包括旧快照、未被引用的文件和列统计信息,最后用一次扫描读取剩余的每个文件,证明其中没有这个人的数据。

本书所治理的在线学习平台,自 2024 年 1 月第一次注册以来,把所有数据都保存在同一个 Iceberg 目录中:每一张支持工单,以及每一条学习事件和它所来自的 IP 地址。没有任何规定说明这两类数据应保留多久。学习者 3590 撤回了对营销邮件的同意,几周后又要求平台删除其数据。前者应当从下一次查询起就改变营销部门所能看到的内容。后者则应当使任何地方都不再留有该学习者的副本:既不在由其数据构建的表中,也不在 Iceberg 特意保留的每张表的旧版本中。

本章的主题由三个术语指称。保留期限是平台对每类数据的保存时长。同意是一个人对其数据某一种用途(例如营销邮件)的认可,这种认可可以撤回。删除是应请求移除一个人的数据。本章为本书的共享代码增加两样东西:平台的保留规则,位于 governance/retention.toml;以及模块 lib.retention,它负责应用这些规则,规划并执行删除,在事后清理并检查文件,还负责维护删除登记册。它使用第 4 章中的 lib.lineage 来查找某个学习者的数据流向了哪里,并使用第 2 章中 lib.metadata 里的分类。每个示例都在本地目录上运行 PyIceberg 0.12.0。

这部分尚未翻译,因此以英文显示。

Keep retention rules in a reviewed file

最近一次修改见于版次 2026.11.27

A retention rule says how long one kind of data is kept, from which date the time counts, and which rows it applies to. The EU's General Data Protection Regulation (GDPR) states the principle as storage limitation: personal data are to be "kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed" (Article 5(1)(e)). Its Recital 39 asks the controller to set time limits for erasure or for a periodic review (GDPR). NIST's catalog of security and privacy controls describes the same job, asking that information be disposed of, destroyed or erased once its retention period ends (control SI-12(3)) (NIST SP 800-53 Rev. 5). The platform's owners and lawyers decide how long is necessary; the code applies what they decide, every time.

That argues for rules as config, reviewed like the access policies, and for code that applies them to a day it is given, not the clock. A job that reads the clock deletes different rows depending on when it runs: a test can't pin its result, and a rerun after a failure can delete more than the run it replaces.

The platform's rules are in governance/retention.toml. The periods are the platform's own choices for this book's case, not legal advice:

[tables."support.tickets"]
keep_days = 730
counted_from = "opened_at"        # tickets record when they were opened, not when they were closed
only_when = { column = "status", equals = "closed" }

[tables."learning.events"]
keep_days = 365
counted_from = "occurred_at"

[tables."billing.payments"]
keep_days = 3650                  # a placeholder for the period the platform's legal team sets for payment records
counted_from = "paid_at"

Closed tickets are kept two years, events one year, and payments 3,650 days, a placeholder that the erasure plan relies on. The tickets rule should count from when a ticket was closed, but the table records only opened_at, so a ticket opened 25 months ago and closed last week goes at once. The platform lives with that because adding a closed_at column is the support team's change, not the platform team's; until they make it, the rule errs toward deleting early. lib.retention.due turns a rule and a day into an Iceberg filter. Counted on the book's "today", Oct 1, 2026, and 90 days later, the rows each rule selects come out like this:

retention-rules-due.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""The rows each retention rule makes due on the book's today and 90 days later, counted against the day the job is given.The `rows due` lines are the counts the prose gives."""from datetime import timedeltafrom lib import retentionfrom lib.catalog import open_catalogfrom lib.platform import TODAYrules = retention.rules()["tables"]catalog = open_catalog(sorted(rules))for name, rule in sorted(rules.items()):    state = rule.get("only_when")    print(f"{name}: kept {rule['keep_days']} days from {rule['counted_from']}"          + (f", once {state['column']} is {state['equals']}" if state else ""))# the book's today, then 90 days later: the day is passed in, never read from the clockfor day in (TODAY, TODAY + timedelta(days=90)):    for name, rule in sorted(rules.items()):        table = catalog.load_table(name)        # the earliest date kept: keep_days before the day        print(f"{name} on {day}, kept from:", retention.cutoff(rule, day))        print(f"{name} on {day}, rows due:", f"{retention.count(table, retention.due(rule, day)):,}")        print(f"{name} on {day}, rows in the table:", f"{retention.count(table):,}")
输出python retention-rules-due.py
billing.payments: kept 3650 days from paid_at
learning.events: kept 365 days from occurred_at
support.tickets: kept 730 days from opened_at, once status is closed
billing.payments on 2026-10-01, kept from: 2016-10-03
billing.payments on 2026-10-01, rows due: 0
billing.payments on 2026-10-01, rows in the table: 17,562
learning.events on 2026-10-01, kept from: 2025-10-01
learning.events on 2026-10-01, rows due: 9,060
learning.events on 2026-10-01, rows in the table: 100,000
support.tickets on 2026-10-01, kept from: 2024-10-01
support.tickets on 2026-10-01, rows due: 58
support.tickets on 2026-10-01, rows in the table: 2,000
billing.payments on 2026-12-30, kept from: 2017-01-01
billing.payments on 2026-12-30, rows due: 0
billing.payments on 2026-12-30, rows in the table: 17,562
learning.events on 2026-12-30, kept from: 2025-12-30
learning.events on 2026-12-30, rows due: 15,545
learning.events on 2026-12-30, rows in the table: 100,000
support.tickets on 2026-12-30, kept from: 2024-12-30
support.tickets on 2026-12-30, rows due: 111
support.tickets on 2026-12-30, rows in the table: 2,000

The rows due depend on the day passed in and the data, and on nothing else. On Oct 1, 2026, the events rule keeps events from 2025-10-01 on, so 9,060 of the 100,000 events are due, and on Dec 30, 90 days later, 15,545 are. The tickets rule makes 58 of the 2,000 tickets due on Oct 1, all closed and opened over two years ago. On Dec 30 it makes 111 due. The payments rule makes 0 of the 17,562 payments due on Oct 1 and 0 on Dec 30.

Working rule: Keep each table's retention period, the column it counts from and the state a row must be in, in a reviewed file, and apply it against a day the scheduler passes in, never the clock. Count from the date the rule means. When the table doesn't record that date, as the platform's tickets don't record when they closed, ask the table's owner to add the column rather than let the rule count from another date for good, and say in the rule's file which way it errs until then.

用行删除清除过期行

最近一次修改见于版次 2026.11.27

有了文件中的规则,保留作业便按规则删除数据。Iceberg 写入器删除行有两种方式,由表属性 write.delete.mode 设定,默认值为 copy-on-write(Iceberg 配置)。写时复制(copy-on-write)会重写每个受影响的数据文件,去掉被删除的行。读时合并(merge-on-read)则不动数据文件,而在其旁写入一个删除文件。规范中有两种删除文件:位置删除,以数据文件及行在其中的位置标记一行(在版本 3 及以上的表中为删除向量);等值删除,以一列或多列的值标记一行(Iceberg 表规范)。PyIceberg 0.12.0 的 Table.delete 始终采用写时复制:即使设置为 merge-on-read,它也只是警告尚不支持读时合并,然后照样重写文件(源码)。无论哪种方式,一次删除都是一次提交,即一个新快照,而此前的快照仍然保留。

lib.retention.apply 对每条规则,删除 due 针对给定日期返回的行,并把该规则的表和日期记录在提交的快照摘要中,因此表自身的历史就能说明这些行为何被删除。对 2026 年 10 月 1 日运行两次,分别读取当前状态和通过时间旅行读取,可以看到一次删除留下了什么,最后是按存续时间使旧快照过期:

apply-retention.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""Applies the retention rules for the book's today as Iceberg row deletes, which leave the old rows in older snapshots.The rerun line is zero, and the time-travel lines show the rows the snapshot before each delete still returns."""from datetime import datetime, timedelta, timezonefrom lib import retentionfrom lib.catalog import open_catalogfrom lib.platform import TODAYrules = retention.rules()["tables"]catalog = open_catalog(sorted(rules))# each table's snapshot from before the deletes, which the time-travel lines readbefore = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in rules}deleted = retention.apply(catalog, TODAY)for name, rows in deleted.items():    print(f"{name}, rows deleted:", f"{rows:,}")print("rows deleted by running again for the same day:", sum(retention.apply(catalog, TODAY).values()))for name in (name for name, rows in deleted.items() if rows):    rule, table = rules[name], catalog.load_table(name)    summary = table.current_snapshot().summary    print(f"{name}, snapshots:", " then ".join(s.summary.operation.value for s in table.snapshots()))    # copy-on-write writes the rows it keeps again as new records, which is what this line counts    print(f"{name}, rows written again to delete the others:", f"{int(summary['added-records']):,}")    print(f"{name}, the commit records:", summary["retention-rule"], "on", summary["retention-day"])    rows_due = retention.due(rule, TODAY)    print(f"{name}, rows due, read now:", f"{retention.count(table, rows_due):,}")    print(f"{name}, rows due, read by time travel:", f"{retention.count(table, rows_due, before[name]):,}")    # Snapshots carry real commit times, so their age is measured on the clock, by the table's own setting.    print(f"{name}, snapshots expired by age today:", retention.expire_by_age(table))    # six days on, the snapshot from before the delete is older than the table's five-day setting    later = datetime.now(timezone.utc) + timedelta(days=6)    print(f"{name}, snapshots expired by age six days later:", retention.expire_by_age(table, later))
输出python apply-retention.py
billing.payments, rows deleted: 0
learning.events, rows deleted: 9,060
support.tickets, rows deleted: 58
rows deleted by running again for the same day: 0
learning.events, snapshots: append then overwrite
learning.events, rows written again to delete the others: 90,940
learning.events, the commit records: learning.events on 2026-10-01
learning.events, rows due, read now: 0
learning.events, rows due, read by time travel: 9,060
learning.events, snapshots expired by age today: 0
learning.events, snapshots expired by age six days later: 1
support.tickets, snapshots: append then overwrite
support.tickets, rows written again to delete the others: 1,942
support.tickets, the commit records: support.tickets on 2026-10-01
support.tickets, rows due, read now: 0
support.tickets, rows due, read by time travel: 58
support.tickets, snapshots expired by age today: 0
support.tickets, snapshots expired by age six days later: 1

重试运行是无害的:同一天的第二次运行删除了 0 行,也没有产生提交。第一次运行删除了 9,060 条事件、58 张工单和 0 笔付款。每个被修改的表都从 append 快照变成了 overwrite 快照,摘要还显示了写时复制的代价:为了删除旧事件,PyIceberg 把其余 90,940 行重新写了一遍,因为该表只是一个未分区的文件。如果按 occurred_at 的月份分区,同样的删除就可以直接丢弃整个文件,当文件的每一行都匹配时,Table.delete 正是这样做的。在 PyIceberg 0.12.0 中,写入按月分区的表需要 pyiceberg-core 额外依赖,本书的示例没有安装它。

删除并不会把这些行从表的早期快照中移除。按现在的状态读取,learning.events 仍有 0 行超出保留期;但通过时间旅行读取,删除之前的快照仍会返回全部 9,060 行,其中包括 IP 地址,而存放这些行的 Parquet 文件仍在磁盘上。lib.retention.expire_by_age 使早于表自身 history.expire.max-snapshot-age-ms(此处为五天)的快照过期,时间以时钟为准。它在删除之后立即使 learning.events 的 0 个快照过期,并在六天后使 1 个快照过期。只有旧快照已过期并且没有快照引用的文件已被移除,保留作业才算完成:这与擦除请求所需的清理相同。

工作准则: 以 Iceberg 行删除的方式应用保留策略,一次一条规则,并把规则和日期记录在每次提交的快照摘要中。使作业对给定日期具有幂等性。按规则起算所依据的列对大表分区,使其删除可以丢弃整个文件而不是重写文件;在 PyIceberg 中,需安装其 pyiceberg-core 额外依赖才能写入这类表。只有在旧快照已过期且其文件已被移除之后,才算作业完成。

最近一次修改见于版次 2026.11.27

数据保留按时间删除行。同意状态的变化决定谁可以使用这些数据,而撤回同意与要求删除并不是同一个事件,因此平台的代码把它们当作两条路径处理。欧盟《通用数据保护条例》(GDPR)第 7(3) 条规定:“数据主体有权随时撤回其同意。撤回同意不影响在撤回之前基于同意而进行的处理的合法性。”(GDPR)。同一条例的第 17(1)(b) 条把撤回同意列为删除的理由之一,但前提是该处理已不再有其他法律依据。在代码中,撤回只修改一个属性,而删除则会删去行。

在平台上,同意状态是访问策略所读取的数据属性,这一点已在第 5 章中设定:市场部的行过滤器只保留 marketing_consent 为真的学员,课程助手的行过滤器只保留作者的 ai_training_consent 为真的评论。因此,撤回同意无需改动策略。CRM 是学员管理账户的应用,它会把新值同步到 crm.learners。市场部随后运行的下一次查询会经过第 6 章构建的、用于改写每条查询的执行层,返回的学员就少了一位。

撤回学员 3590 的营销同意,然后查看市场部看到的内容,以及撤回前、撤回后和营销流水线再次运行后各表中保存的数据。

撤回在下一次查询时生效,并且没有删除任何内容:撤回前市场部看到 1,861 名学员,其中包括学员 3590;撤回后看到 1,860 名,不再包括学员 3590。市场部的行过滤器仍然是 [{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}],crm.learners 中仍保存着 5,000 行。没有任何内容被删除;学员的账户、报名记录和历史记录都保持原样。

问题出在副本上。marketing.audience 是营销流水线每晚构建的名单,它在构建时应用同意过滤器,而策略提供给市场部的这张表本身不带同意过滤器。撤回之后,它仍保存着学员 3590 的 1 行。在有人对该表的列进行分类之前,策略会对所有读取者遮盖这些列;一旦有人完成分类,正如第 4 章所建议的,那么在下一次构建之前,从该表发出的营销活动就会触达已经表示拒绝的人。只有流水线再次运行后,计数才会降到 0。训练语料库的构建同样带有同意过滤器,第 9 章会在训练之前,按当前的同意状态检查每一次构建。

只有真假两个值的列所表达的信息也太少:对于平台询问同意之前就已注册的所有人,ai_training_consent 为空值,而假可能表示拒绝,也可能表示已撤回。数据隐私词汇表(Data Privacy Vocabulary)是 W3C 社区组的报告,并非标准,它列出了同意可能处于的各种状态,其中包括已给予、已拒绝、已撤回、已过期和未知(DPV 2.3)。

工作准则: 把同意状态放在查询运行的地方:放在一张由学员所用系统保持更新的表中,由策略作为行过滤器读取,这样撤回只改变下一次查询的结果,不删除任何内容。如果平台的法务人员认定撤回同意同时也是删除的理由,则还应让它走删除路径。每当同意状态变化时,重建所有用同意过滤器构建的表,或者为这些表的读取者提供同样的过滤器。把每项同意记录为一个状态及其变更时间,并由此推导出策略所读取的真假属性,将未知视为未给予。

依据血缘规划删除

最近一次修改见于版次 2026.11.27

撤回同意只改变一个属性;而删除必须在所有位置找到这个人。GDPR 第 17(1) 条赋予个人在所列理由成立时,要求无不当延迟地删除其数据的权利(GDPR)。平台把一项请求视为涉及所有副本,因此它的首要任务是一个工程问题:列出这个人的数据在哪里,清单分为两半。保存学习者 ID 的源表来自目录:第 2 章已经确定,每一张带有 learner_id 的表,无论其分类如何,都属于每一次删除。由这些表构建出的表来自血缘:第 4 章对列图的遍历,从每张源表出发,可以到达其行所影响过的每一张表。

有些数据要保留。同一法规第 17(3) 条规定了删除不适用的情形,范围限于处理所必需的程度:在其五项之中,有一项是为遵守控制者所须遵守的欧盟或成员国法律规定的法律义务(GDPR)。例外是否适用、适用于哪张表、持续多久,由法务团队决定,绝不由代码决定。代码为这一决定提供唯一的位置 governance/retention.toml,法务团队在其中写下每一项例外及其理由。本书中的理由只是占位文字,因为我们不做任何法律判断:

[erasure.exceptions."billing.payments"]
reason = "PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from."
decided_by = "legal team"         # a role, never a person's name
erase_columns = ["card_number"]   # what the obligation doesn't need; an empty list keeps the whole row

删除时,对于例外表的行,会保留到该表的保留规则将其删除为止,同时立即清除 erase_columns 所列的列:这里是完整的银行卡号,平台的元数据称其“根本不应存储”。lib.retention.exceptions 会拒绝没有理由的例外,也会拒绝针对没有任何规则会使其终结的表所设的例外。

找到一张派生表,并不等于找到了学习者在其中的行。lib.retention.plan 会查看遍历所到达的每张表的入向血缘边,寻找未经改动而复制的列,按 OpenLineage 的说法就是 IDENTITY 边,其来源是学习者的 ID,或其某一行的 ID,例如某条评价的 ID。若存在这样的列,计划就按该列删除。若不存在,则学习者的行只是影响了该表,计划会在源数据被删除之后重新运行该表的作业。计划按请求提出时平台的状态来读取,这已是撤回同意数周之后:crm.learners 保存着学习者 3590 已撤回的同意,而第 4 章中构建分析表、营销表和 AI 表的四条管道此后已经运行,其血缘也已记录。

学习者 3590 的计划每一步打印一行,写明每张表在该处的处理:

erasure-plan.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 · tldextract 5.4.0 通过
"""The steps that erase one learner: the tables with their learner_id column, then the tables lineage says their rows reach,with the exception the legal team recorded. The counts show how many steps each route found; the last two lines are the text the scan searches for and the rows that name the learner."""from lib import pii, retentionLEARNER = 3590catalog, graph = retention.open_platform()           # weeks after the learner withdrew marketing consentsteps = retention.plan(catalog, graph, LEARNER)for step in steps:    # no column is copied from the learner here: the table's own job runs again once its sources are erased    if step.column is None:        print(f"{step.table}: run {retention.job_for(step.table)} again")    elif step.kept:        print(f"{step.table}: keep, under an exception, and erase {', '.join(step.cleared)}")    else:        print(f"{step.table}: delete where {step.column} in {list(step.values)}"              + (f", a copy of {step.via}" if step.via else ""))print("tables in the plan:", len(steps))# a step with a column and no lineage edge came from the table's own learner_id columnprint("tables found by their learner_id column:", sum(1 for step in steps if step.column and not step.via))# found by lineage: a copy of another table's ID, or a rebuild, because the learner's rows only shaped the tableprint("tables found by lineage:", sum(1 for step in steps if step.via or step.column is None))for step in (step for step in steps if step.kept):    print(f"the reason recorded for {step.table}:", step.kept)theirs = retention.rows_of([LEARNER])for name in ("billing.payments", "marketing.audience"):    print(f"the learner's rows in {name}:", retention.count(catalog.load_table(name), theirs))# the values the scan will search for, noted before any row is deletedfound = retention.traces(catalog, steps, LEARNER)# The phone number is printed through lib.pii.masked, which keeps its shape and its last four digits.print("the learner's identifying values, as the scan searches for them:", pii.masked(", ".join(found.texts)))# rows the plan keeps, in any string column, whose text holds one of those valuesnamed = retention.mentions(catalog, steps, found)print("rows the plan doesn't delete whose text names the learner:", sum(named.values()))
输出python erasure-plan.py
billing.payments: keep, under an exception, and erase card_number
crm.learners: delete where learner_id in [3590]
learning.enrollments: delete where learner_id in [3590]
learning.events: delete where learner_id in [3590]
learning.reviews: delete where learner_id in [3590]
support.tickets: delete where learner_id in [3590]
analytics.course_performance: run course_performance again
analytics.learner_activity: delete where learner_id in [3590], a copy of crm.learners.learner_id
marketing.audience: delete where learner_id in [3590], a copy of crm.learners.learner_id
ai.review_corpus: delete where review_id in [1199], a copy of learning.reviews.review_id
tables in the plan: 10
tables found by their learner_id column: 6
tables found by lineage: 4
the reason recorded for billing.payments: PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from.
the learner's rows in billing.payments: 2
the learner's rows in marketing.audience: 0
the learner's identifying values, as the scan searches for them: •••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e
rows the plan doesn't delete whose text names the learner: 0

计划涉及 10 张表:6 张带有 learner_id 列的源表,以及血缘新增的 4 张。按例外规定,计划保留学习者的 2 笔付款。各步骤由列和血缘决定,而不由行决定,所以计划列出了两张不含该学习者任何行的表:support.tickets,因为学习者 3590 从未提交过工单;以及 marketing.audience,其中有 0 行,因为管道在撤回同意之后重建了它。ai.review_corpus 根本没有 learner_id,但血缘表明它的 review_id 是 learning.reviews.review_id 的 IDENTITY 副本,所以计划按该 ID 删除学习者的评价。analytics.course_performance 只保存每门课程的计数和平均值,所以计划从已删除数据的源表重建它。重建只需运行一次管道,因此代码无需判断曾把该学习者计算在内的平均值是否仍属于其数据。

文本可能在计划不删除的行中提到某个人,例如另一位学习者的工单。示例会在每个字符串列中搜索 governance/retention.toml 所列的识别值,并找到 0 行这样的数据。搜索会把每个值转为小写后取前 15 个字符,在文本任意位置进行匹配:这里是 •••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e,打印时电话号码已被掩码,截取为 15 个字符的原因见最终扫描的说明。拼写方式不同的姓名,或不带分隔符的电话号码,则需要第 3 章中的检测器。

工作准则: 每一次删除都依据目录和血缘图来规划,而不是依据某人维护的清单:先是每一张带有此人 ID 列的表,再是血缘表明其行所到达的每一张表。对派生表,按血缘显示为 ID 未经改动的副本的列来删除,并重建那些只是受此人的行影响的派生表。把例外放在配置中,由法务团队连同理由写下,让计划读取并展示它们;绝不让代码来决定例外。先运行第 4 章中针对没有任何作业血缘可解释的表的检查,因为计划无法触及血缘没有显示的副本。

这部分尚未翻译,因此以英文显示。

Erase the rows the plan lists

最近一次修改见于版次 2026.11.27

The plan says where learner 3590's data is, and lib.retention.erase carries it out in the plan's order. It deletes each step's rows with Table.delete, sources first, writes the rows an exception keeps again with the exception's columns empty, and runs the job of each table the plan rebuilds once the sources are done. Every commit records the request's ID, erasure-0001, in its snapshot summary. Iceberg keeps a summary in the table's metadata files after its snapshot expires, in all of them by default, or only the last write.metadata.previous-versions-max (100 by default) once write.metadata.delete-after-commit.enabled is set (Iceberg maintenance, Iceberg configuration). So the summary holds only the request's ID, which the erasure register pairs with a keyed hash of the learner's ID, and never the learner's email.

The erasure runs on learner 3590's plan under one request ID; reading crm.learners and learning.events now and by time travel shows what it leaves:

erase-learner.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""Carries out the erasure plan with Iceberg row deletes. The current snapshots no longer hold the learner, but the snapshotsbefore them still do, and the `read by time travel` lines show it."""from lib import retentionLEARNER = 3590catalog, graph = retention.open_platform()steps = retention.plan(catalog, graph, LEARNER)# each table's snapshot from before the erasure, which the time-travel lines readbefore = {step.table: catalog.load_table(step.table).current_snapshot().snapshot_id for step in steps}done = retention.erase(catalog, steps, "erasure-0001")       # the request's ID goes into every commit's summaryfor table, rows in done.items():    # a rebuilt table reports 'rebuilt' rather than a count of rows    print(f"{table}, rows deleted:" if rows != "rebuilt" else f"{table}:", rows)print("tables one row was deleted from:", ", ".join(table for table, rows in done.items() if rows == 1))theirs = retention.rows_of([LEARNER])payments = catalog.load_table("billing.payments").scan(row_filter=theirs).to_arrow()print("billing.payments, the learner's rows kept:", payments.num_rows)# the exception erases card_number to null, so the values that are not null are the card numbers leftprint("billing.payments, card numbers left in them:", payments.num_rows - payments["card_number"].null_count)for name in ("crm.learners", "learning.events"):    table = catalog.load_table(name)    print(f"the learner's rows in {name}, read now:", retention.count(table, theirs))    print(f"the learner's rows in {name}, read by time travel:", retention.count(table, theirs, before[name]))summary = catalog.load_table("crm.learners").current_snapshot().summary# copy-on-write writes the rows it keeps again as new records, so this counts every other row of the fileprint("crm.learners, rows written again to delete one:", f"{int(summary['added-records']):,}")print("crm.learners, the commit records:", summary["erasure-request"])# snapshots of tables the plan deleted from (not rebuilt, not kept) that still return a row the erasure deletedholding = [(table, snapshot) for step in steps if step.column is not None           for table in [catalog.load_table(step.table)] for snapshot in table.snapshots()           if not step.kept and retention.count(table, retention.rows(step), snapshot.snapshot_id)]print("snapshots that still hold the rows the erasure deleted:", len(holding))print("of them current:", sum(snapshot == table.current_snapshot() for table, snapshot in holding))
输出python erase-learner.py
billing.payments, rows deleted: 0
crm.learners, rows deleted: 1
learning.enrollments, rows deleted: 2
learning.events, rows deleted: 19
learning.reviews, rows deleted: 1
support.tickets, rows deleted: 0
analytics.course_performance: rebuilt
analytics.learner_activity, rows deleted: 1
marketing.audience, rows deleted: 0
ai.review_corpus, rows deleted: 1
tables one row was deleted from: crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus
billing.payments, the learner's rows kept: 2
billing.payments, card numbers left in them: 0
the learner's rows in crm.learners, read now: 0
the learner's rows in crm.learners, read by time travel: 1
the learner's rows in learning.events, read now: 0
the learner's rows in learning.events, read by time travel: 19
crm.learners, rows written again to delete one: 4,999
crm.learners, the commit records: erasure-0001
snapshots that still hold the rows the erasure deleted: 6
of them current: 0

The erasure took the learner out of every table in the plan holding any. It deleted 19 of the learner's events, 2 enrollments, and one row each from crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus. It deleted nothing from support.tickets and marketing.audience, which held none, and it rebuilt analytics.course_performance.

billing.payments keeps the learner's 2 payments, with 0 card numbers left in them. Read now, crm.learners holds 0 rows for the learner. Read by time travel to the snapshot before the erasure, it returns 1, with name, email, phone and address, and learning.events returns all 19 of the learner's events the same way.

To delete one row of crm.learners, PyIceberg wrote the other 4,999 into a new file, as copy-on-write does, and the old file, with the learner in it, belongs to the snapshot before.

Across the plan's tables, 6 snapshots still hold the rows the erasure deleted, and 0 of them are current. That is what snapshots are for: a reader can go back to any of them. For an erasure, it means the delete took the learner out of the table but not out of storage.

Working rule: Carry out an erasure as row deletes in the plan's order, and record only a request ID in each commit, never a value of the person's. Treat the delete as the first of three steps: the snapshots before it still hold the person, and anyone allowed to time travel can read them until they're expired.

这部分尚未翻译,因此以英文显示。

Expire snapshots by the table's own settings

最近一次修改见于版次 2026.11.27

After the erasure's deletes, learner 3590's rows live on in the snapshots before them. The Iceberg spec says snapshots expire so that the data files they removed or replaced can be physically deleted (Iceberg table spec). Each table sets how in its properties (Iceberg configuration): gc.enabled allows garbage collection, such as expiring snapshots and removing orphan files; history.expire.min-snapshots-to-keep is how many snapshots to keep however old they are; and history.expire.max-snapshot-age-ms is the age at which the rest may go. In Iceberg's Java library, regularly expiring snapshots deletes unused data files (Iceberg maintenance).

PyIceberg 0.12.0 does less. Its table.maintenance.expire_snapshots() commits one change, which removes the snapshots from the table's metadata and deletes no file (source), though its documentation offers it to "clean up table metadata and reduce storage costs" (PyIceberg API). Nothing in it reads gc.enabled, and only its command-line tool reads the history.expire settings. An issue asks for expiry to delete files too (issue 2604). Two copies of crm.learners each have 3 snapshots, the load and two deletes. One copy has gc.enabled=false, the other history.expire.min-snapshots-to-keep=3, and each is expired two ways: by age with PyIceberg, older_than the current time, and with lib.retention.expire, which reads the two settings first.

expiry-ignores-table-settings.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""PyIceberg's snapshot expiry ignores two of the table's own settings, gc.enabled and min-snapshots-to-keep, so it expiressnapshots they would keep. The book's expiry reads both first, and the `left by PyIceberg's expiry` lines show the difference."""from datetime import datetime, timezonefrom pyiceberg.expressions import EqualTofrom lib import retentionfrom lib.catalog import open_catalogdef learners_with(settings: dict):    """crm.learners with three snapshots: the load, then two deletes."""    table = open_catalog(["crm.learners"]).load_table("crm.learners")    with table.transaction() as transaction:        transaction.set_properties(settings)    # two deletes, so there are snapshots from before them for the expiry to consider    for learner in (3590, 3591):        table.delete(EqualTo("learner_id", learner))    return tablefor settings in ({"gc.enabled": "false"}, {"history.expire.min-snapshots-to-keep": "3"}):    setting = ", ".join(f"{key}={value}" for key, value in settings.items())    table = learners_with(settings)    print(f"{setting}, snapshots:", len(table.snapshots()))    try:        print(f"{setting}, the book's expiry expires:", retention.expire(table))    except retention.RetentionError as error:        print(f"{setting}, the book's expiry refuses:", error)    # PyIceberg's expiry by age, as of now: it reads neither setting    table.maintenance.expire_snapshots().older_than(datetime.now(timezone.utc)).commit()    print(f"{setting}, snapshots left by PyIceberg's expiry:", len(table.snapshots()))
输出python expiry-ignores-table-settings.py
gc.enabled=false, snapshots: 3
gc.enabled=false, the book's expiry refuses: crm.learners has gc.enabled=false, which turns off garbage collection for the table
gc.enabled=false, snapshots left by PyIceberg's expiry: 1
history.expire.min-snapshots-to-keep=3, snapshots: 3
history.expire.min-snapshots-to-keep=3, the book's expiry expires: 0
history.expire.min-snapshots-to-keep=3, snapshots left by PyIceberg's expiry: 1

PyIceberg's expiry ignores both settings. It left 1 of the first table's 3 snapshots, and 1 of the second's. older_than expires every snapshot older than the time it's given except those a branch or tag points at, here only the current one.

The book's expiry refused the first table, crm.learners has gc.enabled=false, which turns off garbage collection for the table, and expired 0 snapshots of the second. An owner sets these properties for reasons the expiring code can't see, and in PyIceberg they are only advice.

For an erasure, both settings mean the job can't finish: a table that keeps a snapshot from before the delete still holds the person. A job that stopped at that table would already have deleted rows from the tables before it and left the rest unexpired, so the check belongs before the first delete. lib.retention.erase_everywhere runs lib.retention.blocked on every table in the plan first: garbage collection off, more than one snapshot to keep, or a branch or tag besides main. If any table fails, the job refuses to start and names every such table.

Here crm.learners has garbage collection off, and learning.events keeps three snapshots. Then the job and the scan run:

erasure-refuses-to-start.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""The erasure job checks every table in its plan before it changes any, and refuses to start when a table's own settingswould keep the learner. The scan still runs, from the traces the refusal carries, and finds the learner in the files."""from datetime import timedeltafrom lib import retentionLEARNER = 3590catalog, graph = retention.open_platform()# each setting stops expiry from removing the snapshot from before the delete: garbage collection off, or three snapshots keptfor name, settings in (("crm.learners", {"gc.enabled": "false"}),                       ("learning.events", {"history.expire.min-snapshots-to-keep": "3"})):    with catalog.load_table(name).transaction() as transaction:        transaction.set_properties(settings)# every table's current snapshot, to count the tables the job changesbefore = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in retention.tables(catalog)}try:    # min_age of zero: one writer and nothing in flight, so no file needs a minimum age to be safe to sweep    steps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0001", min_age=timedelta(0))except retention.ErasureRefused as refusal:    for name, why in refusal.tables.items():        print(f"the job refuses, because of {name}:", why)    found = refusal.found# a table the job changed has a new current snapshotprint("tables the job changed:", sum(catalog.load_table(name).current_snapshot().snapshot_id != snapshot                                     for name, snapshot in before.items()))report = retention.scan(catalog, found)print("tables whose files hold the learner:", sum(1 for _, holding in report.values() if holding))print("files holding the learner:", sum(holding for _, holding in report.values()))
输出python erasure-refuses-to-start.py
the job refuses, because of crm.learners: gc.enabled=false turns off garbage collection for the table
the job refuses, because of learning.events: history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete
tables the job changed: 0
tables whose files hold the learner: 7
files holding the learner: 8

The job refuses. It names both tables, crm.learners because gc.enabled=false turns off garbage collection for the table, and learning.events because history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete, and 0 tables changed.

The scan runs either way, from the traces the refusal carries, and finds the learner in 8 files across 7 tables. A person then decides with the tables' owners, rather than code overriding an owner's setting.

Working rule: Expire snapshots through one function of your own that reads gc.enabled and the history.expire settings before it expires anything, and refuses when garbage collection is off, because PyIceberg's expiry reads neither; check again at every PyIceberg release. Before an erasure deletes anything, check every table in its plan the same way, and refuse the whole erasure when one would keep the person. For an erasure, expire every snapshot older than the delete. For retention, expire by age, as history.expire.max-snapshot-age-ms says, measured on the clock, since snapshots carry real commit times.

删除没有任何快照引用的文件

最近一次修改见于版次 2026.11.31

旧快照过期之后,只被它们引用的文件就成了未被引用的文件。Iceberg 把未被引用的文件称为孤儿文件(orphan):它位于表的存储位置之下,却不在任何快照的元数据中。Iceberg 的 Java 库用一个单独的操作来删除这些文件,并警告说,若删除孤儿文件时所用的保留间隔短于任何写入完成所需的预期时间,则是危险的;其默认间隔为 3 天(Iceberg maintenance)。PyIceberg 0.12.0 没有这样的操作。其发布说明中没有列出任何此类操作(release notes)。请求添加该操作的议题被项目的过期机器人关闭,并标记为不计划实现(issue 1200)。两个添加该操作的拉取请求未经合并即被关闭(1958、3358)。它唯一会删除数据文件的调用是 purge_table,而它删除的是整张表(source)。

因此,lib.retention 自行删除这些文件,依据是 PyIceberg 以 table.inspect 形式提供的元数据表所显示的、一张表所引用的内容。lib.retention.referenced 收集以下各项:元数据文件(包括当前的和已记录在日志中的)、每个快照的清单列表(manifest list)、每个清单(manifest,清单列出数据文件及每个文件的列统计信息)、任一快照中有效的每个数据文件和删除文件,以及所有统计信息文件。表的存储位置之下的其余文件都是未被引用的,lib.retention.sweep 会删除其中早于调用方所传最小年龄的那些。governance/retention.toml 把该年龄设为 72 小时,即 Java 的默认值,由 lib.retention.min_file_age 读取。删除个人数据之后,计划涉及的各表先做过期处理,再先用该年龄、后不设年龄做清理,并逐个文件检查其中是否还有该学习者:

sweep-unreferenced-files.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""After the erasure, expire the old snapshots, then remove the files no snapshot references. PyIceberg's expiry deletes nofile and has no orphan-file removal, so lib.retention.sweep does it with the inspect tables; the last lines show the manifest entries still holding the learner's email."""from datetime import timedeltafrom lib import retentionLEARNER = 3590catalog, graph = retention.open_platform()steps = retention.plan(catalog, graph, LEARNER)found = retention.traces(catalog, steps, LEARNER)retention.erase(catalog, steps, "erasure-0001")tables = [catalog.load_table(step.table) for step in steps]print("snapshots expired:", sum(retention.expire(table) for table in tables))print("files under the plan's tables:", sum(len(retention.files_under(table.location())) for table in tables))print("files no metadata references:", sum(len(retention.unreferenced(table)) for table in tables))print("files holding the learner, after expiry:", sum(held for _, held in retention.scan(catalog, found).values()))# the configured minimum age, 72 hours, from governance/retention.tomlprint("files swept with the configured minimum age:", sum(retention.sweep(t, retention.min_file_age()) for t in tables))# One writer, nothing in flight: no file is too young to remove, so the examples sweep with no minimum age.print("files swept with no minimum age:", sum(retention.sweep(table, timedelta(0)) for table in tables))print("files holding the learner, after the sweep:", sum(held for _, held in retention.scan(catalog, found).values()))for table in tables:    # every manifest entry, including those of deleted files, with the bounds it keeps for the email column    entries = table.inspect.entries()    for status, metrics in zip(entries["status"].to_pylist(), entries["readable_metrics"].to_pylist()):        bounds = metrics.get("email") or {}        # bounds are compared lowercased: the learner's email is typed in capitals in the manifest        kept = (bounds.get("lower_bound"), bounds.get("upper_bound"))        if any(text in bound.lower() for bound in kept if bound for text in found.texts):            print(f"{'.'.join(table.name()[-2:])}, manifest entry with status {status}, email bounds:",                  bounds["lower_bound"], "to", bounds["upper_bound"])
输出python sweep-unreferenced-files.py
snapshots expired: 8
files under the plan's tables: 100
files no metadata references: 28
files holding the learner, after expiry: 9
files swept with the configured minimum age: 0
files swept with no minimum age: 28
files holding the learner, after the sweep: 1
crm.learners, manifest entry with status 2, email bounds: ALEXA590@POST.EX to zofia.haase@exan

过期处理删除的是快照,而不是文件。在计划涉及的各表之下的文件中,有 28 个未被任何元数据引用,有 9 个含有该学习者。按配置的 72 小时进行清理,删除的未被引用文件为 0 个,因为所有文件都是几秒钟前才写入的。这些示例传入的最小年龄为零,因为它们的目录只有一个进程在写入。不设最小年龄的清理会删除全部未被引用的文件,而仍有 1 个文件含有该学习者:这是一个清单,crm.learners 的当前快照引用着它。

规范说明了原因:文件被替换或删除时,其清单条目会记录删除该文件的快照以及状态 2(已删除)(Iceberg table spec),而这个条目保留着被删除文件的列统计信息。默认情况下,在指标模式 truncate(16) 下(Iceberg configuration),这些信息包括每个字符串列的最小值(截断为 16 个字符)和最大值(截断为 16 个字符,且最后一个字符加一,使其仍然是文件中所有值的上界)(source)。crm.learners 中已删除的条目保留着电子邮件的上下界 ALEXA590@POST.EX to zofia.haase@exan。前者是学习者 3590 的电子邮件,以大写输入,在平台上所有其他电子邮件之前排序。后者是学习者 4998 的,是平台上最大的,其第 16 个字符 m 变成了 n。没有哪个学习者能幸免:每个数据文件的条目都保留着每一列的最小值和最大值,所以由许多小文件构成的表会暴露许多学习者。

工作准则: 每次过期之后都要删除未被引用的文件,由于 PyIceberg 没有现成的操作,就用检查表自行构建清理,并在生产环境中保留以天计的最小年龄,使正在进行的写入保住它的文件。只有在单个进程写入的场合才使用零年龄。不要因为没有数据文件含有某人,就认为此人已被擦除:清单中可能仍保留着他们的值。

这部分尚未翻译,因此以英文显示。

Keep personal values out of manifests

最近一次修改见于版次 2026.11.27

The sweep left the learner's email in a manifest, as a column bound. Iceberg lets a table choose, per column, what its manifests keep, with the property write.metadata.metrics.column.<name>: none keeps nothing, counts keeps value and null counts, truncate(n) adds bounds cut to n characters, and full adds whole bounds (Iceberg configuration). PyIceberg 0.12.0 looks the property up by the column's name when it writes a file (source).

Typed by hand, those properties would drift from the classifications. lib.retention.properties generates them from governance/metadata.toml and the [statistics] section of governance/retention.toml: none for every column classified personal or sensitive, and for every column nobody has classified, the same fail-closed default the access policy uses. It adds the history.expire settings from the rules' [snapshots] section. lib.retention.apply_properties sets them on every table and rewrites each table's data once, because a metrics mode applies only to the files written after it's set; expiry and a sweep then remove the files from before. Learner 4998, whose email is the largest bound in crm.learners' manifest, is the test case, erased on two platforms, one with default statistics and one with the generated properties:

statistics-off-for-personal-columns.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""Generates each table's properties from the classifications, so manifests keep no bounds on personal columns, then erasesthe learner whose email is the platform's largest, on a platform with the properties and on one without. The `files holding` lines are the proof."""from datetime import timedeltafrom lib import retentionLEARNER = 4998for statistics_off in (False, True):    catalog, graph = retention.open_platform(statistics_off)    label = "statistics off for protected columns" if statistics_off else "default statistics"    if statistics_off:        generated = retention.properties(catalog)        # the per-column metrics settings; the history.expire settings are not counted here        turned_off = [key for wanted in generated.values() for key in wanted if key.startswith(retention.METRICS)]        print("columns with statistics turned off:", len(turned_off))        # tables with no entry in governance/metadata.toml, which the generated properties still cover        unclassified = [name for name in generated if name not in retention.metadata.load()]        print("of them in tables nobody has classified:",              sum(key.startswith(retention.METRICS) for name in unclassified for key in generated[name]))        print("crm.learners, properties set:", len(generated["crm.learners"]))        for key, value in sorted(generated["crm.learners"].items()):            print(f"  {key} = {value}")    # crm.learners has one data file, so its one manifest entry keeps the table's email bounds    entry = catalog.load_table("crm.learners").inspect.entries()["readable_metrics"].to_pylist()[0]    print(f"{label}, crm.learners email bounds:", entry["email"]["lower_bound"], "to", entry["email"]["upper_bound"])    columns = retention.metadata.load()["crm.learners"].columns.values()    personal = [column.name for column in columns if column.classification == "personal"]    print(f"{label}, crm.learners personal columns:", len(personal))    with_bounds = sum(entry[name]["lower_bound"] is not None for name in personal)    print(f"{label}, of them with bounds in the manifest:", with_bounds)    steps, found, _ = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0002", min_age=timedelta(0))    print(f"{label}, files holding learner {LEARNER} after erasure, expiry and sweep:",          sum(held for _, held in retention.scan(catalog, found).values()))
输出python statistics-off-for-personal-columns.py
default statistics, crm.learners email bounds: ALEXA590@POST.EX to zofia.haase@exan
default statistics, crm.learners personal columns: 7
default statistics, of them with bounds in the manifest: 7
default statistics, files holding learner 4998 after erasure, expiry and sweep: 1
columns with statistics turned off: 24
of them in tables nobody has classified: 10
crm.learners, properties set: 9
  history.expire.max-snapshot-age-ms = 432000000
  history.expire.min-snapshots-to-keep = 1
  write.metadata.metrics.column.city = none
  write.metadata.metrics.column.date_of_birth = none
  write.metadata.metrics.column.email = none
  write.metadata.metrics.column.full_name = none
  write.metadata.metrics.column.notes = none
  write.metadata.metrics.column.phone = none
  write.metadata.metrics.column.street_address = none
statistics off for protected columns, crm.learners email bounds: None to None
statistics off for protected columns, crm.learners personal columns: 7
statistics off for protected columns, of them with bounds in the manifest: 0
statistics off for protected columns, files holding learner 4998 after erasure, expiry and sweep: 0

With default statistics, 1 file still holds learner 4998 after the erasure, expiry and sweep, through a raised upper bound. The manifest keeps bounds for 7 of the table's personal columns, and its email bounds are ALEXA590@POST.EX to zofia.haase@exan.

Turning statistics off for personal columns is what keeps learner 4998 out of the files. The email bounds are None to None, the manifest keeps bounds for 0 of the table's 7 personal columns, and 0 files hold the learner after the erasure, expiry and sweep.

The generated properties turn statistics off for 24 columns, 10 of them in marketing.audience and ai.review_corpus, the two tables that chapter 2 found with no metadata. crm.learners gets 9 properties: the two expiry settings, and none for each personal column.

The cost is pruning. Iceberg uses the bounds in manifests to skip files a query's filter can't match (Iceberg table spec), so without them, a query that filters on email reads every file. The platform filters and erases by learner_id, which is internal and keeps its statistics. counts is the other choice: it keeps the counts that help a planner, and no values. And because PyIceberg 0.12.0 keeps the property under the column's old name on a rename, leaving the new name without it, the properties are generated again on every schema change, as chapter 2 asks of every view of the metadata keyed by name.

Working rule: Generate per-column metrics properties from your classifications, none or counts for every personal, sensitive or unclassified column, and apply them when a table is created or, for an existing table, with one rewrite followed by expiry and a sweep. Keep statistics on the ID columns you filter and erase by. Generate the properties again after every schema change.

这部分尚未翻译,因此以英文显示。

Prove the erasure by reading every file

最近一次修改见于版次 2026.11.31

Each step so far removed something the one before left behind, and left something of its own: the delete, the expiry, the sweep and the statistics settings. That is the case for checking the result rather than trusting the steps. The European Data Protection Board's report on how controllers handle erasure, adopted in February 2026, found backups, and anonymisation used in place of deletion, among the weak points, and recommends that controllers "Verify that erasure has been carried out and be able to demonstrate such erasure" (EDPB, 2026).

lib.retention.scan reads every file under the catalog's warehouse, each with its own format's reader, and looks for the learner's traces. lib.retention.traces notes the traces before the erasure: the IDs of every row the plan deletes, by column, and the first 15 characters, lowercased, of each identifying value. Fifteen, because a manifest keeps a largest value's first 15 characters unchanged and raises the 16th by one, so a search for 16 misses a person whose value is a file's largest, as learner 4998's email is. A Parquet file is read as rows. A manifest or manifest list, both Avro, is read record by record with PyIceberg's Avro reader: every string in it, bounds included, and the bounds of each ID column, decoded by the schema the manifest carries. A metadata file is read as text. In a table an exception keeps, the learner's IDs are expected, so there the scan looks only for values, including those of the columns the exception erased. lib.retention.erase_everywhere runs the whole job: plan, note the traces, check every table, erase, expire, sweep. On the platform with the generated properties, the whole job runs and ends with the scan:

verify-erasure.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""Runs the whole erasure job on the platform with statistics off for protected columns, then scans every file under thewarehouse. The `files holding the learner` line is the proof; the rest is evidence of IDs and counts, never the learner's values."""from datetime import timedeltafrom lib import retentionLEARNER, REQUEST = 3590, "erasure-0001"catalog, graph = retention.open_platform(statistics_off=True)# min_age of zero: one writer, so nothing is in flight that a sweep could takesteps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))report = retention.scan(catalog, found)for table, (files, holding) in report.items():    print(f"{table}: {files} files read, {holding} holding the learner")print("request:", REQUEST)print("tables visited:", len(steps))# each table the job deleted from has a count of rows; the one it reran has 'rebuilt' insteadprint("rows deleted:", sum(rows for rows in done.values() if rows != "rebuilt"))print("tables rebuilt:", ", ".join(table for table, rows in done.items() if rows == "rebuilt"))print("tables kept under an exception, columns erased:",      "; ".join(f"{step.table}, {', '.join(step.cleared)}" for step in steps if step.kept))# every snapshot but the current one: the ones a reader can still time travel toprint("snapshots a reader can still time travel to:",      sum(len(catalog.load_table(step.table).snapshots()) - 1 for step in steps))print("files no metadata references:",      sum(len(retention.unreferenced(catalog.load_table(name))) for name in retention.tables(catalog)))print("tables missing their generated properties:", ", ".join(retention.missing_properties(catalog)))print("files read:", sum(files for files, _ in report.values()))print("files holding the learner:", sum(holding for _, holding in report.values()))
输出python verify-erasure.py
ai.review_corpus: 11 files read, 0 holding the learner
analytics.course_performance: 5 files read, 0 holding the learner
analytics.learner_activity: 11 files read, 0 holding the learner
billing.payments: 12 files read, 0 holding the learner
crm.instructors: 8 files read, 0 holding the learner
crm.learners: 11 files read, 0 holding the learner
learning.courses: 8 files read, 0 holding the learner
learning.enrollments: 11 files read, 0 holding the learner
learning.events: 11 files read, 0 holding the learner
learning.reviews: 11 files read, 0 holding the learner
marketing.audience: 8 files read, 0 holding the learner
support.tickets: 8 files read, 0 holding the learner
request: erasure-0001
tables visited: 10
rows deleted: 25
tables rebuilt: analytics.course_performance
tables kept under an exception, columns erased: billing.payments, card_number
snapshots a reader can still time travel to: 0
files no metadata references: 0
tables missing their generated properties: analytics.course_performance
files read: 115
files holding the learner: 0

The scan finds the learner in 0 of the 115 files it reads, and no reader can time travel to the learner, since 0 older snapshots remain, and 0 files are unreferenced. The lines after the per-table counts are the evidence: the request's ID, the tables visited, the rows deleted, the table the job rebuilt, the table an exception keeps and the column erased from it, billing.payments, card_number, with the reason in governance/retention.toml, and the scan's counts. They name no value of the learner's, so keeping them puts none back, which chapter 8 relies on when it writes them to the audit log.

One line is a warning: analytics.course_performance lacks its generated properties, because lib.pipelines.run dropped the table and created it again without them. That table has no personal columns. The next table a pipeline rebuilds might have them, and the scan would then report its manifests' bounds as a failed erasure, so the check belongs after every pipeline run.

IDs can stay in a manifest too, since the ID columns keep their statistics. An erased ID that was a file's smallest or largest, such as learner 5000's, stays in the deleted entry's bounds, and the scan reports it. One more commit to the table, followed by an expiry and a sweep, removes it, because PyIceberg's next append leaves out a manifest that holds only deleted files (source).

The scan covers the warehouse only. Exports, backups, the source systems and logs outside it can hold the learner too, and the same traces can be searched for in each the platform can read.

Working rule: End every erasure with a scan of every file the platform stores, for the person's IDs and identifying values, read with each format's own reader, and keep its result, counts and IDs only, as the evidence. Search for as much of each value as every bound keeps unchanged, which is one character less than the truncation length. Treat any file that still holds the person as a failed erasure, not a warning. Search the stores outside the warehouse that you control, and list in the record the ones you can't search.

生产环境中会出什么问题

最近一次修改见于版次 2026.11.27

删除操作一旦成功,无论数据是已经消失还是仍留在存储中,看上去都是一样的,擦除失败正是这样发生的。这里的第一种失败,会撤销一次已经完成的擦除。

被擦除的学员又回来了

最近一次修改见于版次 2026.11.27

除了 CRM 的同步之外,数据仓库还会每晚根据应用的导出数据重新加载整个 crm.learners,而这次擦除只在数据仓库中执行。请按计划所述,在所有位置擦除学员 3590,然后让夜间重新加载以及读取该表的各条管道运行:

erased-learner-returns.py Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 通过
"""An erased learner comes back: the nightly reload of crm.learners from the application's export brings the rows back, andthe pipelines rebuild from them. A check against the erasure register catches the rows."""from datetime import timedeltafrom lib import pipelines, retentionLEARNER, REQUEST = 3590, "erasure-0001"catalog, graph = retention.open_platform()retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))register = [retention.register_entry(REQUEST, LEARNER)]      # the request's ID and a keyed hash of the learner's IDprint("the erasure register's entry:", register[0]["request"], register[0]["learner"])learners = catalog.load_table("crm.learners")print("the learner's rows in crm.learners after the erasure:", retention.count(learners, retention.rows_of([LEARNER])))# the nightly reload: the application's export still holds the learner, since nobody erased it therelearners.overwrite(retention.crm_export())# the pipelines that read crm.learners rebuild from the reloaded tablefor name in ("learner_activity", "marketing_audience"):    pipelines.run(catalog, name)for table in ("crm.learners", "analytics.learner_activity", "marketing.audience"):    print(f"the learner's rows in {table} after the nightly run:",          retention.count(catalog.load_table(table), retention.rows_of([LEARNER])))# hashes each learner_id in every table and looks it up in the register, so no learner's ID is printedfor table, rows in retention.erased_rows(catalog, register).items():    print(f"the check against the register, rows of erased learners in {table}:", rows)
输出python erased-learner-returns.py
the erasure register's entry: erasure-0001 c25d8a8b12426596f0c0b1ebfb890246b42ea535791f07256efd16fc6c2cb84b
the learner's rows in crm.learners after the erasure: 0
the learner's rows in crm.learners after the nightly run: 1
the learner's rows in analytics.learner_activity after the nightly run: 1
the learner's rows in marketing.audience after the nightly run: 0
the check against the register, rows of erased learners in analytics.learner_activity: 1
the check against the register, rows of erased learners in crm.learners: 1

只在数据仓库中完成的擦除无法持久。擦除之后,crm.learners 中该学员有 0 行;夜间运行之后,它又变为 1 行,而由它重建的 analytics.learner_activity 则有 1 行。marketing.audience 之所以保持为空,只是因为该学员的同意状态重新加载后仍为已撤回,并且它所关联的该学员的报名记录已被擦除。

能发现这一问题的检查会维护一份擦除登记表:对每个请求,只记录其 ID 和被擦除学员 ID 的带密钥哈希(HMAC-SHA256),不记录其他任何内容。密钥是擦除作业的机密,保管方式与第 6 章中执行层的假名化密钥相同,但必须使用另一把密钥,因为持有该密钥的人可以对每个学员 ID 做哈希,进而查出登记表指向的是谁。在每次加载、恢复和重建之后,lib.retention.erased_rows 会对每个含有该列的表中的每个 learner_id 做哈希并在登记表中查找,因此恢复或回填的数据也能得到检查,而登记表中无需保存任何人的 ID。它在 analytics.learner_activity 和 crm.learners 中找到了该学员,而略过了 billing.payments,因为例外规定保留了该表的行。审计日志中关于这次擦除的记录见第 8 章,其中只含有相同的请求 ID 和计数,不含学员本人信息,因此登记表是请求与学员唯一相遇的地方,且以哈希形式出现。我们将 ID 的带密钥哈希视为假名化数据,而非匿名化数据。

工作准则: 先在源系统中擦除,或在同一请求中一并擦除,并维护一份已擦除 ID 的带密钥哈希登记表,每次加载都据此过滤,每次运行都据此检查,密钥作为机密与数据分开保管。将含有已擦除 ID 的行视为事故,如同对待泄露一样。

其他故障

最近一次修改见于版次 2026.11.27
  • 旧快照和旧文件仍保留“已删除”的数据。 保留或擦除作业如果在删除之后就停止,所有数据仍可通过时间旅行读取;在 PyIceberg 中,快照过期之后这些文件也仍然留在存储中。应在同一个作业中运行快照过期和清扫,然后再扫描。
  • 其他引擎以读时合并(merge-on-read)方式删除。 PyIceberg 能够读取其他引擎写入的删除向量(发布说明)。以这种方式删除的行会一直留在原数据文件中,直到压缩重写这些文件;而且删除文件可能保留这些行的值,因为两种删除文件都可以编码被删除行的值(Iceberg 表规范)。应在快照过期和清扫之前压缩受影响的文件,并且同时扫描删除文件。
  • 备份会保留清扫所删除的一切内容。 备份是EDPB 报告指出的薄弱环节之一。应与负责备份的人员商定备份如何擦除或到期清除,并将结论记录在每一项请求中。
  • 无人擦除的衍生表。 不发送血缘信息的作业会写出一份计划无法看到的副本,就像第 4 章中未接入检测的导出那样。每次擦除之前都要运行 lib.lineage.unaccounted。
  • 重建的表会丢失其属性。 先删除再创建表的流水线会丢失生成的统计设置,下一次擦除就会把值留在清单中。每次运行之后都要重新应用这些属性,或者在创建表时就带上这些属性。
  • 重命名的列会丢失其设置。 在 PyIceberg 中,指标属性仍然保留在该列的旧名称之下,因此重命名之后新名称没有对应的设置。每次模式变更时都要重新生成这些属性。
  • 保留期从错误的日期起算。 如果规则的含义是“关闭之后”,却从开立日期起算,就会过早删除;如果从加载日期起算,则会过晚删除。应存储规则所指的那个日期。

本章的决定

最近一次修改见于版次 2026.11.27
  • **保留规则是配置,**按作业所给定的日期来应用,绝不读取时钟。每个保留期限都是平台自己的选择,记录在规则中;已关闭的工单从开启时起算,因为该表没有 closed_at。
  • **同意是策略在查询时读取的属性;**撤回同意不会删除任何内容,而用同意过滤条件构建的表会被重建。
  • **删除依据目录和血缘来规划:**先找出每个含有 learner_id 列的表,再找出血缘所及的每个表。
  • **删除的例外由法务团队决定,**连同理由一起记录在 governance/retention.toml 中;在表的保留规则终止这些数据之前,代码会按其所述予以保留。
  • **提交记录请求 ID 和规则名称,**绝不记录个人的值。
  • **本书的过期处理会读取每个表的设置,**而 PyIceberg 自带的过期处理不会;删除操作在执行任何删除之前,会先检查计划中的每个表。
  • **本书的清扫会移除未被引用的文件,**生产环境中的最短文件存留时间为 72 小时。
  • **清单不保留受保护列的任何值:**指标模式为 none,由分类结果生成。
  • **每次删除都以一次扫描结束,**扫描每个文件,查找 ID 以及每个身份标识值的 15 个字符;扫描得到的计数就是证据。
  • **删除登记册保存每个已删除 ID 的带密钥哈希,**每次加载、恢复和重建之后都要核对。保留与删除都以剩余的内容来核验:每次删除后进行扫描,保存哈希的登记册,以及从作业所给定的日期起算的规则。扫描可能通过,而数据仍留在它没有查看的地方,因此每次加载、恢复和重建之后都要再次核对登记册。

本章,在本版次中

版次 2026.11.31示例于 2026年11月19日 通过本章 11个示例 个示例中有 11 个通过

译自英文版图书,版次 2026.11.31。

引用本节

格式

每种格式均引用英文原书,本译本以其为准。

每种格式都会注明该版次。其链接会打开当前版次,节标题处会说明该节文本最后一次修改的时间。访问日期和检索日期即为您打开此页的当天。

本页内容

第 7 章,共 10 章: 保留期限、同意与删除

  1. Keep retention rules in a reviewed file
  2. 用行删除清除过期行
  3. 在查询运行的地方读取同意状态
  4. 依据血缘规划删除
  5. Erase the rows the plan lists
  6. Expire snapshots by the table's own settings
  7. 删除没有任何快照引用的文件
  8. Keep personal values out of manifests
  9. Prove the erasure by reading every file
  10. 生产环境中会出什么问题
  11. 被擦除的学员又回来了
  12. 其他故障
  13. 本章的决定

返回顶部全部章节