保存期限、同意與刪除
閱讀約需 106 分鐘 分鐘,13 節 中共有 11 個範例。
本頁內容
讀完本章後,你將能把保存期限規則維護為經過審查的設定、讓撤回同意影響下一次查詢,並從個人資料流經的每一個資料表中刪除該人的資料。你會依據傳入的日期套用規則、從血緣(lineage)找出這些資料表,並且只保留法務團隊已記錄例外的資料。你也會清除在 Iceberg 資料表中列刪除後殘留的內容,包括舊快照、未被參照的檔案與欄位統計資訊,最後以一次掃描讀取所有剩餘檔案,證明沒有任何檔案含有該人的資料。
本書所治理的線上學習平台,自 2024 年 1 月第一位使用者註冊以來,所有資料都保存在同一個 Iceberg 目錄(catalog)中:每一張客服工單,以及每一筆學習事件和其來源的 IP 位址。目前沒有任何規定說明這兩者應保存多久。學習者 3590 撤回了對行銷電子郵件的同意,數週後又要求平台刪除其資料。前者應使行銷團隊從下一次查詢起看到不同的結果。後者則不應在任何地方留下該學習者的副本:不論是由其資料建立的資料表,或是 Iceberg 刻意保留的各資料表舊版本。
本章以三個詞彙命名主題。保存期限(retention)是平台對每一類資料的保存時間。同意(consent)是個人對其資料某項用途(例如行銷電子郵件)的同意,且可以撤回。刪除(erasure)是依請求移除個人的資料。本章為本書的共用程式碼新增兩項內容:平台的保存期限規則,位於 governance/retention.toml;以及模組 lib.retention,它負責套用這些規則、規劃並執行刪除、事後清理並檢查檔案,同時維護刪除登記簿。本章使用第 4 章的 lib.lineage 來找出學習者資料的流向,並使用第 2 章中 lib.metadata 的分類。所有範例都在本機目錄上執行 pyiceberg 0.12.0。
此部分尚未翻譯,因此以英文呈現。
Keep retention rules in a reviewed file
A retention rule says how long one kind of data is kept, from which date the time counts, and which rows it applies to. The EU's General Data Protection Regulation (GDPR) states the principle as storage limitation: personal data are to be "kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed" (Article 5(1)(e)). Its Recital 39 asks the controller to set time limits for erasure or for a periodic review (GDPR). NIST's catalog of security and privacy controls describes the same job, asking that information be disposed of, destroyed or erased once its retention period ends (control SI-12(3)) (NIST SP 800-53 Rev. 5). The platform's owners and lawyers decide how long is necessary; the code applies what they decide, every time.
That argues for rules as config, reviewed like the access policies, and for code that applies them to a day it is given, not the clock. A job that reads the clock deletes different rows depending on when it runs: a test can't pin its result, and a rerun after a failure can delete more than the run it replaces.
The platform's rules are in governance/retention.toml. The periods are the platform's own choices for this book's case, not legal advice:
[tables."support.tickets"]
keep_days = 730
counted_from = "opened_at" # tickets record when they were opened, not when they were closed
only_when = { column = "status", equals = "closed" }
[tables."learning.events"]
keep_days = 365
counted_from = "occurred_at"
[tables."billing.payments"]
keep_days = 3650 # a placeholder for the period the platform's legal team sets for payment records
counted_from = "paid_at"
Closed tickets are kept two years, events one year, and payments 3,650 days, a placeholder that the erasure plan relies on. The tickets rule should count from when a ticket was closed, but the table records only opened_at, so a ticket opened 25 months ago and closed last week goes at once. The platform lives with that because adding a closed_at column is the support team's change, not the platform team's; until they make it, the rule errs toward deleting early. lib.retention.due turns a rule and a day into an Iceberg filter. Counted on the book's "today", Oct 1, 2026, and 90 days later, the rows each rule selects come out like this:
1"""The rows each retention rule makes due on the book's today and 90 days later, counted against the day the job is given.2The `rows due` lines are the counts the prose gives."""3from datetime import timedelta45from lib import retention6from lib.catalog import open_catalog7from lib.platform import TODAY89rules = retention.rules()["tables"]10catalog = open_catalog(sorted(rules))11for name, rule in sorted(rules.items()):12 state = rule.get("only_when")13 print(f"{name}: kept {rule['keep_days']} days from {rule['counted_from']}"14 + (f", once {state['column']} is {state['equals']}" if state else ""))1516# the book's today, then 90 days later: the day is passed in, never read from the clock17for day in (TODAY, TODAY + timedelta(days=90)):18 for name, rule in sorted(rules.items()):19 table = catalog.load_table(name)20 # the earliest date kept: keep_days before the day21 print(f"{name} on {day}, kept from:", retention.cutoff(rule, day))22 print(f"{name} on {day}, rows due:", f"{retention.count(table, retention.due(rule, day)):,}")23 print(f"{name} on {day}, rows in the table:", f"{retention.count(table):,}")
python retention-rules-due.pybilling.payments: kept 3650 days from paid_at
learning.events: kept 365 days from occurred_at
support.tickets: kept 730 days from opened_at, once status is closed
billing.payments on 2026-10-01, kept from: 2016-10-03
billing.payments on 2026-10-01, rows due: 0
billing.payments on 2026-10-01, rows in the table: 17,562
learning.events on 2026-10-01, kept from: 2025-10-01
learning.events on 2026-10-01, rows due: 9,060
learning.events on 2026-10-01, rows in the table: 100,000
support.tickets on 2026-10-01, kept from: 2024-10-01
support.tickets on 2026-10-01, rows due: 58
support.tickets on 2026-10-01, rows in the table: 2,000
billing.payments on 2026-12-30, kept from: 2017-01-01
billing.payments on 2026-12-30, rows due: 0
billing.payments on 2026-12-30, rows in the table: 17,562
learning.events on 2026-12-30, kept from: 2025-12-30
learning.events on 2026-12-30, rows due: 15,545
learning.events on 2026-12-30, rows in the table: 100,000
support.tickets on 2026-12-30, kept from: 2024-12-30
support.tickets on 2026-12-30, rows due: 111
support.tickets on 2026-12-30, rows in the table: 2,000
The rows due depend on the day passed in and the data, and on nothing else. On Oct 1, 2026, the events rule keeps events from 2025-10-01 on, so 9,060 of the 100,000 events are due, and on Dec 30, 90 days later, 15,545 are. The tickets rule makes 58 of the 2,000 tickets due on Oct 1, all closed and opened over two years ago. On Dec 30 it makes 111 due. The payments rule makes 0 of the 17,562 payments due on Oct 1 and 0 on Dec 30.
Working rule: Keep each table's retention period, the column it counts from and the state a row must be in, in a reviewed file, and apply it against a day the scheduler passes in, never the clock. Count from the date the rule means. When the table doesn't record that date, as the platform's tickets don't record when they closed, ask the table's owner to add the column rather than let the rule count from another date for good, and say in the rule's file which way it errs until then.
以列刪除移除過期的資料列
有了寫在檔案中的規則,保留作業就會刪除規則所指定的內容。Iceberg 寫入器刪除資料列的方式有兩種,由資料表屬性 write.delete.mode 決定,預設值為 copy-on-write(Iceberg configuration)。Copy-on-write 會重寫每個受影響的資料檔,並略去被刪除的資料列。Merge-on-read 則不動資料檔,而是在它旁邊寫入一個刪除檔。規格中有兩種刪除檔:位置刪除(position delete)以資料檔及其中的位置標記一列(在版本 3 或以上的資料表中為刪除向量),等值刪除(equality delete)則以一個或多個欄位的值標記一列(Iceberg table spec)。PyIceberg 0.12.0 的 Table.delete 一律採用 copy-on-write:即使設定了 merge-on-read,它也只會警告尚不支援 merge-on-read,並照樣重寫檔案(原始碼)。無論哪種方式,刪除都是一次提交,也就是一個新的快照,而它之前的快照仍然存在。
lib.retention.apply 會針對每條規則,刪除 due 為指定日期所傳回的資料列,並把該規則的資料表與日期記錄在該次提交的快照摘要中,因此資料表本身的歷史就說明了這些資料列為何被刪除。以下針對 2026 年 10 月 1 日執行兩次,分別以現在的狀態與時間旅行(time travel)讀取,展示刪除之後留下了什麼,最後則是依存留時間讓舊快照過期:
1"""Applies the retention rules for the book's today as Iceberg row deletes, which leave the old rows in older snapshots.2The rerun line is zero, and the time-travel lines show the rows the snapshot before each delete still returns."""3from datetime import datetime, timedelta, timezone45from lib import retention6from lib.catalog import open_catalog7from lib.platform import TODAY89rules = retention.rules()["tables"]10catalog = open_catalog(sorted(rules))11# each table's snapshot from before the deletes, which the time-travel lines read12before = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in rules}1314deleted = retention.apply(catalog, TODAY)15for name, rows in deleted.items():16 print(f"{name}, rows deleted:", f"{rows:,}")17print("rows deleted by running again for the same day:", sum(retention.apply(catalog, TODAY).values()))1819for name in (name for name, rows in deleted.items() if rows):20 rule, table = rules[name], catalog.load_table(name)21 summary = table.current_snapshot().summary22 print(f"{name}, snapshots:", " then ".join(s.summary.operation.value for s in table.snapshots()))23 # copy-on-write writes the rows it keeps again as new records, which is what this line counts24 print(f"{name}, rows written again to delete the others:", f"{int(summary['added-records']):,}")25 print(f"{name}, the commit records:", summary["retention-rule"], "on", summary["retention-day"])26 rows_due = retention.due(rule, TODAY)27 print(f"{name}, rows due, read now:", f"{retention.count(table, rows_due):,}")28 print(f"{name}, rows due, read by time travel:", f"{retention.count(table, rows_due, before[name]):,}")29 # Snapshots carry real commit times, so their age is measured on the clock, by the table's own setting.30 print(f"{name}, snapshots expired by age today:", retention.expire_by_age(table))31 # six days on, the snapshot from before the delete is older than the table's five-day setting32 later = datetime.now(timezone.utc) + timedelta(days=6)33 print(f"{name}, snapshots expired by age six days later:", retention.expire_by_age(table, later))
python apply-retention.pybilling.payments, rows deleted: 0
learning.events, rows deleted: 9,060
support.tickets, rows deleted: 58
rows deleted by running again for the same day: 0
learning.events, snapshots: append then overwrite
learning.events, rows written again to delete the others: 90,940
learning.events, the commit records: learning.events on 2026-10-01
learning.events, rows due, read now: 0
learning.events, rows due, read by time travel: 9,060
learning.events, snapshots expired by age today: 0
learning.events, snapshots expired by age six days later: 1
support.tickets, snapshots: append then overwrite
support.tickets, rows written again to delete the others: 1,942
support.tickets, the commit records: support.tickets on 2026-10-01
support.tickets, rows due, read now: 0
support.tickets, rows due, read by time travel: 58
support.tickets, snapshots expired by age today: 0
support.tickets, snapshots expired by age six days later: 1
重試執行不會造成損害:針對同一天的第二次執行刪除了 0 列,也沒有產生提交。第一次執行刪除了 9,060 筆事件、58 張工單與 0 筆付款。每個被更動的資料表,都從 append 快照變成 overwrite 快照,而摘要顯示了 copy-on-write 的代價:為了刪除舊事件,PyIceberg 把其餘 90,940 列又寫了一次,因為這個資料表是單一的未分割檔案。若依 occurred_at 的月份分割,同樣的刪除就能直接捨棄整個檔案,而不必重寫;當一個檔案的每一列都符合條件時,Table.delete 就是這麼做的。在 PyIceberg 0.12.0 中,寫入依月份分割的資料表需要 pyiceberg-core 額外套件,本書的範例並未安裝。
刪除不會把這些資料列從資料表較早的快照中移除。以現在的狀態讀取時,learning.events 仍有 0 列超過保留期限;但以時間旅行讀取時,刪除之前的快照仍會傳回全部 9,060 列,包含 IP 位址,而存放它們的 Parquet 檔案也仍在磁碟上。lib.retention.expire_by_age 會讓早於資料表自身 history.expire.max-snapshot-age-ms(此處為五天,以時鐘計算)的快照過期。它在刪除後立即讓 learning.events 的 0 個快照過期,並在六天後讓 1 個快照過期。在舊快照過期,且不被任何快照參照的檔案也被移除之前,保留作業並未完成:這與抹除資料所需的清理工作相同。
實用原則: 以 Iceberg 的列刪除逐條套用保留規則,並在每次提交的快照摘要中記錄規則與日期。讓作業對於指定的日期具備冪等性。大型資料表應依規則計算起點的欄位分割,使其刪除能捨棄整個檔案而不必重寫;在 PyIceberg 中,請安裝其 pyiceberg-core 額外套件以寫入這類資料表。唯有在舊快照已過期、其檔案也已移除之後,才算完成這項作業。
在查詢執行之處讀取同意狀態
資料保存期限依資料的存放時間移除資料列。同意狀態改變的則是誰可以使用這些資料,而撤回同意與要求刪除並非同一個事件,因此平台的程式碼把兩者視為兩條路徑。歐盟《一般資料保護規則》(GDPR)第 7 條第 3 項規定:「資料主體有權隨時撤回其同意。撤回同意不影響撤回前基於同意所為處理之合法性。」(GDPR)同一規則的第 17 條第 1 項第 (b) 款將撤回同意列為刪除的理由之一,但僅限於處理行為已無其他法律依據的情形。在程式碼中,撤回只會變更一個屬性,刪除則會移除資料列。
在平台上,同意狀態是存取政策所讀取的資料屬性,設定方式如第 5 章所述:行銷部門的資料列篩選條件只保留 marketing_consent 為 true 的學習者,課程助理的篩選條件則只保留作者 ai_training_consent 為 true 的評論。因此撤回同意不需要更動政策。CRM 是學習者管理帳號的應用程式,它會把新的值同步到 crm.learners,而行銷部門下一次查詢時,會透過第 6 章建置、用來改寫每一個查詢的執行強制層,得到少一位學習者的結果。
請撤回學習者 3590 的行銷同意,然後查看行銷部門所見的資料,以及資料表在撤回前、撤回後,和行銷管線再次執行後所存放的內容。
1"""A learner withdraws marketing consent: one attribute changes, so marketing's next query returns one learner fewer and2no row is deleted. The table built with the consent filter keeps the learner until its pipeline runs again, as the audience lines show."""3import json45import pyarrow as pa6from pyiceberg.expressions import EqualTo78from lib import pipelines, policy, retention9from lib.catalog import open_catalog10from lib.enforce import Enforcer1112LEARNER = 359013catalog = open_catalog(["crm.learners", "learning.enrollments", "learning.courses"])14# the audience is built with the consent filter applied at build time, as the nightly job builds it15pipelines.run(catalog, "marketing_audience")16decision = policy.decide(policy.request("marketing-1", "crm.learners", ["learner_id"]))17print("marketing's row filters on crm.learners:", json.dumps(decision["row_filters"]))181920def report(when: str) -> list[int]:21 # marketing's query runs through the enforcement layer, which applies its row filters22 seen = [row[0] for row in Enforcer(catalog).sql("select learner_id from crm.learners", "marketing-1").fetchall()]23 print(f"{when}, learners marketing sees:", f"{len(seen):,}")24 print(f"{when}, marketing sees learner {LEARNER}:", LEARNER in seen)25 print(f"{when}, rows stored in crm.learners:", f"{retention.count(catalog.load_table('crm.learners')):,}")26 audience = retention.count(catalog.load_table("marketing.audience"), EqualTo("learner_id", LEARNER))27 print(f"{when}, learner {LEARNER}'s rows in marketing.audience:", audience)28 return seen293031before = report("before")32assert LEARNER in before # marketing's query returns the learner before the withdrawal33learners = catalog.load_table("crm.learners")34row = learners.scan(row_filter=EqualTo("learner_id", LEARNER)).to_arrow()35row = row.set_column(row.schema.get_field_index("marketing_consent"), "marketing_consent", pa.array([False]))36learners.overwrite(row, overwrite_filter=EqualTo("learner_id", LEARNER)) # what the CRM's sync writes37after = report("after the withdrawal")38assert LEARNER not in after # the next query, through the consent filter, drops the learner39pipelines.run(catalog, "marketing_audience")40report("after the audience is built again")
python consent-withdrawal.pymarketing's row filters on crm.learners: [{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}]
before, learners marketing sees: 1,861
before, marketing sees learner 3590: True
before, rows stored in crm.learners: 5,000
before, learner 3590's rows in marketing.audience: 1
after the withdrawal, learners marketing sees: 1,860
after the withdrawal, marketing sees learner 3590: False
after the withdrawal, rows stored in crm.learners: 5,000
after the withdrawal, learner 3590's rows in marketing.audience: 1
after the audience is built again, learners marketing sees: 1,860
after the audience is built again, marketing sees learner 3590: False
after the audience is built again, rows stored in crm.learners: 5,000
after the audience is built again, learner 3590's rows in marketing.audience: 0
撤回在下一次查詢時生效,且沒有移除任何資料:撤回前行銷部門看到 1,861 位學習者,其中包括學習者 3590;撤回後看到 1,860 位,不再包含學習者 3590。行銷部門的資料列篩選條件仍然是 [{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}],而 crm.learners 仍存放 5,000 筆資料列。沒有任何東西被刪除;學習者的帳號、選課紀錄與歷程都維持原狀。
問題出在副本。marketing.audience 是行銷管線每晚建立的名單,它在建立時套用同意篩選條件,而政策提供給行銷部門的這張資料表本身沒有同意篩選條件。撤回之後,它仍保有學習者 3590 的 1 筆資料列。在有人為該資料表的欄位分類之前,政策會對所有讀取者遮蔽這些欄位;一旦有人依第 4 章的建議完成分類,在下一次建置之前從該資料表寄出的行銷活動,就會送到已經拒絕的人手上。直到管線再次執行,筆數才會降為 0。訓練語料庫的建置同樣使用同意篩選條件,而第 9 章會在訓練前,依當下的同意狀態檢查每一次建置。
只用真假值的欄位所表達的資訊也太少:對於在平台詢問同意之前就已註冊的所有人,ai_training_consent 為 null,而 false 可能代表拒絕,也可能代表已撤回。資料隱私詞彙(Data Privacy Vocabulary)是 W3C 社群小組的報告,並非標準,它列出了同意可能處於的各種狀態,包括已給予、已拒絕、已撤回、已過期與未知(DPV 2.3)。
實用原則: 把同意狀態放在查詢執行之處:存放在一張由學習者所使用的系統持續更新的資料表中,由政策以資料列篩選條件的方式讀取,使撤回同意改變的是下一次查詢,而不刪除任何資料。若平台的法務人員判定撤回同意同時也是刪除的理由,則也要將它送入刪除路徑。每當同意狀態改變,就重建所有以同意篩選條件建置的資料表,或讓其讀取者套用相同的篩選條件。將每一筆同意記錄為一個狀態及其變更時間,並由此推導出政策所讀取的真假值屬性,把未知視為未給予。
從歷程血緣規劃資料刪除
撤回同意只會改動一項屬性;資料刪除則必須在所有地方找出這個人。GDPR 第 17 條第 1 項賦予當事人權利,可在所列的理由成立時,要求不無故拖延地刪除其資料(GDPR)。平台將一項請求視為涵蓋每一份副本,因此它的首要工作是一個工程問題:列出這個人的資料在哪裡,而這份清單分成兩半。持有學習者 ID 的來源資料表來自資料目錄:第 2 章已決定,每一個含有 learner_id 的資料表,無論其分類為何,都屬於每一次資料刪除的範圍。由這些來源表建立出來的資料表則來自歷程血緣:第 4 章從每個來源資料表出發,走訪欄位圖,便能抵達其資料列所影響的每一個資料表。
有些資料會保留。同一法規的第 17 條第 3 項規定了不適用刪除的情形,範圍以處理行為所必需者為限:其五款之中,包括為遵守控管者所適用的歐盟或會員國法律所定之法律義務(GDPR)。例外是否適用、適用於哪個資料表、持續多久,是法務團隊的決定,絕非程式碼的決定。程式碼為這項決定提供一個固定位置,即 governance/retention.toml,由法務團隊在其中寫下每一項例外及其理由。本書中的理由只是佔位文字,因為我們不做任何法律判斷:
[erasure.exceptions."billing.payments"]
reason = "PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from."
decided_by = "legal team" # a role, never a person's name
erase_columns = ["card_number"] # what the obligation doesn't need; an empty list keeps the whole row
資料刪除會保留例外資料表的資料列,直到該表的保存規則將其刪除為止,並立即從中清除 erase_columns 所列的欄位:此處即完整的信用卡號,平台的中繼資料指出它「根本不應被儲存」。lib.retention.exceptions 會拒絕沒有理由的例外,也會拒絕針對沒有任何規則可終結保存的資料表所設的例外。
找到衍生資料表,並不等於找到該學習者在其中的資料列。lib.retention.plan 會檢視走訪所抵達的每個資料表的血緣邊,尋找從學習者 ID 或其某筆資料列的 ID(例如某則評論的 ID)原封不動複製而來的欄位,以 OpenLineage 的術語來說就是 IDENTITY 邊。若有,計畫便依該欄位刪除。若沒有,表示該學習者的資料列只是影響了這個資料表,計畫就會在來源資料清除後重新執行該表的工作。計畫讀取的是請求當下的平台狀態,也就是撤回同意數週之後:crm.learners 保有學習者 3590 已撤回的同意,而第 4 章中建立分析、行銷與 AI 資料表的四條管線,在那之後都已執行過,其血緣也已記錄。
學習者 3590 的計畫每個步驟印出一行,列出每個資料表及其在該處的處理方式:
1"""The steps that erase one learner: the tables with their learner_id column, then the tables lineage says their rows reach,2with the exception the legal team recorded. The counts show how many steps each route found; the last two lines are the text the scan searches for and the rows that name the learner."""3from lib import pii, retention45LEARNER = 35906catalog, graph = retention.open_platform() # weeks after the learner withdrew marketing consent7steps = retention.plan(catalog, graph, LEARNER)8for step in steps:9 # no column is copied from the learner here: the table's own job runs again once its sources are erased10 if step.column is None:11 print(f"{step.table}: run {retention.job_for(step.table)} again")12 elif step.kept:13 print(f"{step.table}: keep, under an exception, and erase {', '.join(step.cleared)}")14 else:15 print(f"{step.table}: delete where {step.column} in {list(step.values)}"16 + (f", a copy of {step.via}" if step.via else ""))1718print("tables in the plan:", len(steps))19# a step with a column and no lineage edge came from the table's own learner_id column20print("tables found by their learner_id column:", sum(1 for step in steps if step.column and not step.via))21# found by lineage: a copy of another table's ID, or a rebuild, because the learner's rows only shaped the table22print("tables found by lineage:", sum(1 for step in steps if step.via or step.column is None))23for step in (step for step in steps if step.kept):24 print(f"the reason recorded for {step.table}:", step.kept)25theirs = retention.rows_of([LEARNER])26for name in ("billing.payments", "marketing.audience"):27 print(f"the learner's rows in {name}:", retention.count(catalog.load_table(name), theirs))28# the values the scan will search for, noted before any row is deleted29found = retention.traces(catalog, steps, LEARNER)30# The phone number is printed through lib.pii.masked, which keeps its shape and its last four digits.31print("the learner's identifying values, as the scan searches for them:", pii.masked(", ".join(found.texts)))32# rows the plan keeps, in any string column, whose text holds one of those values33named = retention.mentions(catalog, steps, found)34print("rows the plan doesn't delete whose text names the learner:", sum(named.values()))
python erasure-plan.pybilling.payments: keep, under an exception, and erase card_number
crm.learners: delete where learner_id in [3590]
learning.enrollments: delete where learner_id in [3590]
learning.events: delete where learner_id in [3590]
learning.reviews: delete where learner_id in [3590]
support.tickets: delete where learner_id in [3590]
analytics.course_performance: run course_performance again
analytics.learner_activity: delete where learner_id in [3590], a copy of crm.learners.learner_id
marketing.audience: delete where learner_id in [3590], a copy of crm.learners.learner_id
ai.review_corpus: delete where review_id in [1199], a copy of learning.reviews.review_id
tables in the plan: 10
tables found by their learner_id column: 6
tables found by lineage: 4
the reason recorded for billing.payments: PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from.
the learner's rows in billing.payments: 2
the learner's rows in marketing.audience: 0
the learner's identifying values, as the scan searches for them: •••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e
rows the plan doesn't delete whose text names the learner: 0
計畫共走訪 10 個資料表:6 個含有 learner_id 欄位的來源資料表,以及由歷程血緣加入的 4 個。依據例外,它保留該學習者的 2 筆付款紀錄。各步驟是依欄位與歷程血緣決定,而非依資料列,所以計畫列出了兩個不含該學習者任何資料列的資料表:support.tickets,因為學習者 3590 從未開過支援單;以及 marketing.audience,它含有 0 筆資料列,因為管線在撤回同意之後重建過它。ai.review_corpus 完全沒有 learner_id,但歷程血緣顯示其 review_id 是 learning.reviews.review_id 的 IDENTITY 複本,因此計畫依該 ID 刪除這位學習者的評論。analytics.course_performance 只存放每門課程的計數與平均值,所以計畫從已清除的來源重建它。重建只需執行一次管線,因此程式碼不必判斷曾經計入該學習者的平均值是否仍屬其個人資料。
文字可能在計畫不刪除的資料列中指名某個人,例如另一位學習者的支援單。範例會在每個字串欄位中搜尋 governance/retention.toml 所列的識別值,並找到 0 筆這類資料列。搜尋會比對每個值的前 15 個字元,轉為小寫,在文字中的任何位置皆可:此處為 •••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e,列印時電話號碼已遮蔽,截為 15 個字元的原因見最終掃描的說明。若姓名的輸入方式不同,或電話號碼沒有分隔符號,則需要第 3 章的偵測器。
實用原則: 每一次資料刪除都應依據資料目錄與歷程血緣圖來規劃,而非依據某人維護的清單:先找出每個含有該人 ID 欄位的資料表,再找出歷程血緣顯示其資料列所抵達的每個資料表。依據歷程血緣顯示為 ID 原封不動複本的欄位,從衍生資料表中刪除,並重建那些只是受該人資料列影響的衍生資料表。例外應放在設定檔中,由法務團隊連同理由撰寫,讓計畫讀取並顯示;絕不讓程式碼自行決定任何一項例外。請先執行第 4 章針對沒有任何工作的歷程血緣能涵蓋之資料表所做的檢查,因為計畫無法觸及歷程血緣未顯示的副本。
此部分尚未翻譯,因此以英文呈現。
Erase the rows the plan lists
The plan says where learner 3590's data is, and lib.retention.erase carries it out in the plan's order. It deletes each step's rows with Table.delete, sources first, writes the rows an exception keeps again with the exception's columns empty, and runs the job of each table the plan rebuilds once the sources are done. Every commit records the request's ID, erasure-0001, in its snapshot summary. Iceberg keeps a summary in the table's metadata files after its snapshot expires, in all of them by default, or only the last write.metadata.previous-versions-max (100 by default) once write.metadata.delete-after-commit.enabled is set (Iceberg maintenance, Iceberg configuration). So the summary holds only the request's ID, which the erasure register pairs with a keyed hash of the learner's ID, and never the learner's email.
The erasure runs on learner 3590's plan under one request ID; reading crm.learners and learning.events now and by time travel shows what it leaves:
1"""Carries out the erasure plan with Iceberg row deletes. The current snapshots no longer hold the learner, but the snapshots2before them still do, and the `read by time travel` lines show it."""3from lib import retention45LEARNER = 35906catalog, graph = retention.open_platform()7steps = retention.plan(catalog, graph, LEARNER)8# each table's snapshot from before the erasure, which the time-travel lines read9before = {step.table: catalog.load_table(step.table).current_snapshot().snapshot_id for step in steps}1011done = retention.erase(catalog, steps, "erasure-0001") # the request's ID goes into every commit's summary12for table, rows in done.items():13 # a rebuilt table reports 'rebuilt' rather than a count of rows14 print(f"{table}, rows deleted:" if rows != "rebuilt" else f"{table}:", rows)15print("tables one row was deleted from:", ", ".join(table for table, rows in done.items() if rows == 1))16theirs = retention.rows_of([LEARNER])17payments = catalog.load_table("billing.payments").scan(row_filter=theirs).to_arrow()18print("billing.payments, the learner's rows kept:", payments.num_rows)19# the exception erases card_number to null, so the values that are not null are the card numbers left20print("billing.payments, card numbers left in them:", payments.num_rows - payments["card_number"].null_count)21for name in ("crm.learners", "learning.events"):22 table = catalog.load_table(name)23 print(f"the learner's rows in {name}, read now:", retention.count(table, theirs))24 print(f"the learner's rows in {name}, read by time travel:", retention.count(table, theirs, before[name]))2526summary = catalog.load_table("crm.learners").current_snapshot().summary27# copy-on-write writes the rows it keeps again as new records, so this counts every other row of the file28print("crm.learners, rows written again to delete one:", f"{int(summary['added-records']):,}")29print("crm.learners, the commit records:", summary["erasure-request"])30# snapshots of tables the plan deleted from (not rebuilt, not kept) that still return a row the erasure deleted31holding = [(table, snapshot) for step in steps if step.column is not None32 for table in [catalog.load_table(step.table)] for snapshot in table.snapshots()33 if not step.kept and retention.count(table, retention.rows(step), snapshot.snapshot_id)]34print("snapshots that still hold the rows the erasure deleted:", len(holding))35print("of them current:", sum(snapshot == table.current_snapshot() for table, snapshot in holding))
python erase-learner.pybilling.payments, rows deleted: 0
crm.learners, rows deleted: 1
learning.enrollments, rows deleted: 2
learning.events, rows deleted: 19
learning.reviews, rows deleted: 1
support.tickets, rows deleted: 0
analytics.course_performance: rebuilt
analytics.learner_activity, rows deleted: 1
marketing.audience, rows deleted: 0
ai.review_corpus, rows deleted: 1
tables one row was deleted from: crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus
billing.payments, the learner's rows kept: 2
billing.payments, card numbers left in them: 0
the learner's rows in crm.learners, read now: 0
the learner's rows in crm.learners, read by time travel: 1
the learner's rows in learning.events, read now: 0
the learner's rows in learning.events, read by time travel: 19
crm.learners, rows written again to delete one: 4,999
crm.learners, the commit records: erasure-0001
snapshots that still hold the rows the erasure deleted: 6
of them current: 0
The erasure took the learner out of every table in the plan holding any. It deleted 19 of the learner's events, 2 enrollments, and one row each from crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus. It deleted nothing from support.tickets and marketing.audience, which held none, and it rebuilt analytics.course_performance.
billing.payments keeps the learner's 2 payments, with 0 card numbers left in them. Read now, crm.learners holds 0 rows for the learner. Read by time travel to the snapshot before the erasure, it returns 1, with name, email, phone and address, and learning.events returns all 19 of the learner's events the same way.
To delete one row of crm.learners, PyIceberg wrote the other 4,999 into a new file, as copy-on-write does, and the old file, with the learner in it, belongs to the snapshot before.
Across the plan's tables, 6 snapshots still hold the rows the erasure deleted, and 0 of them are current. That is what snapshots are for: a reader can go back to any of them. For an erasure, it means the delete took the learner out of the table but not out of storage.
Working rule: Carry out an erasure as row deletes in the plan's order, and record only a request ID in each commit, never a value of the person's. Treat the delete as the first of three steps: the snapshots before it still hold the person, and anyone allowed to time travel can read them until they're expired.
此部分尚未翻譯,因此以英文呈現。
Expire snapshots by the table's own settings
After the erasure's deletes, learner 3590's rows live on in the snapshots before them. The Iceberg spec says snapshots expire so that the data files they removed or replaced can be physically deleted (Iceberg table spec). Each table sets how in its properties (Iceberg configuration): gc.enabled allows garbage collection, such as expiring snapshots and removing orphan files; history.expire.min-snapshots-to-keep is how many snapshots to keep however old they are; and history.expire.max-snapshot-age-ms is the age at which the rest may go. In Iceberg's Java library, regularly expiring snapshots deletes unused data files (Iceberg maintenance).
PyIceberg 0.12.0 does less. Its table.maintenance.expire_snapshots() commits one change, which removes the snapshots from the table's metadata and deletes no file (source), though its documentation offers it to "clean up table metadata and reduce storage costs" (PyIceberg API). Nothing in it reads gc.enabled, and only its command-line tool reads the history.expire settings. An issue asks for expiry to delete files too (issue 2604). Two copies of crm.learners each have 3 snapshots, the load and two deletes. One copy has gc.enabled=false, the other history.expire.min-snapshots-to-keep=3, and each is expired two ways: by age with PyIceberg, older_than the current time, and with lib.retention.expire, which reads the two settings first.
1"""PyIceberg's snapshot expiry ignores two of the table's own settings, gc.enabled and min-snapshots-to-keep, so it expires2snapshots they would keep. The book's expiry reads both first, and the `left by PyIceberg's expiry` lines show the difference."""3from datetime import datetime, timezone45from pyiceberg.expressions import EqualTo67from lib import retention8from lib.catalog import open_catalog91011def learners_with(settings: dict):12 """crm.learners with three snapshots: the load, then two deletes."""13 table = open_catalog(["crm.learners"]).load_table("crm.learners")14 with table.transaction() as transaction:15 transaction.set_properties(settings)16 # two deletes, so there are snapshots from before them for the expiry to consider17 for learner in (3590, 3591):18 table.delete(EqualTo("learner_id", learner))19 return table202122for settings in ({"gc.enabled": "false"}, {"history.expire.min-snapshots-to-keep": "3"}):23 setting = ", ".join(f"{key}={value}" for key, value in settings.items())24 table = learners_with(settings)25 print(f"{setting}, snapshots:", len(table.snapshots()))26 try:27 print(f"{setting}, the book's expiry expires:", retention.expire(table))28 except retention.RetentionError as error:29 print(f"{setting}, the book's expiry refuses:", error)30 # PyIceberg's expiry by age, as of now: it reads neither setting31 table.maintenance.expire_snapshots().older_than(datetime.now(timezone.utc)).commit()32 print(f"{setting}, snapshots left by PyIceberg's expiry:", len(table.snapshots()))
python expiry-ignores-table-settings.pygc.enabled=false, snapshots: 3
gc.enabled=false, the book's expiry refuses: crm.learners has gc.enabled=false, which turns off garbage collection for the table
gc.enabled=false, snapshots left by PyIceberg's expiry: 1
history.expire.min-snapshots-to-keep=3, snapshots: 3
history.expire.min-snapshots-to-keep=3, the book's expiry expires: 0
history.expire.min-snapshots-to-keep=3, snapshots left by PyIceberg's expiry: 1
PyIceberg's expiry ignores both settings. It left 1 of the first table's 3 snapshots, and 1 of the second's. older_than expires every snapshot older than the time it's given except those a branch or tag points at, here only the current one.
The book's expiry refused the first table, crm.learners has gc.enabled=false, which turns off garbage collection for the table, and expired 0 snapshots of the second. An owner sets these properties for reasons the expiring code can't see, and in PyIceberg they are only advice.
For an erasure, both settings mean the job can't finish: a table that keeps a snapshot from before the delete still holds the person. A job that stopped at that table would already have deleted rows from the tables before it and left the rest unexpired, so the check belongs before the first delete. lib.retention.erase_everywhere runs lib.retention.blocked on every table in the plan first: garbage collection off, more than one snapshot to keep, or a branch or tag besides main. If any table fails, the job refuses to start and names every such table.
Here crm.learners has garbage collection off, and learning.events keeps three snapshots. Then the job and the scan run:
1"""The erasure job checks every table in its plan before it changes any, and refuses to start when a table's own settings2would keep the learner. The scan still runs, from the traces the refusal carries, and finds the learner in the files."""3from datetime import timedelta45from lib import retention67LEARNER = 35908catalog, graph = retention.open_platform()9# each setting stops expiry from removing the snapshot from before the delete: garbage collection off, or three snapshots kept10for name, settings in (("crm.learners", {"gc.enabled": "false"}),11 ("learning.events", {"history.expire.min-snapshots-to-keep": "3"})):12 with catalog.load_table(name).transaction() as transaction:13 transaction.set_properties(settings)14# every table's current snapshot, to count the tables the job changes15before = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in retention.tables(catalog)}1617try:18 # min_age of zero: one writer and nothing in flight, so no file needs a minimum age to be safe to sweep19 steps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0001", min_age=timedelta(0))20except retention.ErasureRefused as refusal:21 for name, why in refusal.tables.items():22 print(f"the job refuses, because of {name}:", why)23 found = refusal.found24# a table the job changed has a new current snapshot25print("tables the job changed:", sum(catalog.load_table(name).current_snapshot().snapshot_id != snapshot26 for name, snapshot in before.items()))27report = retention.scan(catalog, found)28print("tables whose files hold the learner:", sum(1 for _, holding in report.values() if holding))29print("files holding the learner:", sum(holding for _, holding in report.values()))
python erasure-refuses-to-start.pythe job refuses, because of crm.learners: gc.enabled=false turns off garbage collection for the table
the job refuses, because of learning.events: history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete
tables the job changed: 0
tables whose files hold the learner: 7
files holding the learner: 8
The job refuses. It names both tables, crm.learners because gc.enabled=false turns off garbage collection for the table, and learning.events because history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete, and 0 tables changed.
The scan runs either way, from the traces the refusal carries, and finds the learner in 8 files across 7 tables. A person then decides with the tables' owners, rather than code overriding an owner's setting.
Working rule: Expire snapshots through one function of your own that reads gc.enabled and the history.expire settings before it expires anything, and refuses when garbage collection is off, because PyIceberg's expiry reads neither; check again at every PyIceberg release. Before an erasure deletes anything, check every table in its plan the same way, and refuse the whole erasure when one would keep the person. For an erasure, expire every snapshot older than the delete. For retention, expire by age, as history.expire.max-snapshot-age-ms says, measured on the clock, since snapshots carry real commit times.
此部分尚未翻譯,因此以英文呈現。
Remove the files no snapshot references
Once the old snapshots are expired, the files only they referenced are unreferenced. An unreferenced file, which Iceberg calls an orphan, is under the table's location but in no snapshot's metadata. Its Java library removes them in a separate action, and warns that removing orphan files with a retention interval shorter than the time expected for any write to complete is dangerous; its default interval is 3 days (Iceberg maintenance). PyIceberg 0.12.0 has no such action. Its release notes list none (release notes). The issue that asked for one was closed by the project's stale bot, marked not planned (issue 1200). Two pull requests that added one were closed without being merged (1958, 3358). Its only call that deletes data files is purge_table, which deletes the whole table (source).
So lib.retention removes them itself, from what the metadata tables PyIceberg exposes as table.inspect say a table references. lib.retention.referenced collects the metadata files, current and logged, every snapshot's manifest list, every manifest (a manifest lists data files, with each file's column statistics), every data and delete file live in any snapshot, and any statistics files. Every other file under the table's location is unreferenced, and lib.retention.sweep deletes the ones older than the minimum age its caller passes. governance/retention.toml sets that age to 72 hours, Java's default, and lib.retention.min_file_age reads it. After the erasure, the plan's tables are expired, swept with that age and then with none, and every file is checked for the learner:
1"""After the erasure, expire the old snapshots, then remove the files no snapshot references. PyIceberg's expiry deletes no2file and has no orphan-file removal, so lib.retention.sweep does it with the inspect tables; the last lines show the manifest entries still holding the learner's email."""3from datetime import timedelta45from lib import retention67LEARNER = 35908catalog, graph = retention.open_platform()9steps = retention.plan(catalog, graph, LEARNER)10found = retention.traces(catalog, steps, LEARNER)11retention.erase(catalog, steps, "erasure-0001")12tables = [catalog.load_table(step.table) for step in steps]13print("snapshots expired:", sum(retention.expire(table) for table in tables))14print("files under the plan's tables:", sum(len(retention.files_under(table.location())) for table in tables))15print("files no metadata references:", sum(len(retention.unreferenced(table)) for table in tables))16print("files holding the learner, after expiry:", sum(held for _, held in retention.scan(catalog, found).values()))17# the configured minimum age, 72 hours, from governance/retention.toml18print("files swept with the configured minimum age:", sum(retention.sweep(t, retention.min_file_age()) for t in tables))19# One writer, nothing in flight: no file is too young to remove, so the examples sweep with no minimum age.20print("files swept with no minimum age:", sum(retention.sweep(table, timedelta(0)) for table in tables))21print("files holding the learner, after the sweep:", sum(held for _, held in retention.scan(catalog, found).values()))2223for table in tables:24 # every manifest entry, including those of deleted files, with the bounds it keeps for the email column25 entries = table.inspect.entries()26 for status, metrics in zip(entries["status"].to_pylist(), entries["readable_metrics"].to_pylist()):27 bounds = metrics.get("email") or {}28 # bounds are compared lowercased: the learner's email is typed in capitals in the manifest29 kept = (bounds.get("lower_bound"), bounds.get("upper_bound"))30 if any(text in bound.lower() for bound in kept if bound for text in found.texts):31 print(f"{'.'.join(table.name()[-2:])}, manifest entry with status {status}, email bounds:",32 bounds["lower_bound"], "to", bounds["upper_bound"])
python sweep-unreferenced-files.pysnapshots expired: 8
files under the plan's tables: 100
files no metadata references: 28
files holding the learner, after expiry: 9
files swept with the configured minimum age: 0
files swept with no minimum age: 28
files holding the learner, after the sweep: 1
crm.learners, manifest entry with status 2, email bounds: ALEXA590@POST.EX to zofia.haase@exan
Expiry removes snapshots, not files. Of the files under the plan's tables, 28 are referenced by nothing, and 9 hold the learner. The sweep at the configured 72 hours deletes 0 unreferenced files, since every file was written seconds before. The examples pass an age of zero because one process writes their catalog. The sweep with no minimum age deletes all the unreferenced files, and 1 file still holds the learner: a manifest that the current snapshot of crm.learners references.
The spec says why: when a file is replaced or deleted, its manifest entry stores the snapshot in which the file was deleted and status 2 (deleted) (Iceberg table spec), and that entry keeps the deleted file's column statistics. By default, under the metrics mode truncate(16) (Iceberg configuration), those include each string column's smallest value, cut to 16 characters, and its largest, cut to 16 with the last character raised by one, so that it still bounds every value in the file (source). The deleted entry in crm.learners keeps the email bounds ALEXA590@POST.EX to zofia.haase@exan. The first is learner 3590's email, typed in capitals, which sorts before every other email on the platform. The second is learner 4998's, the platform's largest, whose 16th character, an m, became an n. No learner is safe from this: each data file's entry keeps the smallest and largest value of every column, so a table of many small files names many learners.
Working rule: Remove unreferenced files after every expiry, with a sweep of your own built on the inspect tables, since PyIceberg has none, and keep a minimum age of days in production so that a write in progress keeps its files. Use an age of zero only where a single process writes. Don't count a person erased because no data file holds them: the manifests may still hold their values.
此部分尚未翻譯,因此以英文呈現。
Keep personal values out of manifests
The sweep left the learner's email in a manifest, as a column bound. Iceberg lets a table choose, per column, what its manifests keep, with the property write.metadata.metrics.column.<name>: none keeps nothing, counts keeps value and null counts, truncate(n) adds bounds cut to n characters, and full adds whole bounds (Iceberg configuration). PyIceberg 0.12.0 looks the property up by the column's name when it writes a file (source).
Typed by hand, those properties would drift from the classifications. lib.retention.properties generates them from governance/metadata.toml and the [statistics] section of governance/retention.toml: none for every column classified personal or sensitive, and for every column nobody has classified, the same fail-closed default the access policy uses. It adds the history.expire settings from the rules' [snapshots] section. lib.retention.apply_properties sets them on every table and rewrites each table's data once, because a metrics mode applies only to the files written after it's set; expiry and a sweep then remove the files from before. Learner 4998, whose email is the largest bound in crm.learners' manifest, is the test case, erased on two platforms, one with default statistics and one with the generated properties:
1"""Generates each table's properties from the classifications, so manifests keep no bounds on personal columns, then erases2the learner whose email is the platform's largest, on a platform with the properties and on one without. The `files holding` lines are the proof."""3from datetime import timedelta45from lib import retention67LEARNER = 49988for statistics_off in (False, True):9 catalog, graph = retention.open_platform(statistics_off)10 label = "statistics off for protected columns" if statistics_off else "default statistics"11 if statistics_off:12 generated = retention.properties(catalog)13 # the per-column metrics settings; the history.expire settings are not counted here14 turned_off = [key for wanted in generated.values() for key in wanted if key.startswith(retention.METRICS)]15 print("columns with statistics turned off:", len(turned_off))16 # tables with no entry in governance/metadata.toml, which the generated properties still cover17 unclassified = [name for name in generated if name not in retention.metadata.load()]18 print("of them in tables nobody has classified:",19 sum(key.startswith(retention.METRICS) for name in unclassified for key in generated[name]))20 print("crm.learners, properties set:", len(generated["crm.learners"]))21 for key, value in sorted(generated["crm.learners"].items()):22 print(f" {key} = {value}")23 # crm.learners has one data file, so its one manifest entry keeps the table's email bounds24 entry = catalog.load_table("crm.learners").inspect.entries()["readable_metrics"].to_pylist()[0]25 print(f"{label}, crm.learners email bounds:", entry["email"]["lower_bound"], "to", entry["email"]["upper_bound"])26 columns = retention.metadata.load()["crm.learners"].columns.values()27 personal = [column.name for column in columns if column.classification == "personal"]28 print(f"{label}, crm.learners personal columns:", len(personal))29 with_bounds = sum(entry[name]["lower_bound"] is not None for name in personal)30 print(f"{label}, of them with bounds in the manifest:", with_bounds)31 steps, found, _ = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0002", min_age=timedelta(0))32 print(f"{label}, files holding learner {LEARNER} after erasure, expiry and sweep:",33 sum(held for _, held in retention.scan(catalog, found).values()))
python statistics-off-for-personal-columns.pydefault statistics, crm.learners email bounds: ALEXA590@POST.EX to zofia.haase@exan
default statistics, crm.learners personal columns: 7
default statistics, of them with bounds in the manifest: 7
default statistics, files holding learner 4998 after erasure, expiry and sweep: 1
columns with statistics turned off: 24
of them in tables nobody has classified: 10
crm.learners, properties set: 9
history.expire.max-snapshot-age-ms = 432000000
history.expire.min-snapshots-to-keep = 1
write.metadata.metrics.column.city = none
write.metadata.metrics.column.date_of_birth = none
write.metadata.metrics.column.email = none
write.metadata.metrics.column.full_name = none
write.metadata.metrics.column.notes = none
write.metadata.metrics.column.phone = none
write.metadata.metrics.column.street_address = none
statistics off for protected columns, crm.learners email bounds: None to None
statistics off for protected columns, crm.learners personal columns: 7
statistics off for protected columns, of them with bounds in the manifest: 0
statistics off for protected columns, files holding learner 4998 after erasure, expiry and sweep: 0
With default statistics, 1 file still holds learner 4998 after the erasure, expiry and sweep, through a raised upper bound. The manifest keeps bounds for 7 of the table's personal columns, and its email bounds are ALEXA590@POST.EX to zofia.haase@exan.
Turning statistics off for personal columns is what keeps learner 4998 out of the files. The email bounds are None to None, the manifest keeps bounds for 0 of the table's 7 personal columns, and 0 files hold the learner after the erasure, expiry and sweep.
The generated properties turn statistics off for 24 columns, 10 of them in marketing.audience and ai.review_corpus, the two tables that chapter 2 found with no metadata. crm.learners gets 9 properties: the two expiry settings, and none for each personal column.
The cost is pruning. Iceberg uses the bounds in manifests to skip files a query's filter can't match (Iceberg table spec), so without them, a query that filters on email reads every file. The platform filters and erases by learner_id, which is internal and keeps its statistics. counts is the other choice: it keeps the counts that help a planner, and no values. And because PyIceberg 0.12.0 keeps the property under the column's old name on a rename, leaving the new name without it, the properties are generated again on every schema change, as chapter 2 asks of every view of the metadata keyed by name.
Working rule: Generate per-column metrics properties from your classifications, none or counts for every personal, sensitive or unclassified column, and apply them when a table is created or, for an existing table, with one rewrite followed by expiry and a sweep. Keep statistics on the ID columns you filter and erase by. Generate the properties again after every schema change.
此部分尚未翻譯,因此以英文呈現。
Prove the erasure by reading every file
Each step so far removed something the one before left behind, and left something of its own: the delete, the expiry, the sweep and the statistics settings. That is the case for checking the result rather than trusting the steps. The European Data Protection Board's report on how controllers handle erasure, adopted in February 2026, found backups, and anonymisation used in place of deletion, among the weak points, and recommends that controllers "Verify that erasure has been carried out and be able to demonstrate such erasure" (EDPB, 2026).
lib.retention.scan reads every file under the catalog's warehouse, each with its own format's reader, and looks for the learner's traces. lib.retention.traces notes the traces before the erasure: the IDs of every row the plan deletes, by column, and the first 15 characters, lowercased, of each identifying value. Fifteen, because a manifest keeps a largest value's first 15 characters unchanged and raises the 16th by one, so a search for 16 misses a person whose value is a file's largest, as learner 4998's email is. A Parquet file is read as rows. A manifest or manifest list, both Avro, is read record by record with PyIceberg's Avro reader: every string in it, bounds included, and the bounds of each ID column, decoded by the schema the manifest carries. A metadata file is read as text. In a table an exception keeps, the learner's IDs are expected, so there the scan looks only for values, including those of the columns the exception erased. lib.retention.erase_everywhere runs the whole job: plan, note the traces, check every table, erase, expire, sweep. On the platform with the generated properties, the whole job runs and ends with the scan:
1"""Runs the whole erasure job on the platform with statistics off for protected columns, then scans every file under the2warehouse. The `files holding the learner` line is the proof; the rest is evidence of IDs and counts, never the learner's values."""3from datetime import timedelta45from lib import retention67LEARNER, REQUEST = 3590, "erasure-0001"8catalog, graph = retention.open_platform(statistics_off=True)9# min_age of zero: one writer, so nothing is in flight that a sweep could take10steps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))1112report = retention.scan(catalog, found)13for table, (files, holding) in report.items():14 print(f"{table}: {files} files read, {holding} holding the learner")1516print("request:", REQUEST)17print("tables visited:", len(steps))18# each table the job deleted from has a count of rows; the one it reran has 'rebuilt' instead19print("rows deleted:", sum(rows for rows in done.values() if rows != "rebuilt"))20print("tables rebuilt:", ", ".join(table for table, rows in done.items() if rows == "rebuilt"))21print("tables kept under an exception, columns erased:",22 "; ".join(f"{step.table}, {', '.join(step.cleared)}" for step in steps if step.kept))23# every snapshot but the current one: the ones a reader can still time travel to24print("snapshots a reader can still time travel to:",25 sum(len(catalog.load_table(step.table).snapshots()) - 1 for step in steps))26print("files no metadata references:",27 sum(len(retention.unreferenced(catalog.load_table(name))) for name in retention.tables(catalog)))28print("tables missing their generated properties:", ", ".join(retention.missing_properties(catalog)))29print("files read:", sum(files for files, _ in report.values()))30print("files holding the learner:", sum(holding for _, holding in report.values()))
python verify-erasure.pyai.review_corpus: 11 files read, 0 holding the learner
analytics.course_performance: 5 files read, 0 holding the learner
analytics.learner_activity: 11 files read, 0 holding the learner
billing.payments: 12 files read, 0 holding the learner
crm.instructors: 8 files read, 0 holding the learner
crm.learners: 11 files read, 0 holding the learner
learning.courses: 8 files read, 0 holding the learner
learning.enrollments: 11 files read, 0 holding the learner
learning.events: 11 files read, 0 holding the learner
learning.reviews: 11 files read, 0 holding the learner
marketing.audience: 8 files read, 0 holding the learner
support.tickets: 8 files read, 0 holding the learner
request: erasure-0001
tables visited: 10
rows deleted: 25
tables rebuilt: analytics.course_performance
tables kept under an exception, columns erased: billing.payments, card_number
snapshots a reader can still time travel to: 0
files no metadata references: 0
tables missing their generated properties: analytics.course_performance
files read: 115
files holding the learner: 0
The scan finds the learner in 0 of the 115 files it reads, and no reader can time travel to the learner, since 0 older snapshots remain, and 0 files are unreferenced. The lines after the per-table counts are the evidence: the request's ID, the tables visited, the rows deleted, the table the job rebuilt, the table an exception keeps and the column erased from it, billing.payments, card_number, with the reason in governance/retention.toml, and the scan's counts. They name no value of the learner's, so keeping them puts none back, which chapter 8 relies on when it writes them to the audit log.
One line is a warning: analytics.course_performance lacks its generated properties, because lib.pipelines.run dropped the table and created it again without them. That table has no personal columns. The next table a pipeline rebuilds might have them, and the scan would then report its manifests' bounds as a failed erasure, so the check belongs after every pipeline run.
IDs can stay in a manifest too, since the ID columns keep their statistics. An erased ID that was a file's smallest or largest, such as learner 5000's, stays in the deleted entry's bounds, and the scan reports it. One more commit to the table, followed by an expiry and a sweep, removes it, because PyIceberg's next append leaves out a manifest that holds only deleted files (source).
The scan covers the warehouse only. Exports, backups, the source systems and logs outside it can hold the learner too, and the same traces can be searched for in each the platform can read.
Working rule: End every erasure with a scan of every file the platform stores, for the person's IDs and identifying values, read with each format's own reader, and keep its result, counts and IDs only, as the evidence. Search for as much of each value as every bound keeps unchanged, which is one character less than the truncation length. Treat any file that still holds the person as a failed erasure, not a warning. Search the stores outside the warehouse that you control, and list in the record the ones you can't search.
正式環境中會出什麼問題
刪除操作成功時,無論資料已經消失,還是仍留在儲存空間中,外觀都沒有差別,抹除失敗正是這樣發生的。這裡的第一種失敗,是讓已完成的抹除被復原。
被清除的學員又回來了
除了 CRM 的同步之外,資料倉儲每晚還會依據應用程式的匯出檔,重新載入整個 crm.learners,而這次的清除只在資料倉儲中執行。請依計畫所述,在所有地方清除學員 3590,接著讓夜間重新載入以及讀取該資料表的各條管線執行:
1"""An erased learner comes back: the nightly reload of crm.learners from the application's export brings the rows back, and2the pipelines rebuild from them. A check against the erasure register catches the rows."""3from datetime import timedelta45from lib import pipelines, retention67LEARNER, REQUEST = 3590, "erasure-0001"8catalog, graph = retention.open_platform()9retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))10register = [retention.register_entry(REQUEST, LEARNER)] # the request's ID and a keyed hash of the learner's ID11print("the erasure register's entry:", register[0]["request"], register[0]["learner"])12learners = catalog.load_table("crm.learners")13print("the learner's rows in crm.learners after the erasure:", retention.count(learners, retention.rows_of([LEARNER])))1415# the nightly reload: the application's export still holds the learner, since nobody erased it there16learners.overwrite(retention.crm_export())17# the pipelines that read crm.learners rebuild from the reloaded table18for name in ("learner_activity", "marketing_audience"):19 pipelines.run(catalog, name)20for table in ("crm.learners", "analytics.learner_activity", "marketing.audience"):21 print(f"the learner's rows in {table} after the nightly run:",22 retention.count(catalog.load_table(table), retention.rows_of([LEARNER])))2324# hashes each learner_id in every table and looks it up in the register, so no learner's ID is printed25for table, rows in retention.erased_rows(catalog, register).items():26 print(f"the check against the register, rows of erased learners in {table}:", rows)
python erased-learner-returns.pythe erasure register's entry: erasure-0001 c25d8a8b12426596f0c0b1ebfb890246b42ea535791f07256efd16fc6c2cb84b
the learner's rows in crm.learners after the erasure: 0
the learner's rows in crm.learners after the nightly run: 1
the learner's rows in analytics.learner_activity after the nightly run: 1
the learner's rows in marketing.audience after the nightly run: 0
the check against the register, rows of erased learners in analytics.learner_activity: 1
the check against the register, rows of erased learners in crm.learners: 1
只在資料倉儲中完成的清除無法持久。清除之後,crm.learners 中該學員有 0 列;夜間執行之後,它又有 1 列,而由它重建的 analytics.learner_activity 則有 1 列。marketing.audience 之所以仍保持乾淨,只是因為該學員的同意以已撤回的狀態回傳,而它所連結的註冊紀錄已被清除。
能偵測出這個問題的檢查會維護一份清除登錄簿:每筆請求只記錄其 ID,以及被清除學員 ID 的金鑰雜湊(HMAC-SHA256),其他一概不記。金鑰是清除作業的機密,其保管方式與第 6 章中強制執行層的假名化金鑰相同,但必須是另一把金鑰,因為持有該金鑰的人可以對每個學員 ID 計算雜湊,進而找出登錄簿所指的是誰。每次載入、還原與重建之後,lib.retention.erased_rows 會對每個含有 learner_id 欄位的資料表,將該欄位的每個值做雜湊並查詢登錄簿,因此還原或回補資料時也能受到檢查,而登錄簿本身不必存放任何人的 ID。它在 analytics.learner_activity 與 crm.learners 中找到了該學員,並略過 billing.payments,因為該表的資料列依例外規定予以保留。稽核日誌在第 8 章中對這次清除的紀錄,只含相同的請求 ID 與筆數,絕不含學員本人,所以請求與學員只會在登錄簿中以雜湊的形式相遇。我們將 ID 的金鑰雜湊視為假名化資料,絕不視為匿名化資料。
實用原則: 先在來源系統中清除,或在同一筆請求中一併清除,並維護一份被清除 ID 的金鑰雜湊登錄簿,每次載入都依它過濾,每次執行都對它檢查,且金鑰作為機密與資料分開保管。出現已清除 ID 的資料列,應如同發生外洩一樣,視為事件處理。
其他失敗情形
- 舊快照與舊檔案仍保留「已刪除」的資料。 保存期限或抹除作業若在刪除之後就停止,所有資料仍可透過時間旅行(time travel)讀取,而在 PyIceberg 中,快照過期後這些檔案同樣會留在儲存空間。請在同一項作業中執行快照過期與清除,再進行掃描。
- 其他引擎以 merge-on-read 方式刪除。 PyIceberg 會讀取其他引擎寫入的刪除向量(deletion vector)(版本說明)。以這種方式刪除的資料列,在壓實(compaction)重寫之前仍留在原資料檔中,而刪除檔也可能保留這些資料列的值,因為兩種刪除檔都可以編碼被刪除資料列的值(Iceberg 資料表規格)。請在快照過期與清除之前,先壓實受影響的檔案,並且連刪除檔也一併掃描。
- 備份會保留清除作業所刪除的一切。 備份是 EDPB 報告指出的弱點之一。請與負責備份的人員共同決定備份如何抹除或自然到期,並將決定記錄在每一筆請求中。
- 無人抹除的衍生資料表。 不送出血緣(lineage)資訊的作業,會寫出計畫看不見的副本,就像第 4 章未經檢測的匯出作業那樣。每次抹除之前,請先執行
lib.lineage.unaccounted。 - 重建的資料表會失去其屬性。 先刪除再建立資料表的管線,會失去所產生的統計設定,下一次抹除後,資料值就會留在其資訊清單(manifest)中。請在每次執行後重新套用這些屬性,或在建立資料表時就一併設定。
- 更名的欄位會失去其設定。 在 PyIceberg 中,指標(metrics)屬性仍然掛在欄位的舊名稱之下,因此欄位更名後,新名稱沒有這項設定。每次結構描述變更時,都要重新產生這些屬性。
- 保存期限從錯誤的日期起算。 規則的本意是「結案後」,卻從開案日起算,就會過早刪除;若從載入日期起算,則會過晚刪除。請儲存規則本意所指的那個日期。
本章的決定
- **保存規則屬於設定,**以作業取得的日期套用,絕不使用時鐘。各項保存期限是平台自行的選擇,記錄在規則中;已結案的工單從開立日起算,因為該資料表沒有
closed_at。 - **同意是政策在查詢時讀取的屬性;**撤回同意不會刪除任何資料,而以同意篩選條件建立的資料表會重建。
- **刪除作業依據目錄與血緣來規劃:**先找出每個含有
learner_id欄位的資料表,再找出血緣所觸及的每個資料表。 - **刪除的例外由法務團隊決定,**連同理由記錄在
governance/retention.toml中;程式碼依其所述保留資料,直到該資料表的保存規則使其結束為止。 - **提交紀錄記錄請求 ID 與規則名稱,**絕不記錄個人的值。
- **本書的到期處理會讀取各資料表的設定,**而 PyIceberg 的不會;刪除作業在刪除任何資料之前,會先檢查計畫中的每個資料表。
- **本書的清理作業會移除未被參照的檔案,**正式環境中的最短檔齡為 72 小時。
- **資訊清單不保存受保護欄位的任何值:**指標模式為
none,並由分類結果產生。 - **每次刪除作業都以掃描作結,**掃描每個檔案中的 ID,以及每個識別值的 15 個字元;其計數即為證據。
- **刪除登記簿保存每個已刪除 ID 的金鑰雜湊,**並在每次載入、還原與重建之後檢查。保存與刪除是以剩下的內容來檢查:每次刪除後的掃描、雜湊登記簿,以及從作業取得的日期起算的規則。掃描可能通過,資料卻仍留在未掃描的某處,因此在每次載入、還原與重建之後,都要再次檢查登記簿。
本章,於本版次中
版次 2026.11.31範例於 2026年11 月 19 通過測試本章 11 個範例 個範例中有 11 個通過
譯自英文書,版次 2026.11.31。