在不同格式之间迁移表并核对副本
阅读约需 41分钟 分钟,7节中含4个示例。
读完本章后,你将能够把一张表从一种格式复制到另一种格式,并核对副本包含相同的行。你还能说明复制会遗漏哪些内容,并判断哪些损失可以接受,哪些损失应当中止迁移。
分析师希望把 pageviews 表放进 DuckLake,通过 DuckDB 查询一个目录文件。pageviews 表,即 wiki.pageviews_hourly,自第 1 章起一直存放在 Apache Iceberg 中。另一个团队的表保存在 Delta Lake 中,他们希望把相同的行放进 Iceberg。在这两次迁移中,源表都保持原位,迁移就是复制到一张新表。复制是一次性的写入,大部分工作在于围绕它的各项核对。本章先完成这两次复制,再加入能够发现隐性损失的核对。
本章的示例固定使用 pyiceberg 0.12.0、deltalake 1.6.6 和 duckdb 2.0.0,DuckLake 也由 DuckDB 运行。每个复制示例都在临时仓库中,用 pageviews 切片中相同的三个小时数据构建自己的小型源表,共 1499 行。
副本保留什么、丢失什么
副本在某一时刻读取一张表的行,并把它们以另一种格式写入另一张表。每个目标库各自写出自己的数据文件、元数据和历史。能够完整保留的,是双方都能表示的内容:行,以及双方都支持其类型的列。其余的一切,要么留在源表中,要么必须在目标表上手工重新设置。
丢失的内容用表格来看最清楚。每一行列出表的一个组成部分、副本对它的处理方式,以及本章在何处展示了结果。
| 表的组成部分 | 复制之后 | 本章展示的位置 |
|---|---|---|
| 行及其值 | 保留 | 摘要校验 |
| 列名和类型 | 双方格式都有该类型时保留 | Delta Lake 一节 |
| 提交历史 | 丢失。副本会开始自己的历史 | 两个复制一节 |
| 分区规范 | 除非在目标上设置,否则丢失 | DuckLake 一节 |
| 精度低于微秒的时间戳 | 取决于格式:被拒绝、被截断或被保留 | 时间戳一节 |
历史是读者常常没有注意到的一项损失。Iceberg 以快照记录变更,Delta Lake 则把每次提交记录为一个版本。关于快照与时间旅行的一章已经说明了读者如何使用它们。副本会在目标中开启一段新的历史,因此目标的历史从复制那一刻算起。只要保留源表,源表就保有它自己的历史。副本中没有任何内容指回源表的历史。
副本与注册也不相同。已注册的元数据文件能让一张表在第二个目录中可见,而不触动其文件,因此两个目录读取的是同一份数据。副本则会写出新文件,并保持源表不变,这意味着现在有两张表接受写入,二者可能逐渐产生差异。当目标必须拥有自己的文件和历史时,选择复制。当两个目录应当共享同一组文件时,选择注册,并接受共享文件必须保持原位这一约束。
行之所以得以保留,是因为复制搬运的是值,而不是文件。复制不移动数据文件。它从一张表中读出值,再为另一张表写出新的 Parquet 文件。因此源表与其副本的文件名和文件数量会有所不同,比较文件名或文件数量也就无法用来校验副本。校验必须比较行。
工作准则: 复制之前,列出源表持有而目标不会保留的内容:它的历史、它的分区规范,以及比目标所能存储的精度更细的任何值。为每一项给出一个决定:接受这项损失,在目标上手工设置该属性,或者中止这次迁移。运行手册应当写明每一项决定。
将 pageviews 表复制到 DuckLake
复制时先用 scan_arrow 读取 Iceberg 表,再把这些行交给 DuckLake。DuckLake 按这些行的列创建一张空表,为它设置分区规格,然后用 INSERT ... SELECT 插入这些行。Arrow 是两端之间的桥梁,因为双方各自保留自己的库:pyiceberg 读取源表,DuckDB 通过自带的 ducklake 扩展写入 DuckLake 表。任何一方都不读取对方的文件。目录那一章展示了如何挂载 DuckLake 目录,本章的辅助函数以同样的方式挂载它。
源表按 hour_utc 的小时分区,这正是从按天分区迁移到按小时分区一节最终得到的布局。这里由 pyiceberg 用按小时的规格构建该源表,并追加三个小时的数据,每次追加提交一次:
1"""Copy an Iceberg table into DuckLake through Arrow, then print what the copy kept of the layout and the history.23The point rests on the copy partition transforms and copy files per partition lines, which show whether the4layout arrived with the rows, and on the copy snapshot lines, which show the history starting at the copy."""56from pyiceberg.partitioning import PartitionField, PartitionSpec7from pyiceberg.transforms import HourTransform89from lib.lakehouse import (copy_to_ducklake, create_iceberg_table, ducklake_files_by_partition, ducklake_partition_transforms,10 new_warehouse, read_cut_hour, safe_error, scan_arrow, TABLE)111213def main():14 try:15 with new_warehouse() as wh:16 # source_id 1 is the field ID of hour_utc in the table lib creates17 spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=HourTransform(), name="ts_hour"))18 table = create_iceberg_table(wh, partition_spec=spec)19 for hour in range(3):20 table.append(read_cut_hour(hour))21 arrow = scan_arrow(table)22 print(f"source rows: {arrow.num_rows}")23 print(f"source snapshots: {len(table.metadata.snapshots)}")24 for field in table.spec().fields:25 print(f"source partition field: {field.name}")2627 # the helper creates the copy empty, sets this spec with ALTER, and only then inserts the rows28 target = copy_to_ducklake(wh, arrow, partition_by="hour(hour_utc)")29 print(f"copy rows: {wh.con.execute(f'SELECT count(*) FROM {target}').fetchall()[0][0]}")30 # current=True keeps only the spec still in force; an earlier spec would otherwise be listed too31 print(f"copy partition transforms: {ducklake_partition_transforms(wh.con, table=TABLE, current=True)}")32 # one (partition value, files, rows) tuple per partition, in value order33 files = ducklake_files_by_partition(wh.con, "lake", TABLE)34 print(f"copy partition values: {[value for value, _, _ in files]}")35 print(f"copy files per partition: {[count for _, count, _ in files]}")36 print(f"copy rows per partition: {[int(rows) for _, _, rows in files]}")37 changes = wh.con.execute("SELECT changes FROM ducklake_snapshots('lake') ORDER BY snapshot_id").fetchall()38 # labels start at 1, so label N is the snapshot whose snapshot_id is N-139 for number, (change,) in enumerate(changes, start=1):40 print(f"copy snapshot {number}: {', '.join(sorted(change))}")41 except Exception as exc:42 print(safe_error(exc))43 raise SystemExit(1)444546if __name__ == "__main__":47 main()
python iceberg-to-ducklake-copy.pysource rows: 1499
source snapshots: 3
source partition field: ts_hour
copy rows: 1499
copy partition transforms: ['hour']
copy partition values: ['0', '1', '2']
copy files per partition: [1, 1, 1]
copy rows per partition: [499, 500, 500]
copy snapshot 1: schemas_created
copy snapshot 2: schemas_created
copy snapshot 3: tables_created
copy snapshot 4: tables_altered
copy snapshot 5: tables_inserted_into
辅助函数 scan_arrow 在源表的当前快照上读取数据,并把该快照的全部行作为一张 Arrow 表返回。因此,复制得到的是某一时刻的行,即扫描时看到的最后一次提交。扫描之后才提交的写入方不在复制结果之中,而复制本身不会给出任何提示。在复制的扫描之后提交、却始终没有进入复制结果的提交,就是切换问题,生产环境一节会再回到这个问题。
两个问题决定复制是否成功:行和布局是否都到了,以及带来了什么历史?输出回答了这两个问题。
首先是源表。输出会打印行数、快照数和分区字段。源表有 3 个快照,每次追加一个,并有一个分区字段 ts_hour,即对 hour_utc 做小时变换。
接下来是复制结果的行数,为 1499。行数与源表相同。这是有用的第一道检查,但下文的摘要一节会说明,仅凭它并不足够。
复制可能在不报任何错误的情况下丢失布局,因此语句的顺序很重要。辅助函数先把目标创建为一张空表,在其上运行 ALTER TABLE ... SET PARTITIONED BY (hour(hour_utc)),然后才插入行。DuckLake 分区文档指出分区键“只影响写入表中的新数据”,所以如果在行已经写入之后再执行 ALTER,这些行将保持未分区状态。辅助函数只保留仍然生效的规格,并列出一个变换 hour。这份列表说明复制结果被赋予了什么,而文件说明它实际包含什么:复制写出了 [1, 1, 1] 个文件,行在它们之间的划分为 [499, 500, 500]。若用一条语句完成复制,则不带任何分区规格,列表为空,行落在未分区的文件中。无论哪种方式,行都是齐全的,所以检查必须查看布局。
分区列表通过挂载的元数据库读取目录表 ducklake_partition_column,该表的定义见 DuckLake 规范。在当前锁定的版本中,没有公开函数可以返回它。
最后,快照列表显示复制结果的历史。每一行给出快照编号及其所做的更改。标签统计 DuckLake 记录的每一个快照,包括初始设置,与第 2 章相同。前两个快照创建默认模式和 wiki 模式。第三个创建空表,第四个设置其分区规格,第五个把行写入已经分区的表。源表中三次按小时的追加并没有出现。复制结果的历史从它自己的创建开始,因此对复制结果做时间旅行只能回溯到那里。若要读取第一个小时之后的源表状态,仍然需要 Iceberg 表。
挂载辅助函数还把 DATA_INLINING_ROW_LIMIT 设为 0,因此复制的行以 Parquet 文件的形式写入,上文的计数针对的正是这些文件。
工作准则: 通过 Arrow 并使用目标自己的写入器进行复制。先创建空的目标表,用 ALTER 设置它的分区规格,然后才插入行。打印目标的分区列表以及每个分区的文件数。把复制结果的快照视为复制结果自己的历史,并在最后两节的检查通过之前保留源表。
这部分尚未翻译,因此以英文显示。
Copy the Delta table into a pyiceberg table
The Delta source is built the way a team's pipeline would build it: three appends with deltalake, one per hour. The three appends are versions 0, 1 and 2. The copy reads the current rows with read_delta and appends them to an Iceberg table made with the target schema:
1"""Copy a Delta Lake table into Iceberg through Arrow, then compare the copy with its source.23The point rests on the delta versions and iceberg snapshots lines, which show several commits becoming one,4and on the two column listings, which show the names carried across."""56from deltalake import DeltaTable78from lib.lakehouse import (create_iceberg_table, new_warehouse, print_schema, read_delta, safe_error, scan_arrow,9 write_delta_hours)101112def main():13 try:14 with new_warehouse() as wh:15 path = write_delta_hours(wh.root, range(3))16 source = read_delta(path)17 # Delta numbers its versions from 0, so the count is the last version plus one18 print(f"delta versions: {DeltaTable(path).version() + 1}")19 print(f"delta rows: {source.num_rows}")20 print("delta columns (name, type, nullability):")21 print_schema(source.schema)2223 table = create_iceberg_table(wh)24 table.append(source)25 print(f"iceberg snapshots: {len(table.metadata.snapshots)}")26 print(f"iceberg rows: {scan_arrow(table).num_rows}")27 print("iceberg columns (name, type, requirement):")28 # sorted by name, so the two listings line up row for row29 for field in sorted(table.schema().fields, key=lambda field: field.name):30 # Iceberg calls a nullable field optional, so the listing says optional where Delta says nullable31 requirement = "required" if field.required else "optional"32 print(f"{field.name}\t{field.field_type}\t{requirement}")33 except Exception as exc:34 print(safe_error(exc))35 raise SystemExit(1)363738if __name__ == "__main__":39 main()
python delta-to-iceberg-copy.pydelta versions: 3
delta rows: 1499
delta columns (name, type, nullability):
domain_code string nullable
hour_utc timestamp[us] nullable
page_title string nullable
response_bytes int64 nullable
views int64 nullable
iceberg snapshots: 1
iceberg rows: 1499
iceberg columns (name, type, requirement):
domain_code string optional
hour_utc timestamp optional
page_title string optional
response_bytes long optional
views long optional
The copy has as many rows as the source but leaves its history behind. The Delta table has 3 versions and 1499 rows. The Iceberg table is built from one append of all of them, so it has 1 snapshot, the start of its own history, like the DuckLake copy. Its row count, 1499, is the same as the source's.
The rest of the output lists both schemas, one column per line. Read them side by side:
| Column | Delta Lake (Arrow type) | Iceberg type |
|---|---|---|
domain_code |
string |
string |
hour_utc |
timestamp[us] |
timestamp |
page_title |
string |
string |
response_bytes |
int64 |
long |
views |
int64 |
long |
Names come across unchanged, because the copy takes them from the Arrow schema. The types map one to one. The only spelling change is int64 in Arrow, which Iceberg calls long. Both sides use microseconds for hour_utc, and the timestamp check below explains what happens when they do not.
pyiceberg is strict about names. It checks the Arrow schema against the table's schema before it writes. It stops with a mismatch error when a column name differs, or when a type cannot be promoted to the table's type. It accepts a type it can promote, such as int to long. That check is in the pinned library's _check_pyarrow_schema_compatible function in pyiceberg/io/pyarrow.py. The Delta table also cannot rename a column at this pin, as the rename section of the schema chapter showed. If the two sides disagree on a name, you change the target, not the source. pyiceberg matches the Arrow columns to the table by name, so the target's field IDs come from the schema you write. You write the target schema yourself, rather than letting the first batch decide it.
Working rule: Write the target schema yourself, with the types you want, and print both schemas after the copy, as the example does. Accept a near-equal type such as int64 to long only after you have checked it in the output. Do not accept a name that differs.
在复制之前检查时间戳精度
时间戳是各种格式意见不一致的地方,有时这种不一致是无声的。Iceberg 规范将纳秒时间戳定义为版本 3 的类型,因此版本 2 的表没有可用于存放它们的类型(Iceberg 规范,版本 3 一节)。本示例中的表是版本 2,即默认版本。在当前固定的版本上,pyiceberg 拒绝写入版本 3,所以纳秒值只能由 Iceberg 之外的格式保留。本示例向每种格式写入两个仅在微秒以下有差别的值,即 1000000500 纳秒和 1000000999 纳秒:
1"""Write two timestamps that differ only below a microsecond to Iceberg, Delta Lake and DuckLake, and show what each kept.23The point rests on the Iceberg error line and on the Delta and DuckLake read-back lines."""45import pyarrow as pa6from deltalake import write_deltalake7from pyiceberg.schema import Schema8from pyiceberg.types import NestedField, TimestampType910from lib.lakehouse import copy_to_ducklake, new_warehouse, NAMESPACE, read_delta, safe_error, sql_rows1112# nanoseconds since 1970: one second plus 500 and 999 nanoseconds, so the two values differ below a microsecond13NANOSECONDS = pa.table({"hour_utc": pa.array([1000000500, 1000000999], pa.timestamp("ns"))})141516def main():17 try:18 with new_warehouse() as wh:19 # cast to int64 gives the integer behind each timestamp, so distinct values are counted exactly20 print(f"written distinct values: {len(set(NANOSECONDS['hour_utc'].cast(pa.int64()).to_pylist()))}")21 # TimestampType stores microseconds; a version 2 table has no nanosecond type to declare instead22 table = wh.catalog.create_table(23 f"{NAMESPACE}.precision",24 schema=Schema(NestedField(1, "hour_utc", TimestampType(), required=False)),25 )26 try:27 # this append raises, because the table cannot hold nanoseconds; the except prints the error28 table.append(NANOSECONDS)29 except Exception as exc:30 print(f"iceberg append: {safe_error(exc, wh)}")3132 path = str(wh.root / "precision")33 write_deltalake(path, NANOSECONDS, mode="overwrite")34 # the cast reads the count in the column's unit, which Delta wrote as microseconds, so 1000000 is one second35 kept = read_delta(path)["hour_utc"].cast(pa.int64()).to_pylist()36 print(f"delta read back as microseconds: {kept}")37 print(f"delta distinct values read back: {len(set(kept))}")3839 target = copy_to_ducklake(wh, NANOSECONDS)40 # epoch_ns returns nanoseconds since 1970, so a value cut to microseconds would end in three zeros41 rows = sql_rows(wh.con, f"SELECT epoch_ns(hour_utc) FROM {target}")42 print(f"ducklake epoch nanoseconds read back: {[row[0] for row in rows]}")43 except Exception as exc:44 print(safe_error(exc))45 raise SystemExit(1)464748if __name__ == "__main__":49 main()
python timestamp-precision-check.pywritten distinct values: 2
iceberg append: UnsupportedPyArrowTypeException: Column 'hour_utc' has an unsupported type: timestamp[ns]
delta read back as microseconds: [1000000, 1000000]
delta distinct values read back: 1
ducklake epoch nanoseconds read back: [1000000500, 1000000999]
第一行是输入,即两个不同的值。随后每种格式以各自的方式作出回应。
Iceberg 的追加写入失败。错误为 UnsupportedPyArrowTypeException: Column 'hour_utc' has an unsupported type: timestamp[ns]。该错误指明了列和类型,因此在三种情形中,这一次的失败最为清楚,而且默认情况下不会有任何内容被无声地改动。pyiceberg 的错误提到了一个设置 PYICEBERG_DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE,它会在写入时将纳秒向下转换为微秒。设置之后,同样的追加写入会成功,读回的值是整微秒,只留下一条日志警告。这与 Delta 所做的截断相同,因此只应在已经决定截断这些值的作业中设置它,不要在所有写入方共享的环境中设置。
Delta 的写入成功,读回的结果显示 [1000000, 1000000]。两个值都被截断为整微秒,两个不同的值变成了一个。写入没有引发任何错误,行数也相同。只有对这些值做检查才能发现损失。不同值的数量从 2 降为 1。
DuckLake 保留纳秒。它的表按写入时的样子存储这两个值,示例将它们读回为 [1000000500, 1000000999]。
pageviews 表记录的是小时,因此这种情形根本不会触及该问题。原始请求日志表则会。它的时间戳来自比微秒计数更精细的时钟,其复制必须为这些数位选定一条规则。在第一次复制之前,应对每个时间戳列提出这个问题,因为答案只能从值中看出。本章中的复制使用微秒时间戳,三种格式都能保留,所以各份副本是一致的。真实的表可能包含来自数据源的微秒以下的值,复制必须在第一次写入之前决定如何处理它们。
工作准则: 在第一次复制之前,用一个微秒以下的值检查时间戳精度。预先选定规则:将该列转换为微秒并统计转换改变的值的数量,拒绝这次复制,或者写入能保留纳秒的格式。不要把选择交给写入库,因为无声的截断不会给出任何可据以处理的错误。
用排序摘要检查每份副本
行数相同并不能证明值也相同。副本可以保持行数不变而改动某一个值,行数不会显示这一点,摘要则可以。一张表的摘要是对其排序后各行计算的哈希:digest_rows先对行排序,把每一行写成一行文本,再对结果求哈希。排序使摘要与各行返回的顺序无关,因此无论各种格式以何种顺序返回行,只要两份副本的行相同,摘要就相同。digest_rows在两份副本上运行,也在一份故意改动了一个浏览数的副本上运行:
1"""Check each copy against its source with a sorted digest, and show that one changed value fails it.23The point rests on the two iceberg to ducklake digest lines: the copy matches its source, and a copy with one4changed count does not."""56from lib.lakehouse import (copy_to_ducklake, create_iceberg_table, digest_rows, new_warehouse, read_cut_hour,7 read_delta, safe_error, scan_arrow, sql_rows, write_delta_hours)8910def same(source, copy):11 return digest_rows(source) == digest_rows(copy)121314def check_iceberg_to_ducklake():15 with new_warehouse() as wh:16 table = create_iceberg_table(wh)17 for hour in range(3):18 table.append(read_cut_hour(hour))19 target = copy_to_ducklake(wh, scan_arrow(table))20 source_rows = sql_rows(wh.con, "SELECT * FROM source", source=scan_arrow(table))21 copy_rows = wh.con.execute(f"SELECT * FROM {target}").fetchall()22 first = copy_rows[0]23 # the copy's columns are hour_utc, domain_code, page_title, views, response_bytes, so index 3 is views24 changed = [(*first[:3], first[3] + 1, first[4]), *copy_rows[1:]]25 print(f"iceberg to ducklake rows: {len(source_rows)} and {len(copy_rows)}")26 matches = same(source_rows, copy_rows)27 # printed before the assert, so a failing run still shows the word28 print(f"iceberg to ducklake digests: {'match' if matches else 'differ'}")29 assert matches30 still_matches = same(source_rows, changed)31 print(f"iceberg to ducklake with one changed view count digests: {'match' if still_matches else 'differ'}")32 # one changed count must change the digest33 assert not still_matches343536def check_delta_to_iceberg():37 with new_warehouse() as wh:38 path = write_delta_hours(wh.root, range(3))39 table = create_iceberg_table(wh)40 table.append(read_delta(path))41 source_rows = sql_rows(wh.con, "SELECT * FROM source", source=read_delta(path))42 copy_rows = sql_rows(wh.con, "SELECT * FROM copy", copy=scan_arrow(table))43 print(f"delta to iceberg rows: {len(source_rows)} and {len(copy_rows)}")44 assert same(source_rows, copy_rows)454647def main():48 try:49 check_iceberg_to_ducklake()50 check_delta_to_iceberg()51 except Exception as exc:52 print(safe_error(exc))53 raise SystemExit(1)545556if __name__ == "__main__":57 main()
python verify-copies-by-digest.pyiceberg to ducklake rows: 1499 and 1499
iceberg to ducklake digests: match
iceberg to ducklake with one changed view count digests: differ
delta to iceberg rows: 1499 and 1499
改动一个数字就足以改变摘要,所以摘要一致只是证据,摘要不一致则必须停下。对于Iceberg 到 DuckLake 的副本,输出先打印两个相等的行数,再打印摘要 match。对 Delta 到 Iceberg 的副本做同样的检查,摘要同样一致。改动一个浏览数之后,摘要 differ。
摘要比较的是数据库返回的值,它有一个局限。如果两侧在求哈希之前都被转换成文本,不同类型的两个值可能得到相同的文本:整数 1 与文本 '1' 的文本形式完全相同。应当像示例那样比较带类型的值,并像Delta 到 Iceberg 的副本那样打印模式。摘要也不涉及列名和列类型:两张表的行相同而列名不同,它们的摘要也相同。前面各节中的模式列表检查列名,摘要检查值,所以一份副本两者都需要。
摘要不是文件的哈希。各行按带类型的值排序,每一行变成其 Python 表示形式的一行文本,连接后的文本用 SHA-256 求哈希。值相等的两行产生相等的文本行,这正是比较所需要的。一个值的改变会改变一行文本,从而改变整个摘要。
还要选择检查的时机。摘要只对它运行时存在的那些行成立。此后对源表的任何写入都会使副本过时,写入之前得到的摘要不再适用。检查应当放在对源表的最后一次写入之后,并且在任何人读取副本之前。
工作准则: 在对源表的最后一次写入之后,对两侧计算摘要,使用相同的列和带类型的值。把行数和摘要放在一起比较。在两者都一致之前,不要切换读取方,也不要删除源表。如果复制期间有写入方在运行,就重新复制并重新计算摘要。
这部分尚未翻译,因此以英文显示。
What goes wrong in production
A failed migration usually passes the first check and fails later. These are the failures to look for, and how you would notice each.
- A cutover that races a writer. A digest covers only the rows it read, as the digest section says. The first append after it makes the copy stale, and the digest does not warn you. Stop the writers, copy, digest and then move the readers. If the writers cannot stop, copy in windows and check the rows that arrived after the last window.
- Deleting the source before the checks pass. The source holds the history that the copy does not. Once the source is gone, time travel into its snapshots and versions goes with it. Keep the source until the digests match and every reader has moved. Then decide how long to keep it, in writing.
- A copy with the rows and without the layout. A copy made without the
ALTERwould hold every row and match the digest, with no partition set. The partition list and the files per partition are the checks that show the layout is missing, so print both after the copy. - Timestamps that change without an error. Delta Lake cut the nanosecond values to microseconds and raised no error. The row count stayed the same. A check on the values is the only thing that catches it, so a copy with sub-microsecond data needs that check before it runs.
- Names and types that differ. pyiceberg refuses an append whose column names differ from the table's, or whose type it cannot promote, and that refusal is the clearest failure. The harder case is a column whose name matches and whose type it accepts, such as a
longthat was anint. Print both schemas and read them, rather than trusting that the append succeeded. - Size and memory. The helpers in this chapter read a whole table into memory before they write it. That suits 1499 rows. A table of billions of rows needs a copy that reads and writes in batches, and the digest must then be computed over the same batches on both sides, so it stays a check of the whole table.
- Readers that write to both tables. During a copy, a job that writes to both tables by mistake makes the two tables differ, and the digest will show a mismatch that no single write explains. Name one writer per table in the runbook before the cutover starts.
- A digest over text. A digest over casts can match when the types differ. The check in this chapter compares typed values, and the runbook should say so, so that a later edit to the check does not quietly drop the types.
本章做出的决定
- 复制经由 Arrow 进行,每个目标由拥有其格式的库来写入。 pyiceberg 写入 Iceberg,DuckDB 的
ducklake扩展写入 DuckLake,deltalake 写入 Delta Lake。本书不使用 DuckDB 的iceberg扩展或delta扩展,因此书中没有任何一次复制依赖它们。 - Delta 表通过
read_delta读取, 原因见第 8 章。本章中所有对 Delta 的读取都经由它进行。 - 一次复制是一次性的,会开启一段新的历史,并且在任何人切换到它之前要先经过检查。 在摘要一致且读取方已经迁移之前,源保持原状。
- 目标上的分区规范由人工设置,在插入行之前用
ALTER完成,并将其打印出来。 复制不会把它带过去。 - 每个复制示例都用截取点之后相同的三个小时数据构建自己的源。 每个示例都独立运行,每次检查只读取该示例自己写入的行。
- 检查在带类型的值上进行,在最后一次写入之后,覆盖相同的列。 行数和摘要要一起读取,不能只凭其中之一就判定一次复制是好的。
- 复制使用微秒级时间戳,另有一个单独的示例检查亚微秒值。 本章中的每次复制都保留其微秒值,因此摘要一致。精度示例展示了各种格式如何处理它无法保留的值。
- 这里不使用仅元数据的迁移。 pyiceberg 的
Table.add_files会把已经存在的 Parquet 文件作为数据文件添加到 Iceberg 表中(Table.add_files,位于pyiceberg/table/__init__.py,pyiceberg 0.12.0)。它的 API 文档并未固定到该版本,文档把这一操作描述为提交这些文件而不重写它们。这会让数据文件留在原处,而复制则不会。本章没有运行它,而是改为复制行。
本章,在本版次中
版次 2026.11.10示例于 2026年11月20日 通过本章 4个示例 个示例中有 4 个通过
译自英文版图书,版次 2026.11.10。