Glossary
The 50 terms Open Table Formats defines, each in the book's own words, with the section that defines it.
From Edition 2026.11.10, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.
A
- append
-
For an append, the operation is append, which the Iceberg spec describes as "Only data files were added and no files were removed."
- A kind of: operation
Defined in A commit names the table's files, chapter 1. In the book since Edition 2026.11.10. Permanent link to append
C
- catalog
-
An Iceberg table needs a catalog, which is the registry that maps a table name to the table's current metadata.
Defined in Create the table in a catalog, chapter 1. In the book since Edition 2026.11.10. Permanent link to catalog
- cleanup
-
After expiry, cleanup deletes the files that no remaining snapshot lists.
Defined in Snapshots keep files alive, chapter 7. In the book since Edition 2026.11.10. Permanent link to cleanup
- column mapping
-
The Delta Lake protocol defines an optional column mapping feature that separates the name in the files from the name readers see
Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to column mapping
- commit also commits
-
A change to the table is a commit, which writes new metadata in a single step.
- Comes before: snapshot
Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to commit
- compaction
-
Compaction rewrites many small data files into fewer, larger ones.
- A kind of: commit
Defined in Hourly appends leave small files, chapter 6. In the book since Edition 2026.11.10. Permanent link to compaction
- copy
-
A copy reads the rows of one table at one moment and writes them into another table, in another format.
- Contrasts with: registered name
Defined in What a copy keeps and loses, chapter 9. In the book since Edition 2026.11.10. Permanent link to copy
- copy-on-write
-
In copy-on-write, the writer rewrites each data file that held a matching row and leaves out the deleted rows.
- Contrasts with: merge-on-read
- Implemented by: pyiceberg 0.12.0, deltalake 1.6.6
Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to copy-on-write
- correction also corrections
-
A correction is a set of rows with keys, and for each key the table either already holds a row, which should change, or does not, which should be added.
Defined in Late data arrives as a correction, chapter 5. In the book since Edition 2026.11.10. Permanent link to correction
- cutover problem
-
Commits that land after a copy's scan and never reach the copy are the cutover problem
Defined in Copy the pageviews table into DuckLake, chapter 9. In the book since Edition 2026.11.10. Permanent link to cutover problem
D
- data file also data files
-
A data file is a Parquet file that holds rows of a table.
Defined in Compact the small files that hourly appends leave, chapter 6. In the book since Edition 2026.11.10. Permanent link to data file
- data path
-
The data path, the folder where DuckLake writes Parquet files, is stored in the catalog.
True for duckdb 2.0.0, as this edition pins it.
Defined in Attach a DuckLake catalog, chapter 8. In the book since Edition 2026.11.10. Permanent link to data path
- day transform
-
the day transform puts each row under its calendar day
- Implemented by: pyiceberg 0.12.0
Defined in Move from daily to hourly partitions, chapter 4. In the book since Edition 2026.11.10. Permanent link to day transform
- delete
-
A delete removes the rows that match a predicate.
Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to delete
- delete file also delete files
-
a delete file that records which rows are gone
- Part of: merge-on-read
Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to delete file
- deletion vector also deletion vectors
-
Delta Lake also has deletion vectors, which record deleted row positions in a file.
- A kind of: merge-on-read
Defined in Copy-on-write and merge-on-read, chapter 5. In the book since Edition 2026.11.10. Permanent link to deletion vector
- digest also sorted digest
-
The digest of a table is a hash of its sorted rows
Defined in Check each copy with a sorted digest, chapter 9. In the book since Edition 2026.11.10. Permanent link to digest
- dimension table
-
A dimension table holds labels, not counts
Defined in Merge a changed label into the domains table, chapter 5. In the book since Edition 2026.11.10. Permanent link to dimension table
- display name
-
the name readers use is a display name that can change without touching the files
- Part of: column mapping
Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to display name
- dry run
-
A dry run lists the files that a real vacuum would delete and deletes none of them.
True for deltalake 1.6.6, as this edition pins it.
Defined in Dry run first, chapter 7. In the book since Edition 2026.11.10. Permanent link to dry run
- DuckLake snapshot
-
DuckLake keeps a snapshot for each change to the catalog, and the change includes creating a schema and creating a table.
True for duckdb 2.0.0, as this edition pins it.
- A kind of: snapshot
- Implemented by: duckdb 2.0.0
Defined in Read a commit by its number in each format, chapter 2. In the book since Edition 2026.11.10. Permanent link to DuckLake snapshot
E
- expiry also snapshot expiry
-
expiry removes snapshots from the metadata, so time travel can't reach them
- Comes before: cleanup
- Implemented by: pyiceberg 0.12.0, duckdb 2.0.0
Defined in Snapshots keep files alive, chapter 7. In the book since Edition 2026.11.10. Permanent link to expiry
F
- field ID also field IDs
-
Iceberg gives every column an integer field ID when the column is created, and it matches columns by that ID.
Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to field ID
H
- hour transform
-
the hour transform puts each row under its hour
- Implemented by: pyiceberg 0.12.0
Defined in Move from daily to hourly partitions, chapter 4. In the book since Edition 2026.11.10. Permanent link to hour transform
I
- Iceberg REST catalog also REST catalog
-
The Iceberg REST catalog is a catalog of another kind, a service instead of a file.
- A kind of: catalog
- Contrasts with: SQL catalog
Defined in What a catalog holds, chapter 8. In the book since Edition 2026.11.10. Permanent link to Iceberg REST catalog
K
- key
-
In this book, a row is identified by its key: hour_utc, domain_code and page_title together.
- Comes before: correction, upsert
Defined in Late data arrives as a correction, chapter 5. In the book since Edition 2026.11.10. Permanent link to key
M
- manifest also manifests
-
each manifest names data files along with their statistics
- Part of: metadata
Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to manifest
- manifest list
-
the manifest list names manifests
- Part of: metadata
Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to manifest list
- MERGE
-
A MERGE statement names the target table, the source batch and the key in its ON clause, and each WHEN clause says what to do with a matched or an unmatched row.
- Implemented by: duckdb 2.0.0
Defined in MERGE in DuckLake, chapter 5. In the book since Edition 2026.11.10. Permanent link to MERGE
- merge-on-read
-
In merge-on-read, the writer leaves the data file alone and writes a delete file that records which rows are gone.
- Implemented by: duckdb 2.0.0
Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to merge-on-read
- metadata
-
A table's metadata is a record of the table's schema and of the data files that make up the table at each point in its history.
Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to metadata
- metadata file
-
An Iceberg table keeps a metadata file that names its current snapshot.
- Part of: metadata
Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to metadata file
O
- operation
-
A snapshot's summary has a required field, operation, which is one of append, replace, overwrite or delete.
- Part of: snapshot summary
Defined in Every commit is a snapshot, chapter 2. In the book since Edition 2026.11.10. Permanent link to operation
- orphan also orphan file, orphan files, orphans
-
A file that no snapshot lists is an orphan.
Defined in Find orphan files after expiry, chapter 7. In the book since Edition 2026.11.10. Permanent link to orphan
P
- partition spec also partition specs
-
A partition spec says how a table groups its rows, for example one group per day of hour_utc.
- Part of: metadata
Defined in Partition specs are versioned too, chapter 4. In the book since Edition 2026.11.10. Permanent link to partition spec
- partitionColumns
-
The Delta Lake protocol defines partitionColumns as an array containing the names of columns by which the data is partitioned.
Defined in Partitioning in the other formats, chapter 4. In the book since Edition 2026.11.10. Permanent link to partitionColumns
- physical name
-
every column gets a physical name stored in the files
- Part of: column mapping
Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to physical name
- plan count
-
A plan count is a count of files, not of bytes.
Defined in Check what a query scans, chapter 4. In the book since Edition 2026.11.10. Permanent link to plan count
R
- reader check
-
A reader check, in the sense used here, opens the table through the reader's own interface, before the first production write.
Defined in Deletion vectors need a protocol upgrade, chapter 10. In the book since Edition 2026.11.10. Permanent link to reader check
- reader feature also reader features
-
A reader feature in a Delta table's protocol is a promise that every reader handles it, so a reader without it cannot read the table correctly.
Defined in Deletion vectors need a protocol upgrade, chapter 10. In the book since Edition 2026.11.10. Permanent link to reader feature
- registered name
-
A registered name is a second route to the same table, not a second table.
Defined in Register a table from its metadata file, chapter 8. In the book since Edition 2026.11.10. Permanent link to registered name
S
- scan
-
A scan is the table's own reading of a snapshot.
- Implemented by: pyiceberg 0.12.0
Defined in Query the table with DuckDB, chapter 1. In the book since Edition 2026.11.10. Permanent link to scan
- snapshot also snapshots
-
The Apache Iceberg table spec defines a snapshot as the state of a table at some time, which a reader uses to find every data file in the table.
Defined in A commit names the table's files, chapter 1. In the book since Edition 2026.11.10. Permanent link to snapshot
- snapshot ID also snapshot IDs
-
An Iceberg snapshot ID is a long integer that the spec requires to be unique
- Part of: snapshot
Defined in Append each hour as a commit, chapter 1. In the book since Edition 2026.11.10. Permanent link to snapshot ID
- snapshot summary also summary
-
The snapshot summary is what the table records about its own commits
- Part of: snapshot
Defined in Append each hour as a commit, chapter 1. In the book since Edition 2026.11.10. Permanent link to snapshot summary
- SQL catalog also SqlCatalog
-
The pyiceberg SqlCatalog keeps its tables in a database.
True for pyiceberg 0.12.0, as this edition pins it.
- A kind of: catalog
- Implemented by: pyiceberg 0.12.0
Defined in Register a table in a SQL catalog, chapter 8. In the book since Edition 2026.11.10. Permanent link to SQL catalog
T
- tombstone also remove action
-
A Delta commit that replaces a file adds a remove action to the log, a tombstone that says the file is no longer part of the table, and the file stays on disk.
- Comes before: vacuum
Defined in Vacuum a table with deltalake, chapter 7. In the book since Edition 2026.11.10. Permanent link to tombstone
U
- upsert also upserts
-
An upsert updates the row whose key matches and inserts the row whose key does not.
- Implemented by: pyiceberg 0.12.0
Defined in Upsert a corrected hour, chapter 5. In the book since Edition 2026.11.10. Permanent link to upsert
V
- vacuum
-
Vacuum deletes the files whose tombstones are older than the retention.
True for deltalake 1.6.6, as this edition pins it.
- A kind of: cleanup
- Implemented by: deltalake 1.6.6
Defined in Vacuum a table with deltalake, chapter 7. In the book since Edition 2026.11.10. Permanent link to vacuum
- version also versions, Delta version
-
Delta Lake calls each commit a version
- Contrasts with: snapshot ID
Defined in Every commit is a snapshot, chapter 2. In the book since Edition 2026.11.10. Permanent link to version