Glossary

The 50 terms Open Table Formats defines, each in the book's own words, with the section that defines it.

From Edition 2026.11.10, the book's current edition. Where a definition depends on a package, it says the version this edition pins and runs its examples on.

A

append

For an append, the operation is append, which the Iceberg spec describes as "Only data files were added and no files were removed."

Defined in A commit names the table's files, chapter 1. In the book since Edition 2026.11.10. Permanent link to append

C

catalog

An Iceberg table needs a catalog, which is the registry that maps a table name to the table's current metadata.

Defined in Create the table in a catalog, chapter 1. In the book since Edition 2026.11.10. Permanent link to catalog

cleanup

After expiry, cleanup deletes the files that no remaining snapshot lists.

Defined in Snapshots keep files alive, chapter 7. In the book since Edition 2026.11.10. Permanent link to cleanup

column mapping

The Delta Lake protocol defines an optional column mapping feature that separates the name in the files from the name readers see

Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to column mapping

commit also commits

A change to the table is a commit, which writes new metadata in a single step.

Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to commit

compaction

Compaction rewrites many small data files into fewer, larger ones.

Defined in Hourly appends leave small files, chapter 6. In the book since Edition 2026.11.10. Permanent link to compaction

copy

A copy reads the rows of one table at one moment and writes them into another table, in another format.

Defined in What a copy keeps and loses, chapter 9. In the book since Edition 2026.11.10. Permanent link to copy

copy-on-write

In copy-on-write, the writer rewrites each data file that held a matching row and leaves out the deleted rows.

  • Contrasts with: merge-on-read
  • Implemented by: pyiceberg 0.12.0, deltalake 1.6.6

Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to copy-on-write

correction also corrections

A correction is a set of rows with keys, and for each key the table either already holds a row, which should change, or does not, which should be added.

Defined in Late data arrives as a correction, chapter 5. In the book since Edition 2026.11.10. Permanent link to correction

cutover problem

Commits that land after a copy's scan and never reach the copy are the cutover problem

Defined in Copy the pageviews table into DuckLake, chapter 9. In the book since Edition 2026.11.10. Permanent link to cutover problem

D

data file also data files

A data file is a Parquet file that holds rows of a table.

Defined in Compact the small files that hourly appends leave, chapter 6. In the book since Edition 2026.11.10. Permanent link to data file

data path

The data path, the folder where DuckLake writes Parquet files, is stored in the catalog.

True for duckdb 2.0.0, as this edition pins it.

Defined in Attach a DuckLake catalog, chapter 8. In the book since Edition 2026.11.10. Permanent link to data path

day transform

the day transform puts each row under its calendar day

  • Implemented by: pyiceberg 0.12.0

Defined in Move from daily to hourly partitions, chapter 4. In the book since Edition 2026.11.10. Permanent link to day transform

delete

A delete removes the rows that match a predicate.

Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to delete

delete file also delete files

a delete file that records which rows are gone

Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to delete file

deletion vector also deletion vectors

Delta Lake also has deletion vectors, which record deleted row positions in a file.

Defined in Copy-on-write and merge-on-read, chapter 5. In the book since Edition 2026.11.10. Permanent link to deletion vector

digest also sorted digest

The digest of a table is a hash of its sorted rows

Defined in Check each copy with a sorted digest, chapter 9. In the book since Edition 2026.11.10. Permanent link to digest

dimension table

A dimension table holds labels, not counts

Defined in Merge a changed label into the domains table, chapter 5. In the book since Edition 2026.11.10. Permanent link to dimension table

display name

the name readers use is a display name that can change without touching the files

Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to display name

dry run

A dry run lists the files that a real vacuum would delete and deletes none of them.

True for deltalake 1.6.6, as this edition pins it.

Defined in Dry run first, chapter 7. In the book since Edition 2026.11.10. Permanent link to dry run

DuckLake snapshot

DuckLake keeps a snapshot for each change to the catalog, and the change includes creating a schema and creating a table.

True for duckdb 2.0.0, as this edition pins it.

  • A kind of: snapshot
  • Implemented by: duckdb 2.0.0

Defined in Read a commit by its number in each format, chapter 2. In the book since Edition 2026.11.10. Permanent link to DuckLake snapshot

E

expiry also snapshot expiry

expiry removes snapshots from the metadata, so time travel can't reach them

  • Comes before: cleanup
  • Implemented by: pyiceberg 0.12.0, duckdb 2.0.0

Defined in Snapshots keep files alive, chapter 7. In the book since Edition 2026.11.10. Permanent link to expiry

F

field ID also field IDs

Iceberg gives every column an integer field ID when the column is created, and it matches columns by that ID.

Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to field ID

H

hour transform

the hour transform puts each row under its hour

  • Implemented by: pyiceberg 0.12.0

Defined in Move from daily to hourly partitions, chapter 4. In the book since Edition 2026.11.10. Permanent link to hour transform

I

Iceberg REST catalog also REST catalog

The Iceberg REST catalog is a catalog of another kind, a service instead of a file.

Defined in What a catalog holds, chapter 8. In the book since Edition 2026.11.10. Permanent link to Iceberg REST catalog

K

key

In this book, a row is identified by its key: hour_utc, domain_code and page_title together.

Defined in Late data arrives as a correction, chapter 5. In the book since Edition 2026.11.10. Permanent link to key

M

manifest also manifests

each manifest names data files along with their statistics

Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to manifest

manifest list

the manifest list names manifests

Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to manifest list

MERGE

A MERGE statement names the target table, the source batch and the key in its ON clause, and each WHEN clause says what to do with a matched or an unmatched row.

  • Implemented by: duckdb 2.0.0

Defined in MERGE in DuckLake, chapter 5. In the book since Edition 2026.11.10. Permanent link to MERGE

merge-on-read

In merge-on-read, the writer leaves the data file alone and writes a delete file that records which rows are gone.

  • Implemented by: duckdb 2.0.0

Defined in Delete rows, and what each format writes, chapter 5. In the book since Edition 2026.11.10. Permanent link to merge-on-read

metadata

A table's metadata is a record of the table's schema and of the data files that make up the table at each point in its history.

Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to metadata

metadata file

An Iceberg table keeps a metadata file that names its current snapshot.

Defined in What a table format adds to files, chapter 1. In the book since Edition 2026.11.10. Permanent link to metadata file

O

operation

A snapshot's summary has a required field, operation, which is one of append, replace, overwrite or delete.

Defined in Every commit is a snapshot, chapter 2. In the book since Edition 2026.11.10. Permanent link to operation

orphan also orphan file, orphan files, orphans

A file that no snapshot lists is an orphan.

Defined in Find orphan files after expiry, chapter 7. In the book since Edition 2026.11.10. Permanent link to orphan

P

partition spec also partition specs

A partition spec says how a table groups its rows, for example one group per day of hour_utc.

Defined in Partition specs are versioned too, chapter 4. In the book since Edition 2026.11.10. Permanent link to partition spec

partitionColumns

The Delta Lake protocol defines partitionColumns as an array containing the names of columns by which the data is partitioned.

Defined in Partitioning in the other formats, chapter 4. In the book since Edition 2026.11.10. Permanent link to partitionColumns

physical name

every column gets a physical name stored in the files

Defined in Columns are tracked by ID, chapter 3. In the book since Edition 2026.11.10. Permanent link to physical name

plan count

A plan count is a count of files, not of bytes.

Defined in Check what a query scans, chapter 4. In the book since Edition 2026.11.10. Permanent link to plan count

R

reader check

A reader check, in the sense used here, opens the table through the reader's own interface, before the first production write.

Defined in Deletion vectors need a protocol upgrade, chapter 10. In the book since Edition 2026.11.10. Permanent link to reader check

reader feature also reader features

A reader feature in a Delta table's protocol is a promise that every reader handles it, so a reader without it cannot read the table correctly.

Defined in Deletion vectors need a protocol upgrade, chapter 10. In the book since Edition 2026.11.10. Permanent link to reader feature

registered name

A registered name is a second route to the same table, not a second table.

Defined in Register a table from its metadata file, chapter 8. In the book since Edition 2026.11.10. Permanent link to registered name

S

scan

A scan is the table's own reading of a snapshot.

  • Implemented by: pyiceberg 0.12.0

Defined in Query the table with DuckDB, chapter 1. In the book since Edition 2026.11.10. Permanent link to scan

snapshot also snapshots

The Apache Iceberg table spec defines a snapshot as the state of a table at some time, which a reader uses to find every data file in the table.

Defined in A commit names the table's files, chapter 1. In the book since Edition 2026.11.10. Permanent link to snapshot

snapshot ID also snapshot IDs

An Iceberg snapshot ID is a long integer that the spec requires to be unique

Defined in Append each hour as a commit, chapter 1. In the book since Edition 2026.11.10. Permanent link to snapshot ID

snapshot summary also summary

The snapshot summary is what the table records about its own commits

Defined in Append each hour as a commit, chapter 1. In the book since Edition 2026.11.10. Permanent link to snapshot summary

SQL catalog also SqlCatalog

The pyiceberg SqlCatalog keeps its tables in a database.

True for pyiceberg 0.12.0, as this edition pins it.

  • A kind of: catalog
  • Implemented by: pyiceberg 0.12.0

Defined in Register a table in a SQL catalog, chapter 8. In the book since Edition 2026.11.10. Permanent link to SQL catalog

T

tombstone also remove action

A Delta commit that replaces a file adds a remove action to the log, a tombstone that says the file is no longer part of the table, and the file stays on disk.

Defined in Vacuum a table with deltalake, chapter 7. In the book since Edition 2026.11.10. Permanent link to tombstone

U

upsert also upserts

An upsert updates the row whose key matches and inserts the row whose key does not.

  • Implemented by: pyiceberg 0.12.0

Defined in Upsert a corrected hour, chapter 5. In the book since Edition 2026.11.10. Permanent link to upsert

V

vacuum

Vacuum deletes the files whose tombstones are older than the retention.

True for deltalake 1.6.6, as this edition pins it.

  • A kind of: cleanup
  • Implemented by: deltalake 1.6.6

Defined in Vacuum a table with deltalake, chapter 7. In the book since Edition 2026.11.10. Permanent link to vacuum

version also versions, Delta version

Delta Lake calls each commit a version

Defined in Every commit is a snapshot, chapter 2. In the book since Edition 2026.11.10. Permanent link to version