Open Table Formats
Written with AI. Verified by people. How we make books
Anyone can check this book or correct it. Report an error · How to fix one
How Iceberg, Delta Lake and DuckLake store a table, change it, clean it up, register it in a catalog and move it to another format, on one hourly pageviews table, with examples that run offline on a laptop.
Who it's for For data engineers and analytics engineers who write SQL and Python and now have to decide how their lakehouse tables are stored, changed, cleaned up, shared and moved between formats.
Read on Kindle or any e-reader
The EPUB also works in Apple Books, Kobo and most e-readers.
Send to Kindle
-
Approve our address
Amazon delivers to a Kindle only from addresses you've approved, once for each service that emails it. Add ours to your Approved Personal Document E-mail List.
books@rollingedition.comOpen Amazon's Personal Document Settings. On Amazon, it's under Manage Your Content and Devices, Preferences, Personal Document Settings.
Sent. It'll appear in your Kindle library in a few minutes.
If it doesn't arrive, it's almost always the approval in step 1: check that books@rollingedition.com is on your Approved Personal Document E-mail List.
Each new edition of your saved books will go to your Kindle too.
That isn't a Kindle address. It ends in @kindle.com: find it in Amazon's Personal Document Settings.
You've sent {count} books today, the most we send for one account in a day. Try again tomorrow.
We've sent as many books as we can today. Try again tomorrow.
The book didn't send. Try again in a few minutes.
This edition is too large to email.
Send to Kindle is for members.
That isn't a Kindle address. It ends in @kindle.com: find it in Amazon's Personal Document Settings.
The book didn't send. Try again in a few minutes.
Or download the EPUB and drop it on Amazon's Send to Kindle page.
Kindle is a trademark of Amazon.com, Inc.
Labs
Work through this book in the Lab
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Opens in about 30 seconds.
Run every example exactly as the book did, change the code to see what happens, and try your own ideas on the book's code and data. Nothing to install, and your work is saved. Included with membership: 20 hours a month with Member, or 60 with Member Plus.
Checking your hours…
Checking for your Lab…
Contents
10 chapters, about 2 hours 47 minutes to read or 3 hours 21 minutes to listen to, with 39 examples.
-
1Build a first table, end to end
By the end of this chapter you can commit batches of rows to a table, read the table back in DuckDB, and check the totals.
-
2Read a table as it was at an earlier snapshot
By the end of this chapter you can read a table as it was after an earlier commit, in each of the three formats.
-
3Add and rename columns without rewriting files
By the end of this chapter you can add a column to a table that already holds data, rename a column in Iceberg without rewriting its data files, and say why a rename in Delta Lake is a rewrite of the whole table.
-
4Change partitioning without rewriting old files
By the end of this chapter you can change the partitioning of a table in Iceberg, Delta Lake and DuckLake, and count the files a query plans before and after the change.
-
5Correct rows with deletes, updates and merges
By the end of this chapter you can delete, upsert and merge rows in Iceberg, Delta Lake and DuckLake, say what each write does to the files, and count the snapshots a pyiceberg upsert adds.
-
6Compact the small files that hourly appends leave
By the end of this chapter you can count the data files a table holds, merge them into fewer files in Delta Lake, DuckLake and Iceberg, and say what each merge leaves on disk.
-
7Expire snapshots and remove unreferenced files
By the end of this chapter you can expire old snapshots in each of the three formats, remove the data files that only those snapshots still listed, and show that a read of an expired snapshot fails.
-
8Register tables in a catalog and query them from DuckDB
By the end of this chapter you can register an Iceberg table in a SQL catalog, attach a DuckLake catalog, open a Delta Lake table by its path, and run one DuckDB statement that joins a table from Iceberg with a table from DuckLake.
-
9Move a table between formats and check the copy
By the end of this chapter you can copy a table from one format into another, check that the copy holds the same rows, and name what the copy leaves behind.
-
10Choose a format from what your tools support
By the end of this chapter, you can list every tool that reads or writes your tables, measure what each one supports at its pinned version, and choose a format and its features from those measurements.