Add and rename columns without rewriting files

By the end of this chapter you can add a column to a table that already holds data, rename a column in Iceberg without rewriting its data files, and say why a rename in Delta Lake is a rewrite of the whole table. The versions this book pins are pyiceberg 0.12.0 and deltalake 1.6.6.

The running case is wiki.pageviews_hourly, the table of hourly view counts for six hours of Wikimedia pageviews that the first chapter loads. Two requests reach it after the first three hours have been written. The first asks for a lineage column: each row should record which hourly load wrote it, so that a bad load can be traced back to its rows. The table needs a column for that, and the rows it already holds were written before the column existed. The second is a naming fix: the dashboards call the title column article_title, while the table calls it page_title, the name the dump uses. Both are schema changes on a table that already has data. For each one, the question is what happens to the files that were written before the change.

A schema change is a metadata change in principle. The data files stay where they are, and the format records the added or renamed columns in its metadata. Whether the old files still read correctly depends on how the format finds a column inside a file. A format that finds columns by name has trouble when a name changes. A format that finds them by something that does not change does not. This chapter shows which of the two each format is, what each change costs, and which change the pinned Delta Lake library cannot make.

Columns are tracked by ID

A data file holds values for some columns, and the table's schema says which column each value belongs to. The simple way to match them is by name. That works until a name changes. The schema then names a column the old file does not have under that name, and the reader has to guess what the old values belong to.

A reader that matches by name and finds no column under the old name has two bad options: fail the query, or fill the column with null without saying so. The second is the quieter failure, and a dashboard can show it for weeks before anyone looks. We count the quiet null as the worse failure, which is a judgement, not a measured result. Matching by identity avoids both, because the column keeps its identity whatever its name is.

Iceberg gives every column an integer field ID when the column is created, and it matches columns by that ID. The name is a label that can change. The Iceberg specification says so directly: renaming a field "must change the name, but not the field ID" [1]. When a reader opens an old file, it matches each column by field ID. If the file holds no value for a field, the spec's projection rule first checks two cases that do not apply here, an identity-partition value and a name mapping. It then returns the field's initial-default when one is set, and otherwise returns null: "Return null in all other cases" [1]. The spec defines initial-default for a column added with a default. This chapter does not set one, so an added column reads as null in every old row.

You can see the IDs yourself. The schema of an Iceberg table lists each field with its ID, and the example in the rename section prints the ID of one column before and after a rename. The ID is the value to check when you want to know whether a rename kept the column's identity.

Delta Lake starts from a different place. Its table schema names each column, and a reader matches by name. The Delta Lake protocol defines an optional column mapping feature that separates the two: every column gets a physical name stored in the files, and the name readers use is a display name that can change without touching the files. The protocol says column mapping supports "the renaming and dropping of columns without having to rewrite all the data" [2]. Whether a given library release does that is a separate question. The section on Delta Lake answers it for the pinned release.

Working rule: Expect an Iceberg column to keep its data through a rename, because the reader matches by field ID. Expect a Delta Lake column to match by name unless column mapping is on. Test a rename on a copy of the table before you rely on it, and read the column names back afterwards.

Add load_batch after three hours

The first change is the one the case needs first: a column that does not exist in the files already written. In a pipeline, that happens on an ordinary day. The pipeline starts stamping each load with its number, and the table already holds rows that predate the stamp. The example starts from the first three hours, each written as its own commit, so the table holds three snapshots and three data files. An Iceberg snapshot is the table's state after a commit that changes data. Each append in this chapter adds one, and a schema change adds none.

In Iceberg, you add a column inside a schema update. The call takes the name and the type, and the table assigns the field ID:

with table.update_schema() as update:
    update.add_column("load_batch", IntegerType())

The example runs the three early commits, adds the column, then appends hours 03 to 05. Each of those loads sets load_batch to its hour number, so its rows carry the value. It then counts rows by hour of the day: hours 00 to 02 were written before the add, and hours 03 to 05 after it. Each hour is one write, so the split is also the split by load. Each count is printed with a label, so the text can quote it.

1"""Add load_batch after three hours of appends, in Iceberg and in Delta Lake.23load_batch holds the number of the hourly load that wrote each row. Rows written4before the add have no value for it, so they read as null. Later loads set it.5"""67import pyarrow as pa8from deltalake import DeltaTable, write_deltalake9from deltalake.schema import Field, PrimitiveType10from pyiceberg.types import IntegerType1112from lib.lakehouse import ICEBERG_ID, delta_file_count, iceberg_file_count, new_warehouse, read_cut_hour, read_delta, split_nulls131415def stamp(rows, hour):16    """Return one hourly load with load_batch set to that load's hour number."""17    return rows.append_column("load_batch", pa.array([hour] * rows.num_rows, type=pa.int32()))181920with new_warehouse() as wh:21    table = wh.catalog.create_table(ICEBERG_ID, schema=read_cut_hour(0).schema)22    for hour in range(3):23        table.append(read_cut_hour(hour))24    print(f"iceberg snapshots before the add: {len(table.snapshots())}")25    print(f"iceberg data files before the add: {iceberg_file_count(table)}")26    with table.update_schema() as update:27        update.add_column("load_batch", IntegerType())28    print(f"iceberg snapshots after the add: {len(table.snapshots())}")29    print(f"iceberg data files after the add: {iceberg_file_count(table)}")30    for hour in range(3, 6):31        table.append(stamp(read_cut_hour(hour), hour))32    iceberg_rows = table.scan().to_arrow()33    print(f"iceberg rows in the table: {iceberg_rows.num_rows}")34    split_nulls(wh.con, iceberg_rows, "iceberg", "load_batch", 3, sides=("written before the add", "written after the add"))3536    path = f"{wh.root}/delta/pageviews_hourly"37    for hour in range(3):38        write_deltalake(path, read_cut_hour(hour), mode="append")39    print(f"delta version before the add: {DeltaTable(path).version()}")40    print(f"delta data files before the add: {delta_file_count(DeltaTable(path))}")41    DeltaTable(path).alter.add_columns([Field("load_batch", PrimitiveType("integer"))])42    print(f"delta version after the add: {DeltaTable(path).version()}")43    print(f"delta data files after the add: {delta_file_count(DeltaTable(path))}")44    for hour in range(3, 6):45        write_deltalake(path, stamp(read_cut_hour(hour), hour), mode="append")46    delta_rows = read_delta(path)47    print(f"delta rows in the table: {delta_rows.num_rows}")48    split_nulls(wh.con, delta_rows, "delta", "load_batch", 3, sides=("written before the add", "written after the add"))

iceberg snapshots before the add: 3
iceberg data files before the add: 3
iceberg snapshots after the add: 3
iceberg data files after the add: 3
iceberg rows in the table: 2999
iceberg rows written before the add: 1499
iceberg rows written before the add with load_batch null: 1499
iceberg rows written after the add: 1500
iceberg rows written after the add with load_batch null: 0
delta version before the add: 2
delta data files before the add: 3
delta version after the add: 3
delta data files after the add: 3
delta rows in the table: 2999
delta rows written before the add: 1499
delta rows written before the add with load_batch null: 1499
delta rows written after the add: 1500
delta rows written after the add with load_batch null: 0

Before the add, the table has 1499 rows, and 1499 of them read null for load_batch. After the add and the three later appends, the table has 2999 rows. The 1500 rows written after the add have 0 nulls in load_batch. The snapshot count stays at 3 across the schema change, and the table keeps 3 data files. The change touched the metadata and no data file. The two halves of the example hold the same rows and the same nulls, so a disagreement between them would point at a reader, not at the data.

Delta Lake adds a column through its alter object. The call takes a field, and the table's version goes up by one:

DeltaTable(path).alter.add_columns([Field("load_batch", PrimitiveType("integer"))])

The same example does the Delta half. It writes the three early hours as three appends, adds the column, and appends hours 03 to 05. Delta counts versions from 0, so the first write is version 0. Every version number this chapter prints is Delta's real one. The version moves from 2 to 3, and the table keeps 3 data files. Before the add there are 1499 rows, all null for load_batch. After the add and the three later appends the table has 2999 rows, and the 1500 rows written after the add have 0 nulls. The example's table has no column mapping.

Both formats give the same result for this case. The old rows stay as they were written, and the column reads as null for them. The writer did not have to touch a single old file.

Working rule: Add a column with the format's own call, and never by rewriting the files. Expect the rows written before the add to read null for the column, and check that they do.

Old files read as null

A null in an added column can look like missing data, and a reader who sees it may assume the old files are wrong. They are not. The old files hold the values the pipeline wrote, and the column they lack is filled on read. The rule the reader follows is the one the columns section quotes from the spec: with no initial-default, no value for the field means null.

That makes the null count a useful check. The example counts nulls by hour of the day, and each hour is one write, so the counts show the pattern to expect: every row written before the add is null, and no row written after it is. In Iceberg the output reads 1499 nulls among 1499 rows. In Delta Lake it reads 1499 among 1499. Both match the rule.

If a count falls outside that pattern, a writer sent a different set of columns, or sent the column with nulls, and the cause is in the pipeline, not in the format. Look at the writer for the rows that break the pattern before you query the column, because a query over a column that is partly null will return an answer that looks reasonable.

Working rule: After an add, count nulls in the added column by hour of the day, where each hour is one write. Rows written before the add should be null, and rows written after it should not be. Any other pattern means a writer sent a different set of columns, or sent nulls in the column, so find that writer before you trust the column.

Rename a column by its ID without rewriting files

With the field ID in place, a rename is one call in Iceberg. The call changes the name the schema reports, and keeps the ID:

with table.update_schema() as update:
    update.rename_column("page_title", "article_title")

The example writes three hours, reads the field ID of page_title, renames the column, and reads the ID again. It counts the data files before and after the call, and it reads the table through DuckDB with the new name in the query.

1"""Rename page_title to article_title in Iceberg, then read old files under the new name.23Iceberg keeps each column's field ID across a rename, so the three files4written under page_title are read under article_title without a rewrite.5"""67import pyarrow as pa89from lib.lakehouse import (10    ICEBERG_ID,11    iceberg_file_count,12    new_warehouse,13    read_cut_hour,14    scan_arrow,15    snapshot_labels,16    sql_rows,17)1819with new_warehouse() as wh:20    table = wh.catalog.create_table(ICEBERG_ID, schema=read_cut_hour(0).schema)21    for hour in range(3):22        table.append(read_cut_hour(hour))2324    before = table.schema().find_field("page_title").field_id25    print(f"page_title field ID before the rename: {before}")26    print(f"data files before the rename: {iceberg_file_count(table)}")27    with table.update_schema() as update:28        update.rename_column("page_title", "article_title")29    print(f"article_title field ID after the rename: {table.schema().find_field('article_title').field_id}")30    print(f"data files after the rename: {iceberg_file_count(table)}")3132    rows = scan_arrow(table)33    print(f"rows: {rows.num_rows}")34    count = sql_rows(wh.con, "SELECT count(article_title) FROM t", t=rows)[0][0]35    print(f"article_title not null: {count}")3637    labels = snapshot_labels([snapshot.snapshot_id for snapshot in table.snapshots()])38    first = table.snapshots()[0].snapshot_id39    print(f"{labels[first]} columns: {', '.join(scan_arrow(table, snapshot_id=first).column_names)}")

page_title field ID before the rename: 3
data files before the rename: 3
article_title field ID after the rename: 3
data files after the rename: 3
rows: 1499
article_title not null: 1499
snapshot 1 columns: hour_utc, domain_code, page_title, views, response_bytes

The field ID of page_title is 3 before the rename, and the field ID of article_title is 3 after it. The table keeps 3 data files on both sides of the call, so no file was rewritten. The DuckDB query counts 1499 non-null article_title values, out of 1499 rows. Every row counts, including the rows that were written under page_title, because the reader matches the files by ID and reads them under article_title.

The dashboards need the name, and the table does not have to be rewritten to give it to them. The data stays as it was written, and the next read sees the renamed column. The example also reads the first snapshot's columns, and they are hour_utc, domain_code, page_title, views and response_bytes. A snapshot reads with the schema it was written with, so an older snapshot still reports the old name. Chapter 2 covers reading older snapshots in general, and the point here is narrower: a rename does not reach back into the history.

The cost comparison is what makes this the right call for Iceberg. A rename that keeps the ID touches the metadata and nothing else, so its cost does not grow with the size of the table. The same rename done by copying the table costs one read and one write of every row. The section on Delta Lake shows what that copy costs in that format.

Working rule: Rename an Iceberg column when its name is wrong, and check the field ID before and after the call. The field ID must be the same, and the data file count must not change. Tell every reader of the table about the rename, because the current schema has only article_title, and an older snapshot still has page_title.

Delta Lake has no rename at this pin

The same rename in Delta Lake has no call at this pin. The Delta Lake table alteration object has no rename method at deltalake 1.6.6, and its methods are listed in the package [3]. The one that looks like a rename is set_column_metadata, because it takes a dictionary of field metadata, and nothing stops you from passing a name key. The call returns without error when you pass a name. It renames nothing on any table, with or without column mapping, and that is the trap this section shows.

The trap example creates a table with column mapping on, which is the protocol's route to a rename, and then sends the call:

DeltaTable(path).alter.set_column_metadata("page_title", {"name": "article_title"})

1"""Rename a Delta Lake column with set_column_metadata, then read the names back.23The call returns without error and writes a commit, but the column keeps its name.4A query that uses the new name fails.5"""67from deltalake import DeltaTable, write_deltalake89from lib.lakehouse import new_warehouse, read_cut_hour, read_delta, safe_error, sql_rows1011with new_warehouse() as wh:12    path = f"{wh.root}/delta/pageviews_hourly"13    write_deltalake(path, read_cut_hour(0),14                    configuration={"delta.columnMapping.mode": "name"})15    print(f"version before the rename: {DeltaTable(path).version()}")16    DeltaTable(path).alter.set_column_metadata("page_title", {"name": "article_title"})17    print("set_column_metadata returned")18    table = DeltaTable(path)19    print(f"version after the rename: {table.version()}")20    page = next(field for field in table.schema().fields if field.name == "page_title")21    print(f"name in page_title metadata after the rename: {page.metadata['name']}")22    print(f"columns read back: {', '.join(read_delta(path).column_names)}")23    try:24        sql_rows(wh.con, "SELECT article_title FROM t", t=read_delta(path))25    except Exception as exc:26        print(safe_error(exc, wh))

version before the rename: 0
set_column_metadata returned
version after the rename: 1
name in page_title metadata after the rename: article_title
columns read back: hour_utc, domain_code, page_title, views, response_bytes
BinderException: Binder Error: Referenced column "article_title" not found in FROM clause!
Candidate bindings: "page_title", "domain_code", "hour_utc", "views", "response_bytes"

LINE 1: SELECT article_title FROM t
               ^^^^^^^^^^^^^

The call returns, and the table's version moves from 0 to 1, so the library wrote a commit. The name key is in the field's metadata. The column is still page_title: the names read back through the library are hour_utc, domain_code, page_title, views and response_bytes. The DuckDB query that asks for article_title fails with a BinderException, and the error lists page_title among the columns it could bind.

The trap is that the call looks like the rename and succeeds at the call. Nothing in the return value says the name is unchanged. Only a read of the names shows the truth.

Column mapping gives no rename at this pin. The protocol's column mapping section describes a rename [2], but deltalake 1.6.6 has no call that performs one. The example creates the table with delta.columnMapping.mode set to name, and the call still changes no name. That shows what set_column_metadata does on a mapped table, and nothing about a rename with mapping, because the library has no call that makes one.

1"""Rename a Delta Lake column by rewriting the table, and read the old version back.23The rewrite gives the current version the new name and writes every data file again.4Older versions keep the names they were written with. Delta numbers versions from 0.5"""67from deltalake import DeltaTable, write_deltalake89from lib.lakehouse import new_warehouse, read_cut_hour, read_delta1011with new_warehouse() as wh:12    path = f"{wh.root}/delta/pageviews_hourly"13    for hour in range(3):14        write_deltalake(path, read_cut_hour(hour), mode="append")15    print(f"data files before the rewrite: {len(DeltaTable(path).file_uris())}")16    before = DeltaTable(path).version()17    print(f"version before the rewrite: {before}")18    print(f"columns before the rewrite: {', '.join(read_delta(path, version=before).column_names)}")1920    renamed = read_delta(path).rename_columns(["hour_utc", "domain_code", "article_title", "views", "response_bytes"])21    write_deltalake(path, renamed, mode="overwrite", schema_mode="overwrite")22    print(f"data files after the rewrite: {len(DeltaTable(path).file_uris())}")23    after = DeltaTable(path).version()24    print(f"version after the rewrite: {after}")25    print(f"columns after the rewrite: {', '.join(read_delta(path).column_names)}")26    print(f"rows after the rewrite: {read_delta(path).num_rows}")

data files before the rewrite: 3
version before the rewrite: 2
columns before the rewrite: hour_utc, domain_code, page_title, views, response_bytes
data files after the rewrite: 1
version after the rewrite: 3
columns after the rewrite: hour_utc, domain_code, article_title, views, response_bytes
rows after the rewrite: 1499

A rewrite is the way to get the name at this pin. The second example reads the whole table through the library, renames the column in memory, and writes the result back with mode="overwrite" and schema_mode="overwrite", which replace the data and the schema together. The table had 3 data files before the rewrite and has 1 after it. The rewrite makes version 3, which reads article_title, and 1499 rows are in it.

The rewrite costs the size of the table, because every row is read and written again. The example's table is small, and the file count falls from 3 to 1. A production table that holds millions of rows pays the same per-row cost, and a team should plan the rewrite as a job of its own. The version before the rewrite, 2, still reads page_title, which the example also prints. The rewrite therefore changes the current version only. Older versions keep the names they were written with. Before you run a rewrite, list the readers that ask for the column by name. The switch happens in one commit, so every reader of the current version moves at the same moment, while a job that pins an older version keeps the old name. Move those jobs with the readers, and record the version number the rewrite replaced until they have moved.

Working rule: Choose Delta Lake column names when the table is created, because at this pin a later rename is a rewrite of the whole table. Do not use set_column_metadata to rename a column: it changes the version and the metadata, and the column keeps its name. After any schema change, read the column names back through the library, as the trap example does, and check them before a query depends on them.

What goes wrong in production

Schema changes fail quietly more often than they fail loudly. Four failures in this chapter are worth knowing, and each one has a way to notice it.

The first is the no-op rename. A set_column_metadata call succeeds, a commit appears in the history, and the column keeps its old name. The failure shows on the first query that uses article_title: the DuckDB query fails with a BinderException. A scheduled job that runs such a query fails on its first run after the change, so the error arrives close to its cause. Read the names back after any change, as the trap example does, before any query runs.

The second is the stale name in history. A rewrite gives the current version the name article_title, but the earlier versions keep the names they were written with, and the rewrite example shows it: version 2 still reads page_title. An Iceberg snapshot does the same: the first snapshot still reports page_title. A query that reads an earlier version or snapshot by article_title finds no such column. Reads by version or snapshot number need the names that were current at that point, so a job that pins a version number also pins its schema.

The third is the null that looks like missing data. An added column reads as null in every row written before the add. A null count on that column is correct, and it is a fault only when it falls outside the before-and-after pattern the add example shows. Count nulls by hour, as the add example does, where each hour is one write, and investigate any row that breaks the pattern.

The fourth is the schema change that does not show in the snapshot list. The Iceberg add and rename both left the snapshot count where it was, and the file count too. A tool that looks for schema changes by counting snapshots will miss them. Compare the schema itself, by field ID and name, when you need to know what changed.

Working rule: After any schema change, do three things before a downstream query runs: read the column names back through the library, count nulls in each added column by hour, where each hour is one write, and compare the schema, not the snapshot count, with what the change was meant to do.

Decisions this chapter made

The table in this section sets the two formats side by side for the changes in this chapter. Each cell comes from an example in this chapter.

• Add a column to a table that already holds data, in both formats. Old rows read it as null, and the add example shows the counts by hour that confirm the pattern. We recommend this over any copy of the table, because it writes no old file.
• Rename an Iceberg column with rename_column, and check that its field ID is the same before and after. We recommend it for any Iceberg column whose name is wrong, because its cost does not depend on the table's size.
• Choose Delta Lake column names when the table is created. At the pinned release, a rename there is a rewrite of the whole table, and that is the cost to plan for when a name is wrong.
• Do not rename a Delta Lake column with set_column_metadata. The call succeeds and changes nothing about the name.
• Do not count on column mapping for a rename at this pin. The library has no call that performs one, so turning mapping on gains no rename.
• Read the column names back after every schema change, in both formats, before a query depends on them.

Sources

1. Apache Iceberg Table Spec, Schema Evolution (Column Projection), https://iceberg.apache.org/spec/, read Nov 6, 2026. Quoted: "Renaming an existing field must change the name, but not the field ID." and "Return null in all other cases."
2. Delta Transaction Log Protocol, Column Mapping section, tag v4.4.1, https://raw.githubusercontent.com/delta-io/delta/v4.4.1/PROTOCOL.md, read Nov 6, 2026. Quoted: "the renaming and dropping of columns without having to rewrite all the data."
3. deltalake 1.6.6, the methods of DeltaTable.alter in the installed package: add_columns, add_constraint, add_feature, drop_column_not_null, drop_constraint, set_column_metadata, set_table_description, set_table_name and set_table_properties. None renames a column. To list them, run dir(DeltaTable(path).alter).