
Schema evolution is the controlled change of a dataset’s structure as producers and consumers develop. Adding a field, renaming a column, or changing a type can affect a reader that still expects yesterday’s data. The aim is to preserve the contracts you intend to support and make breaking changes explicit.
A file that opens successfully can still produce a wrong answer. In the example below, a reader combines two Parquet files after an amount column is renamed. The query succeeds, but its total misses half the intended amount. Explicit consumer checks catch the change before it reaches that calculation.
This guide focuses on file-based datasets. It uses DuckDB to test old and new readers against small Parquet fixtures, then explains how those checks inform a rollout. The same questions arise in event streams and databases, although their compatibility mechanisms differ.
A useful data contract describes names, types, required values, identifiers, units, and the meaning of missing data. It also names the producer and the consumers that must keep working. A schema version identifies a documented contract; a version label alone does not make a change compatible.
Our example has an order_id string and a cents integer. The contract says that amounts are nonnegative USD cents and IDs are nonempty and unique within each input file. The second version adds a nullable region string. For an old file without that field, the new reader returns NULL, meaning unavailable. It does not invent a region or equate missing data with an empty string.
Semantic compatibility requires a separate review. If the value 100 changes from meaning 100 cents to 100 dollars, its name and integer type can remain unchanged while the report becomes wrong. Neither a type check nor a successful cast can discover the producer’s intended unit.
Using the directions described in Confluent’s compatibility documentation, backward compatibility means a new reader can consume old data; forward compatibility means an old reader can consume new data. Full compatibility requires both. The detailed rules depend on the format and the reader’s policy.
For this example, the old reader projects only the two original fields. The new reader explicitly handles an absent region. Those choices produce this matrix:
| Reader | File | Observed behavior |
|---|---|---|
| Old reader | Original schema | Reads order ID and cents |
| Old reader | Added nullable region | Ignores region and reads its required fields |
| New reader | Original schema | Supplies NULL for the absent region |
| New reader | Added nullable region | Reads region, including an explicit NULL |
This does not mean every added optional field is universally compatible. A consumer that rejects unknown fields, uses positional output, or demands a populated region has different requirements. Test the readers you actually operate. If you retain many historical versions, checking only the immediately preceding version also leaves older data untested.
Checked November 17, 2026: the demonstration and 18 local tests passed on Windows with Python 3.12.14 and DuckDB 2.0.0. They cover explicit projections, both compatibility directions, field order, renaming, removal, types, missing values, duplicate IDs, combined-file behavior, and preservation of input bytes. No remote storage, schema registry, table-format engine, deployment, or performance test was run.
In a fresh Python 3.12 virtual environment, install the pinned package:
python -m pip install duckdb==2.0.0
Save the following as schema_demo.py. It creates trusted synthetic files in a temporary directory and removes them on exit. The DuckDB Parquet interface writes the fixtures, inspects their types, and reads named columns. File paths are bound parameters; the only field mapping is an explicit allowlist.
"""Local Parquet compatibility lab using trusted, synthetic fixtures."""
from pathlib import Path
from tempfile import TemporaryDirectory
import duckdb
def write_fixture(connection, path, query):
if path.exists():
raise FileExistsError(path)
connection.sql(query).write_parquet(str(path))
def fixtures(connection, root):
queries = {
"v1": """SELECT * FROM (VALUES
('a', 1200::BIGINT), ('b', 800::BIGINT)) t(order_id, cents)""",
"v2": """SELECT * FROM (VALUES
('c', 1500::BIGINT, 'CH'), ('d', 500::BIGINT, NULL))
t(order_id, cents, region)""",
"reordered": """SELECT region, cents, order_id FROM (VALUES
('c', 1500::BIGINT, 'CH'), ('d', 500::BIGINT, NULL))
t(order_id, cents, region)""",
"renamed": """SELECT * FROM (VALUES
('c', 1500::BIGINT, 'CH'), ('d', 500::BIGINT, NULL))
t(order_id, amount_minor, region)""",
"text_amount": "SELECT 'c' AS order_id, '1500' AS cents",
"removed": "SELECT 'c' AS order_id",
}
paths = {name: root / (name + ".parquet") for name in queries}
for name, query in queries.items():
write_fixture(connection, paths[name], query)
return paths
def read_orders(connection, path, with_region=False, value_column="cents"):
"""Example consumer policy, not a general schema compatibility checker."""
if value_column not in {"cents", "amount_minor"}:
raise ValueError("unsupported field mapping")
description = connection.execute(
"DESCRIBE SELECT * FROM read_parquet(?)", [str(path)]).fetchall()
types = {row[0]: row[1] for row in description}
required = {"order_id": "VARCHAR", value_column: "BIGINT"}
if any(types.get(name) != kind for name, kind in required.items()):
raise ValueError("required field missing or type differs")
region = ""
if with_region:
if "region" in types and types["region"] != "VARCHAR":
raise ValueError("region must be VARCHAR when present")
region = ", region" if "region" in types else ", NULL::VARCHAR AS region"
# Only allowlisted identifiers enter this SQL. File paths are bound values.
rows = connection.execute(
f'SELECT order_id, "{value_column}" AS cents{region} '
'FROM read_parquet(?) ORDER BY order_id', [str(path)]).fetchall()
seen = set()
for row in rows:
key, cents = row[:2]
if not key or key in seen or cents is None or cents < 0:
raise ValueError("invalid order ID or amount")
seen.add(key)
return rows
def main():
with TemporaryDirectory(prefix="schema-lab-") as temp:
connection = duckdb.connect()
try:
paths = fixtures(connection, Path(temp))
assert read_orders(connection, paths["v2"]) == [("c", 1500), ("d", 500)]
legacy = read_orders(connection, paths["v1"], with_region=True)
assert legacy == [("a", 1200, None), ("b", 800, None)]
print("Old reader + added field: accepted")
print("New reader + old file: region is NULL")
for name in ["renamed", "text_amount", "removed"]:
try:
read_orders(connection, paths[name])
except ValueError:
print(f"Old reader + {name}: rejected")
else:
raise AssertionError("expected a contract rejection")
adapted = read_orders(connection, paths["renamed"], True, "amount_minor")
assert adapted == [("c", 1500, "CH"), ("d", 500, None)]
print("Explicit rename adapter: accepted")
# A successful union can still hide a missing required value.
mixed = connection.execute(
"SELECT order_id, cents, amount_minor FROM read_parquet(?, "
"union_by_name=true) ORDER BY order_id",
[[str(paths["v1"]), str(paths["renamed"])]],
).fetchall()
assert mixed == [("a", 1200, None), ("b", 800, None),
("c", None, 1500), ("d", None, 500)]
subtotal = connection.execute(
"SELECT sum(cents) FROM read_parquet(?, union_by_name=true)",
[[str(paths["v1"]), str(paths["renamed"])]],
).fetchone()[0]
assert subtotal == 2000
print(f"Naive mixed-file SUM(cents): {subtotal}; intended total: 4000")
finally:
connection.close()
if __name__ == "__main__":
main()
Run it with:
python schema_demo.py
The observed output was:
Old reader + added field: accepted
New reader + old file: region is NULL
Old reader + renamed: rejected
Old reader + text_amount: rejected
Old reader + removed: rejected
Explicit rename adapter: accepted
Naive mixed-file SUM(cents): 2000; intended total: 4000
The two file versions contain different order IDs so they can also serve as two batches in the combined-file experiment. Both batches total 2,000 cents. The additional fixtures isolate a reordered schema, a rename, numeric text, and a removed field. They are controlled variations, not a migration of real business records.
The reader enforces a deliberately narrow application policy: order IDs must be VARCHAR and amounts must be BIGINT. It rejects the string "1500" even though a separate conversion might parse it. An INTEGER column would also need an explicit policy change. This prevents an automatic conversion from silently deciding the contract, but it is stricter than every possible valid reader.
The example materializes these tiny results in memory and checks uniqueness within one file. It does not validate uniqueness across an entire dataset or stream large inputs. Adapt those boundaries to your workload before using the reader as a production validation job.
| Change | What the example establishes | Release decision |
|---|---|---|
| Add nullable region | Both reader versions work under their explicit policies | Agree on the meaning of missing region |
| Reorder columns | Named projections preserve values and output order | Check any positional consumers separately |
| Rename cents | Both readers reject the missing original name | Use a version-aware adapter or migrate consumers |
| Remove a required field | The reader rejects the file | Retain the field or introduce a breaking contract |
| Store amounts as text | The exact-type policy rejects them | Specify conversion and rejection rules before changing the type |
| Change a unit without changing the schema | Structural validation cannot detect the intended meaning | Review semantics and reconcile representative results |
The rename adapter maps amount_minor to the reader’s cents output. That mapping is valid here because the fixture retains the same USD-cent meaning. A name mapping would be wrong if the new field held another currency or unit. If a transition temporarily writes both names, define which is authoritative and reject disagreements rather than choosing whichever value happens to be present.
Nullability deserves similar care. A nullable field permits missing values; a default supplies a chosen value under a defined rule. Replacing all missing amounts with zero can turn an incomplete dataset into a plausible but incorrect report. The example rejects NULL in its required amount field.
DuckDB’s union_by_name option combines column names across files and fills absent fields with NULL. In the rename experiment, cents and amount_minor become separate columns. The new batch therefore has NULL in cents, even though its amounts exist under the other name.
The query returns all four rows. However, SUM ignores NULL inputs, so SUM(cents) returns 2,000 instead of the intended 4,000. A row-count check alone would pass. Required-value checks and reconciliation against an independent expected total expose the problem.
Validate each input against its declared version before combining it. Where a migration needs an adapter, apply that adapter deliberately and check its result. Avoid a broad file pattern that silently mixes old, new, partially written, or abandoned outputs. The example supplies an exact file list so the experiment’s inputs are visible.
For an additive change, readers can often be updated first to accept both old and new data, followed by producers that emit the new field. Confirm that sequence with the compatibility matrix rather than assuming it applies to every change.
A rollback is not always just reverting code. Once a producer has written data that an old reader cannot understand, rolling back the reader alone may worsen the incident. Test whether recovery means selecting a previous dataset, applying an adapter, or repairing new outputs. For backfills, record the historical interval and source version, then reconcile the rebuilt data before replacing any existing release.
A schema registry can record versions and check configured compatibility rules for supported message formats. It does not replace business-level checks such as currency, uniqueness, or a report’s expected total. The Confluent source above explains one registry’s rules; this lab does not use a registry.
A table format adds another layer. Apache Iceberg, for example, tracks columns with unique IDs and manages supported schema changes through table metadata. Reading a raw collection of Parquet files by name does not automatically apply that table’s evolution rules. Use a compatible table reader when relying on table metadata. No Iceberg behavior was tested here.
Databases have their own migration and replication constraints. PostgreSQL logical replication does not replicate DDL, so schema coordination remains separate work. The PostgreSQL logical replication guide covers that database-specific case. Choose validation at the layer where your producers and consumers actually meet.
Compute with Hivenet supplies CPU infrastructure for customer-operated validation and transformation jobs. Object Storage with Hivenet provides S3-compatible storage for completed datasets and their supporting files. Schema rules, manifests, reader migrations, and release coordination remain part of your application.
Bring a representative dataset, supported reader versions, expected checks, release frequency, and retention needs to the infrastructure decision. Measure memory and runtime for validation, then test the actual upload and reader handoff. The local example proves the stated file behaviors; it does not establish a cloud deployment or imply that an S3-compatible bucket supplies a schema registry or transactional table layer.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.