← Blog
A lavender expanding file folder with cream dividers and an orange tab on a pale background.
Published on
2026-11-17

Schema evolution: test dataset compatibility

Schema evolution is the controlled change of a dataset’s structure as producers and consumers develop. Adding a field, renaming a column, or changing a type can affect a reader that still expects yesterday’s data. The aim is to preserve the contracts you intend to support and make breaking changes explicit.

A file that opens successfully can still produce a wrong answer. In the example below, a reader combines two Parquet files after an amount column is renamed. The query succeeds, but its total misses half the intended amount. Explicit consumer checks catch the change before it reaches that calculation.

This guide focuses on file-based datasets. It uses DuckDB to test old and new readers against small Parquet fixtures, then explains how those checks inform a rollout. The same questions arise in event streams and databases, although their compatibility mechanisms differ.

Define the contract beyond column names

A useful data contract describes names, types, required values, identifiers, units, and the meaning of missing data. It also names the producer and the consumers that must keep working. A schema version identifies a documented contract; a version label alone does not make a change compatible.

Our example has an order_id string and a cents integer. The contract says that amounts are nonnegative USD cents and IDs are nonempty and unique within each input file. The second version adds a nullable region string. For an old file without that field, the new reader returns NULL, meaning unavailable. It does not invent a region or equate missing data with an empty string.

Semantic compatibility requires a separate review. If the value 100 changes from meaning 100 cents to 100 dollars, its name and integer type can remain unchanged while the report becomes wrong. Neither a type check nor a successful cast can discover the producer’s intended unit.

Check compatibility in both directions

Using the directions described in Confluent’s compatibility documentation, backward compatibility means a new reader can consume old data; forward compatibility means an old reader can consume new data. Full compatibility requires both. The detailed rules depend on the format and the reader’s policy.

For this example, the old reader projects only the two original fields. The new reader explicitly handles an absent region. Those choices produce this matrix:

ReaderFileObserved behavior
Old readerOriginal schemaReads order ID and cents
Old readerAdded nullable regionIgnores region and reads its required fields
New readerOriginal schemaSupplies NULL for the absent region
New readerAdded nullable regionReads region, including an explicit NULL

This does not mean every added optional field is universally compatible. A consumer that rejects unknown fields, uses positional output, or demands a populated region has different requirements. Test the readers you actually operate. If you retain many historical versions, checking only the immediately preceding version also leaves older data untested.

Run the Parquet compatibility example

Checked November 17, 2026: the demonstration and 18 local tests passed on Windows with Python 3.12.14 and DuckDB 2.0.0. They cover explicit projections, both compatibility directions, field order, renaming, removal, types, missing values, duplicate IDs, combined-file behavior, and preservation of input bytes. No remote storage, schema registry, table-format engine, deployment, or performance test was run.

In a fresh Python 3.12 virtual environment, install the pinned package:

python -m pip install duckdb==2.0.0

Save the following as schema_demo.py. It creates trusted synthetic files in a temporary directory and removes them on exit. The DuckDB Parquet interface writes the fixtures, inspects their types, and reads named columns. File paths are bound parameters; the only field mapping is an explicit allowlist.

"""Local Parquet compatibility lab using trusted, synthetic fixtures."""
from pathlib import Path
from tempfile import TemporaryDirectory
import duckdb


def write_fixture(connection, path, query):
    if path.exists():
        raise FileExistsError(path)
    connection.sql(query).write_parquet(str(path))


def fixtures(connection, root):
    queries = {
        "v1": """SELECT * FROM (VALUES
            ('a', 1200::BIGINT), ('b', 800::BIGINT)) t(order_id, cents)""",
        "v2": """SELECT * FROM (VALUES
            ('c', 1500::BIGINT, 'CH'), ('d', 500::BIGINT, NULL))
            t(order_id, cents, region)""",
        "reordered": """SELECT region, cents, order_id FROM (VALUES
            ('c', 1500::BIGINT, 'CH'), ('d', 500::BIGINT, NULL))
            t(order_id, cents, region)""",
        "renamed": """SELECT * FROM (VALUES
            ('c', 1500::BIGINT, 'CH'), ('d', 500::BIGINT, NULL))
            t(order_id, amount_minor, region)""",
        "text_amount": "SELECT 'c' AS order_id, '1500' AS cents",
        "removed": "SELECT 'c' AS order_id",
    }
    paths = {name: root / (name + ".parquet") for name in queries}
    for name, query in queries.items():
        write_fixture(connection, paths[name], query)
    return paths


def read_orders(connection, path, with_region=False, value_column="cents"):
    """Example consumer policy, not a general schema compatibility checker."""
    if value_column not in {"cents", "amount_minor"}:
        raise ValueError("unsupported field mapping")
    description = connection.execute(
        "DESCRIBE SELECT * FROM read_parquet(?)", [str(path)]).fetchall()
    types = {row[0]: row[1] for row in description}
    required = {"order_id": "VARCHAR", value_column: "BIGINT"}
    if any(types.get(name) != kind for name, kind in required.items()):
        raise ValueError("required field missing or type differs")
    region = ""
    if with_region:
        if "region" in types and types["region"] != "VARCHAR":
            raise ValueError("region must be VARCHAR when present")
        region = ", region" if "region" in types else ", NULL::VARCHAR AS region"
    # Only allowlisted identifiers enter this SQL. File paths are bound values.
    rows = connection.execute(
        f'SELECT order_id, "{value_column}" AS cents{region} '
        'FROM read_parquet(?) ORDER BY order_id', [str(path)]).fetchall()
    seen = set()
    for row in rows:
        key, cents = row[:2]
        if not key or key in seen or cents is None or cents < 0:
            raise ValueError("invalid order ID or amount")
        seen.add(key)
    return rows


def main():
    with TemporaryDirectory(prefix="schema-lab-") as temp:
        connection = duckdb.connect()
        try:
            paths = fixtures(connection, Path(temp))
            assert read_orders(connection, paths["v2"]) == [("c", 1500), ("d", 500)]
            legacy = read_orders(connection, paths["v1"], with_region=True)
            assert legacy == [("a", 1200, None), ("b", 800, None)]
            print("Old reader + added field: accepted")
            print("New reader + old file: region is NULL")
            for name in ["renamed", "text_amount", "removed"]:
                try:
                    read_orders(connection, paths[name])
                except ValueError:
                    print(f"Old reader + {name}: rejected")
                else:
                    raise AssertionError("expected a contract rejection")
            adapted = read_orders(connection, paths["renamed"], True, "amount_minor")
            assert adapted == [("c", 1500, "CH"), ("d", 500, None)]
            print("Explicit rename adapter: accepted")
            # A successful union can still hide a missing required value.
            mixed = connection.execute(
                "SELECT order_id, cents, amount_minor FROM read_parquet(?, "
                "union_by_name=true) ORDER BY order_id",
                [[str(paths["v1"]), str(paths["renamed"])]],
            ).fetchall()
            assert mixed == [("a", 1200, None), ("b", 800, None),
                             ("c", None, 1500), ("d", None, 500)]
            subtotal = connection.execute(
                "SELECT sum(cents) FROM read_parquet(?, union_by_name=true)",
                [[str(paths["v1"]), str(paths["renamed"])]],
            ).fetchone()[0]
            assert subtotal == 2000
            print(f"Naive mixed-file SUM(cents): {subtotal}; intended total: 4000")
        finally:
            connection.close()


if __name__ == "__main__":
    main()

Run it with:

python schema_demo.py

The observed output was:

Old reader + added field: accepted
New reader + old file: region is NULL
Old reader + renamed: rejected
Old reader + text_amount: rejected
Old reader + removed: rejected
Explicit rename adapter: accepted
Naive mixed-file SUM(cents): 2000; intended total: 4000

The two file versions contain different order IDs so they can also serve as two batches in the combined-file experiment. Both batches total 2,000 cents. The additional fixtures isolate a reordered schema, a rename, numeric text, and a removed field. They are controlled variations, not a migration of real business records.

The reader enforces a deliberately narrow application policy: order IDs must be VARCHAR and amounts must be BIGINT. It rejects the string "1500" even though a separate conversion might parse it. An INTEGER column would also need an explicit policy change. This prevents an automatic conversion from silently deciding the contract, but it is stricter than every possible valid reader.

The example materializes these tiny results in memory and checks uniqueness within one file. It does not validate uniqueness across an entire dataset or stream large inputs. Adapt those boundaries to your workload before using the reader as a production validation job.

Treat each kind of change deliberately

ChangeWhat the example establishesRelease decision
Add nullable regionBoth reader versions work under their explicit policiesAgree on the meaning of missing region
Reorder columnsNamed projections preserve values and output orderCheck any positional consumers separately
Rename centsBoth readers reject the missing original nameUse a version-aware adapter or migrate consumers
Remove a required fieldThe reader rejects the fileRetain the field or introduce a breaking contract
Store amounts as textThe exact-type policy rejects themSpecify conversion and rejection rules before changing the type
Change a unit without changing the schemaStructural validation cannot detect the intended meaningReview semantics and reconcile representative results

The rename adapter maps amount_minor to the reader’s cents output. That mapping is valid here because the fixture retains the same USD-cent meaning. A name mapping would be wrong if the new field held another currency or unit. If a transition temporarily writes both names, define which is authoritative and reject disagreements rather than choosing whichever value happens to be present.

Nullability deserves similar care. A nullable field permits missing values; a default supplies a chosen value under a defined rule. Replacing all missing amounts with zero can turn an incomplete dataset into a plausible but incorrect report. The example rejects NULL in its required amount field.

Do not mistake a combined schema for a valid dataset

DuckDB’s union_by_name option combines column names across files and fills absent fields with NULL. In the rename experiment, cents and amount_minor become separate columns. The new batch therefore has NULL in cents, even though its amounts exist under the other name.

The query returns all four rows. However, SUM ignores NULL inputs, so SUM(cents) returns 2,000 instead of the intended 4,000. A row-count check alone would pass. Required-value checks and reconciliation against an independent expected total expose the problem.

Validate each input against its declared version before combining it. Where a migration needs an adapter, apply that adapter deliberately and check its result. Avoid a broad file pattern that silently mixes old, new, partially written, or abandoned outputs. The example supplies an exact file list so the experiment’s inputs are visible.

Roll out changed data with a recovery path

For an additive change, readers can often be updated first to accept both old and new data, followed by producers that emit the new field. Confirm that sequence with the compatibility matrix rather than assuming it applies to every change.

  1. Record the contract and affected readers. Include field meaning, required values, units, supported historical versions, and an owner for the change.
  2. Test representative old and new files. Include missing values, boundary values, historical variants, and the actual queries or transformations used downstream.
  3. Write a separate candidate output. Use an explicit dataset version or release path. Record the input identities, file list, schema version, row counts, and useful reconciliation totals.
  4. Switch consumers through a controlled handoff. Make the completed release identifiable and verify that readers use the intended files. Decide how competing writers and failed uploads are handled.
  5. Retain a tested recovery route. Keep the previous files and reader version for the agreed period, and monitor correctness after the switch.

A rollback is not always just reverting code. Once a producer has written data that an old reader cannot understand, rolling back the reader alone may worsen the incident. Test whether recovery means selecting a previous dataset, applying an adapter, or repairing new outputs. For backfills, record the historical interval and source version, then reconcile the rebuilt data before replacing any existing release.

Choose tools at the right layer

A schema registry can record versions and check configured compatibility rules for supported message formats. It does not replace business-level checks such as currency, uniqueness, or a report’s expected total. The Confluent source above explains one registry’s rules; this lab does not use a registry.

A table format adds another layer. Apache Iceberg, for example, tracks columns with unique IDs and manages supported schema changes through table metadata. Reading a raw collection of Parquet files by name does not automatically apply that table’s evolution rules. Use a compatible table reader when relying on table metadata. No Iceberg behavior was tested here.

Databases have their own migration and replication constraints. PostgreSQL logical replication does not replicate DDL, so schema coordination remains separate work. The PostgreSQL logical replication guide covers that database-specific case. Choose validation at the layer where your producers and consumers actually meet.

Validate on CPU and store completed dataset versions

Compute with Hivenet supplies CPU infrastructure for customer-operated validation and transformation jobs. Object Storage with Hivenet provides S3-compatible storage for completed datasets and their supporting files. Schema rules, manifests, reader migrations, and release coordination remain part of your application.

Bring a representative dataset, supported reader versions, expected checks, release frequency, and retention needs to the infrastructure decision. Measure memory and runtime for validation, then test the actual upload and reader handoff. The local example proves the stated file behaviors; it does not establish a cloud deployment or imply that an S3-compatible bucket supplies a schema registry or transactional table layer.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.