
Iceberg vs Parquet usually comes down to a practical question: are files enough for your dataset, or do readers need a managed table state across those files? Apache Parquet is a columnar file format. Apache Iceberg is a table format that can use Parquet for its data files. A system can use both.
The difference becomes useful when a dataset changes. Reading a file tells you what is inside that file. Reading an Iceberg table also involves metadata that identifies the table state you requested. This guide demonstrates the distinction with a local catalog, two committed snapshots, and one deliberately unregistered Parquet file.
| Layer | What it contributes | What to choose |
|---|---|---|
| Parquet data file | Columnar data and metadata for interpreting that file. | A supported file format and writer configuration. |
| Iceberg table | Table metadata, schemas, snapshots, and references to the files that belong to a table state. | A table format version supported by your readers and writers. |
| Catalog | A way to find named tables and coordinate their metadata updates. | A catalog implementation and an operational owner. |
| Storage and processing | Somewhere to keep the files and software to read or write them. | Storage, a processing engine, and enough compute for the workload. |
Parquet has its own metadata, including the information a reader needs to locate column data. Calling it a file format does not mean it is an unstructured blob. The Parquet file-format documentation describes that layout; the project overview explains its column-oriented design.
Iceberg adds a table-level description. Its specification defines snapshots and the metadata used to track their contents. In the format-version-2 example below, snapshot metadata leads through manifest lists and manifests to data files. Those files are Parquet. The catalog locates the table's current metadata; it is a separate component from the storage holding the files.
Start with the simplest arrangement that meets your requirements. A fixed export, a small analytical dataset, or a single pipeline that publishes complete immutable batches may work well as Parquet files. You still need a clear rule for selecting the right files and knowing when a batch is ready.
Consider a table layer when consumers need an agreed current state, explicit historical versions, or coordinated changes across a growing dataset. Ask who publishes each change, how readers select a version, and what happens when a write stops halfway through. If your answer is a growing collection of custom file lists and coordination scripts, evaluate what a table format and catalog could replace.
Schema and partition changes are also relevant. Iceberg documents these in its evolution guide. Check the exact operations supported by your engine, library, catalog, and table format version. This example tests append and read behavior, not schema migration or distributed writes.
The example uses PyIceberg, PyArrow, and a SQLite-backed SQL catalog. All data stays in a new local directory. The PyIceberg getting-started guide presents this kind of local setup for demonstration and warns against treating it as a production arrangement. It does not require a cloud account or a separate catalog server.
Create a working directory and install the versions used for this article. These commands use Windows PowerShell:
python -m venv .venv
.venv/Scripts/python.exe -m pip install "pyiceberg[sql-sqlite]==0.12.0" pyarrow==26.0.1 sqlalchemy==2.1.9
Save the following as iceberg_example.py. It creates a uniquely named directory on each run and refuses to reuse an existing one. The first batch contains IDs 1 and 2 with units of 3 and 5; the second adds ID 3 with two units.
import json
from pathlib import Path
from uuid import uuid4
import pyarrow as pa
import pyarrow.parquet as pq
from pyarrow.fs import FileSystem
from pyiceberg.catalog import load_catalog
from sqlalchemy.engine import URL
def open_catalog(root):
return load_catalog(
'local_demo', type='sql',
uri=URL.create('sqlite', database=str(root / 'catalog.db')).render_as_string(),
warehouse=(root / 'warehouse').as_posix(),
**{'py-io-impl': 'pyiceberg.io.pyarrow.PyArrowFileIO'},
)
def summarize(data):
rows = sorted(data.to_pylist(), key=lambda row: row['id'])
return {'ids': [row['id'] for row in rows],
'rows': len(rows), 'units': sum(row['units'] for row in rows)}
def members(table, snapshot_id):
return sorted(task.file.file_path
for task in table.scan(snapshot_id=snapshot_id).plan_files())
def run_example(root):
root = Path(root).resolve()
root.mkdir() # Refuse to overwrite an existing directory.
(root / 'warehouse').mkdir()
catalog = open_catalog(root)
catalog.create_namespace('demo')
schema = pa.schema([pa.field('id', pa.int64()), pa.field('units', pa.int64())])
table = catalog.create_table('demo.orders', schema=schema,
properties={'format-version': '2'})
table.append(pa.table({'id': [1, 2], 'units': [3, 5]}, schema=schema))
first = table.current_snapshot().snapshot_id
first_files = members(table, first)
table.append(pa.table({'id': [3], 'units': [2]}, schema=schema))
second = table.current_snapshot().snapshot_id
second_files = members(table, second)
current = summarize(table.scan().to_arrow())
earlier = summarize(table.scan(snapshot_id=first).to_arrow())
reopened = open_catalog(root).load_table('demo.orders')
# Deliberately place an unregistered file beside committed data files.
# This demonstrates why listing Parquet files is not an Iceberg scan.
_, local_path = FileSystem.from_uri(second_files[0])
data_dir = Path(local_path).parent
loose = data_dir / 'unregistered.parquet'
pq.write_table(pa.table({'id': [999], 'units': [99]}, schema=schema), loose)
raw_files = sorted(data_dir.glob('*.parquet'))
raw = summarize(pa.concat_tables([pq.ParquetFile(p).read() for p in raw_files]))
after_loose = summarize(reopened.scan().to_arrow())
report = {
'root': str(root), 'table_format_version': table.metadata.format_version,
'first_snapshot': first, 'second_snapshot': second,
'second_parent': table.current_snapshot().parent_snapshot_id,
'snapshot_count': len(table.snapshots()),
'first_files': first_files, 'second_files': second_files,
'current': current, 'earlier': earlier,
'reopened_snapshot': reopened.current_snapshot().snapshot_id,
'reopened': summarize(reopened.scan().to_arrow()),
'unregistered_file': loose.as_posix(), 'raw_directory': raw,
'after_unregistered_file': after_loose,
'committed_files_after': members(reopened, second),
'raw_parquet_files': len(raw_files),
'data_formats': sorted({str(t.file.file_format)
for t in table.scan().plan_files()}),
}
(root / 'report.json').write_text(json.dumps(report, indent=2), encoding='utf-8')
catalog.engine.dispose()
reopened.catalog.engine.dispose()
return report
if __name__ == '__main__':
result = run_example(Path.cwd() / ('iceberg-demo-' + uuid4().hex[:8]))
print(json.dumps(result, indent=2))
Run it from that working directory:
.venv/Scripts/python.exe iceberg_example.py
The script prints a report and saves it inside the new iceberg-demo-... directory. Keep that directory while inspecting the result: it contains the catalog database, table metadata, and data files. The script does not delete its fixture or touch an existing dataset.
The first append creates a snapshot containing two rows. The second creates a new snapshot with three rows. table.scan() reads the current state, while table.scan(snapshot_id=first) requests the earlier state. Snapshot IDs are generated values, so they will differ between runs.
In the tested fixture, the second snapshot reused the first Parquet data file and added another. It did not need a second complete copy of the original rows. Opening the catalog again preserved the current snapshot, and a separate Python process could read both versions. The PyIceberg API reference documents table loading, writes, inspection, and scans.
| Read performed | Row IDs | Rows | Total units |
|---|---|---|---|
| First snapshot after both appends | 1, 2 | 2 | 8 |
| Current snapshot | 1, 2, 3 | 3 | 10 |
| Raw directory scan after adding the loose file | 1, 2, 3, 999 | 4 | 109 |
| Iceberg scan after adding the loose file | 1, 2, 3 | 3 | 10 |
The last two rows are the important distinction. The script deliberately writes unregistered.parquet alongside the committed data files. A raw scan that reads every Parquet file there includes ID 999. The Iceberg scan excludes it because the file was never committed to that table. A filename and a directory location do not establish table membership.
This is an artificial, controlled demonstration, not a recommendation to write loose files into a real table directory. It shows why an Iceberg reader must follow table metadata instead of substituting a directory glob. The raw-file comparison uses PyArrow's ParquetFile reader; it is intentionally unaware of the table.
Validation, November 17, 2026: 18 checks passed on Windows with Python 3.12.14, covering snapshot lineage, exact row values, file membership, catalog reopening, and reads from a fresh process. The fixture uses Iceberg table format version 2. No remote object store, concurrent writers, failure recovery, snapshot expiration, or performance benchmark was tested.
Table metadata gives you a way to describe versions; operating that system still needs ownership. Decide which service writes tables, who can change catalog entries, how readers authenticate, and how the catalog and stored files will be backed up and recovered together. A successful local read does not answer those operational questions.
Historical reads also have retention limits. Expiring a snapshot removes it from the history available for time travel. Cleanup must preserve files still needed by retained snapshots, and orphan-file removal must allow for writes that are still in progress. Iceberg's maintenance documentation covers snapshot expiration, orphan files, and compaction. Use the supported procedures for your deployment rather than deleting files merely because their names look old.
Keep storage lifecycle rules consistent with that policy. A bucket rule that removes a referenced object can break reads even when the catalog is healthy. Test retention and recovery with representative data before applying them broadly. This example leaves its small fixture intact and does not demonstrate cleanup.
There is also no universal speed winner in this comparison. Both paths may read Parquet data. Query planning, file sizes, filters, partition layout, network access, and engine support affect the work performed. Measure the queries and update patterns you actually need, including the cost of maintaining metadata and compacting small files.
A bucket gives files a home. It does not, by itself, provide an Iceberg catalog or the software that executes queries. Before moving the example to object storage, verify the engine and catalog combination, endpoint configuration, credentials, required object operations, and supported table features. The PyIceberg configuration reference separates catalog and file-I/O settings.
Hivenet Object Storage is an option to evaluate for dataset files and metadata through its S3-compatible interface. Bring the exact engine, catalog, library versions, and read/write operations you plan to use. Validate that combination before moving a production table. This local example does not establish Hivenet interoperability or a managed Iceberg offering.
For processing that your team operates, Compute with Hivenet provides a separate CPU infrastructure option. Size it from measured memory use, query duration, and simultaneous work. Start the decision with the table behavior your application needs, then choose and test the components that will deliver it.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.