← Blog
A lavender book press with an orange screw handle compressing a stack of ivory sheets.
Published on
2026-11-17

Parquet compression: Snappy, Zstandard, and Gzip

Parquet compression uses codecs such as Snappy, Zstandard, and Gzip to reduce the bytes stored inside a Parquet file. The right choice depends on what can read your files, how your data compresses, and the cost of writing and reading it. A smaller file is useful evidence, but it does not establish which codec will make your workload fastest.

A practical starting point is to compare Snappy and Zstandard against an uncompressed baseline, adding Gzip when it matters to your existing tools or comparison. This guide uses one reproducible dataset and four settings. Every variant is read back and checked before its size is reported.

Choose codecs your readers can use

List the applications that will consume the files, including older batch jobs and export tools. Verify their actual versions with representative files. A successful read in PyArrow proves that particular read path; it does not prove compatibility with every SQL engine or downstream library.

SettingWhy include it in a comparison?What still needs checking?
UncompressedA baseline that keeps the Parquet structure and encoding choicesWhether extra stored and transferred bytes are acceptable
SnappyA codec designed to prioritize speed with moderate compressionActual write and read behavior in your workload
ZstandardAn adjustable compression candidate when file size mattersReader support and the effect of the chosen level
GzipA comparison point for pipelines that already use or require itReader compatibility and processing cost

The Snappy project states its speed-oriented design, while Zstandard exposes a range of compression tradeoffs. Those design goals help choose candidates. They are not performance measurements for your Parquet files.

Parquet applies codecs to data within pages. Choosing Gzip in a Parquet writer does not mean wrapping the entire finished file in a separate .gz archive. The Parquet compression specification describes the codec formats. Compression also provides no access control or encryption by itself.

Keep encoding and file layout constant

Encoding and compression are separate choices. Parquet dictionary encoding can represent repeated values using dictionary entries and identifiers. A codec then compresses the encoded page data. An uncompressed Parquet file can therefore still benefit from dictionary encoding.

Changing row groups, sorting, dictionary settings, and the codec together makes the result difficult to explain. This experiment holds those choices fixed and changes only the codec and its stated level. It uses 100,000 rows in their original order, five row groups of 20,000 rows, statistics enabled, Parquet version 2.6, and data page version 1.0.

The six columns contain sequential integer identifiers, UTC timestamps, four repeated region values, nullable integer amounts, unique SHA-256 hexadecimal strings, and booleans. Two columns contain nulls. Only the region column uses dictionary encoding. The long hexadecimal strings deliberately give the fixture a different compression problem from a table made entirely of repeated category names.

We use Zstandard level 3 and Gzip level 6. Those numbers are not equivalent strength settings across codecs. Snappy has no adjustable compression level in this API. The PyArrow Codec reference describes availability and level support; the script checks that the three compressed codecs are present before using them.

Run the same data through all four settings

In an isolated Python environment, install the version used here with python -m pip install pyarrow==26.0.1. Save the following as compare_codecs.py, then run python compare_codecs.py. Our run used Python 3.12.14 and PyArrow 26.0.1 on Windows.

import hashlib
import json
import platform
import tempfile
from datetime import datetime, timedelta, timezone
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

ROWS = 100_000
ROW_GROUP = 20_000
VARIANTS = [
    ("none", "NONE", None),
    ("snappy", "snappy", None),
    ("zstd", "zstd", 3),
    ("gzip", "gzip", 6),
]


def make_table():
    start = datetime(2026, 1, 1, tzinfo=timezone.utc)
    return pa.table({
        "event_id": pa.array(range(ROWS), type=pa.int64()),
        "event_time": pa.array(
            [start + timedelta(seconds=i) for i in range(ROWS)],
            type=pa.timestamp("us", tz="UTC"),
        ),
        "region": pa.array([
            None if i % 13 == 0 else ("eu", "us", "ap", "af")[i % 4]
            for i in range(ROWS)
        ], type=pa.string()),
        "amount_cents": pa.array([
            None if i % 11 == 0 else (i * 7919) % 100_000
            for i in range(ROWS)
        ], type=pa.int64()),
        "request_id": pa.array([
            hashlib.sha256(f"event-{i}".encode()).hexdigest()
            for i in range(ROWS)
        ], type=pa.string()),
        "accepted": pa.array([i % 5 != 0 for i in range(ROWS)]),
    })


def compare(table, folder):
    results = []
    for label, codec, level in VARIANTS:
        if label != "none" and not pa.Codec.is_available(codec):
            raise RuntimeError(f"Codec unavailable: {codec}")
        path = folder / f"{label}.parquet"
        if path.exists():
            raise FileExistsError(path)
        pq.write_table(
            table, path, compression=codec, compression_level=level,
            row_group_size=ROW_GROUP, use_dictionary=["region"],
            write_statistics=True, version="2.6", data_page_version="1.0",
            data_page_size=1_048_576, dictionary_pagesize_limit=1_048_576,
            use_byte_stream_split=False, write_batch_size=1024,
            store_schema=True,
        )
        restored = pq.read_table(path)
        if not table.schema.equals(restored.schema, check_metadata=True):
            raise AssertionError(f"Schema changed: {label}")
        if not table.equals(restored):
            raise AssertionError(f"Values changed: {label}")
        metadata = pq.read_metadata(path)
        expected = "UNCOMPRESSED" if label == "none" else codec.upper()
        chunks = [metadata.row_group(r).column(c)
                  for r in range(metadata.num_row_groups)
                  for c in range(metadata.num_columns)]
        if any(chunk.compression != expected for chunk in chunks):
            raise AssertionError(f"Unexpected codec metadata: {label}")
        results.append({
            "codec": label, "level": level, "bytes": path.stat().st_size,
            "row_groups": metadata.num_row_groups,
            "schema_equal": True, "values_equal": True,
            "chunk_codec": expected,
        })
    baseline = results[0]["bytes"]
    for result in results:
        result["percent_of_uncompressed"] = round(
            100 * result["bytes"] / baseline, 2
        )
    return results


if __name__ == "__main__":
    folder = Path(tempfile.mkdtemp(prefix="parquet-codecs-", dir="."))
    results = compare(make_table(), folder)
    report = {
        "python": platform.python_version(), "pyarrow": pa.__version__,
        "platform": platform.platform(), "rows": ROWS,
        "folder": str(folder.resolve()), "results": results,
    }
    (folder / "results.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))

Each run creates a new local directory beginning with parquet-codecs-, containing four files and results.json. It performs no bucket operation. The example keeps the dataset in memory and is intended as a small reproducible comparison, not a streaming implementation for a production-sized dataset.

The writer settings are explicit so another reader can repeat the comparison. See the write_table reference when adapting them. PyArrow also supports codec choices by column, but this experiment applies one codec to all columns in each file so the result remains easy to interpret.

Read the size results with the right baseline

These are complete local file sizes, including Parquet metadata, from our fixture. The percentage is compressed file bytes / uncompressed file bytes × 100. It means the share of the baseline size that remains, not the percentage saved.

SettingLevelFile bytesShare of uncompressed size
UncompressedNot set9,208,315100.00%
SnappyNot set7,998,79786.86%
Zstandard34,161,93445.20%
Gzip64,550,53249.42%

Zstandard level 3 produced the smallest file in this run: 4,161,934 bytes, or 45.20% of the uncompressed file's size. Gzip level 6 followed at 4,550,532 bytes. All four files contained the same values and typed schema after reading them back.

This ranking belongs to this data and these settings. A table dominated by already compressed binary payloads, short strings, repeated categories, or a different ordering can behave differently. The result does not establish a universal savings percentage, a preferred level for every dataset, or a speed advantage. We did not measure write time, read time, or CPU consumption.

Whole-file size is also different from the number of bytes a query reads. A query that selects two columns may behave differently from a complete export. Include your actual projections and filters when you later test query behavior.

Verify the contents and the stored codec

Opening a file without an exception is too weak a check for a format experiment. The script compares the complete restored table with the original, including order, and separately checks the typed schema. It also inspects every column chunk's recorded codec. A filename containing zstd is not evidence that Zstandard was used.

Our additional validation passed 37 checks. For each file, it verified all values, the schema, five equal row groups, the expected codec in all 30 column chunks, dictionary encoding limited to the region column, null counts in statistics, the format version, and the reported byte count. Control checks confirmed that reordered data and a changed column type would be detected.

If you adapt the example, preserve the checks before adding more measurements. For a reader-compatibility test, repeat the semantic checks in the other engine rather than accepting a successful file open as sufficient. If a writer silently changes a type or discards timestamp precision, a good compression ratio does not make that output acceptable.

Measure processing cost separately

Once you have valid files, decide what workload matters: daily writes, repeated analytical reads, a full export, or occasional recovery. Test that operation with a dataset large enough to represent your use. Keep the machine, reader version, query, thread settings, and input ordering stable while comparing codecs.

For timing, record warmups and repeated runs, report the spread of results, and state whether caches were warm or cold. Record peak memory and CPU use if they affect capacity. A cached local read is not a measurement of remote object storage, and a short stopwatch result may mostly reflect startup work.

Evaluate stored bytes alongside processing frequency. A file written once and read occasionally presents a different decision from a dataset rewritten every hour and queried continuously. The cloud cost optimization guide provides a broader way to compare spending with completed work. There is no reason to assume that the smallest local file automatically has the lowest total operating cost.

Connect the choice to storage and retention

Compression changes the bytes you store; retention determines how long you keep them. Keep those decisions separate. For AWS-specific transitions and expiration, see the S3 lifecycle policy guide. Do not infer support for every AWS feature from another provider's S3 compatibility.

Hivenet Object Storage offers S3-compatible storage for datasets and other objects. The product page provides a request-access and sales-contact path. Bring a representative file set, its measured sizes, reader requirements, and expected access pattern when assessing fit.

If you also need customer-operated CPU infrastructure for writing or reading those files, evaluate Compute with Hivenet against that workload. This local comparison did not test a Hivenet bucket, a remote query, or a managed analytics service. The next useful test is your complete read and write path with the intended storage endpoint and tools.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.