← Blog
A cream folding fan with lavender ribs and one orange segment on a pale background.
Published on
2026-11-16

Parquet vs CSV: choose the right format for your data

Choose CSV when the recipient needs a simple text export that their tools can read. Choose Parquet when a compatible analytical tool will repeatedly query typed data, especially selected columns from a larger dataset. Keeping both can make sense when you separate the exchange copy from the analytical copy and define how each is produced.

The useful comparison is the complete workflow: who reads the file, which values must survive, how often it is queried, and how changes are managed. This guide includes a small tested conversion example. It demonstrates preservation of values and types, without claiming that either format is always smaller or faster.

Start with the receiving tool and the workload

Apache Parquet is a binary, column-oriented format with encoding and compression support. CSV represents records as delimited text. A CSV header can name the fields, but it does not specify that an account ID is a string or that a balance has two decimal places.

DecisionCSVParquet
Exchange with a receiving applicationUseful when its import path accepts delimited text.Useful when it supports the writer's types, features, and compression.
Inspecting a small sampleReadable as text, with quoting and encoding rules to respect.Needs a compatible reader to inspect the schema and values.
Data typesSupplied by a separate contract or inferred by the reader.Stored in the file schema, with reader compatibility still to check.
Repeated analytical queriesParsing and conversion remain part of the read path.Column selection and suitable filters can reduce read work.
Small one-off handoffOften sufficient if the receiving tool preserves the required values.May add tooling or conversion work without a useful benefit.

Try a representative sample in the actual destination. A spreadsheet import that turns 00123 into 123 has changed an identifier, even though the original CSV contained the correct characters. A successful file open is a weaker check than a successful data import.

Agree on the data contract before converting

For CSV, define encoding, delimiter, quoting, header names, field types, date interpretation, and the distinction between missing and empty values. The Python CSV documentation describes dialect differences and provides a parser and writer; splitting every line on a comma does not handle quoted commas or embedded newlines correctly.

For Parquet, check how your writer and reader map dates, timestamps, decimals, and nested values. Parquet logical types describe how stored values should be interpreted. Support for the format does not guarantee that every application supports every type or feature.

Conversion cannot recover information that was already lost. If an earlier step parsed an identifier as an integer, exporting that integer to Parquet will not restore its leading zeros. Validate the initial read as well as the converted output.

The example below reserves \N for missing values and keeps an empty string distinct. That marker is a convention for this exchange, not a universal CSV rule. Literal text equal to \N is outside this example's contract; if your data can contain it, agree on a different representation and test it with the receiving application.

Test a CSV to Parquet to CSV round trip

This example uses four synthetic records. It includes leading-zero IDs, a missing balance, a genuine zero, an empty note, a missing note, a comma, an accented character, and an embedded newline. Dates and balances have explicit types.

It was run with CPython 3.12.14 and DuckDB 2.0.0 on Windows on November 16, 2026. Following the DuckDB Python requirements, use Python 3.11 or newer in an activated virtual environment and install the version used here:

python -m pip install duckdb==2.0.0

Save the following as roundtrip.py. The program uses an in-memory database and a temporary directory that is removed when it finishes. It does not read business data, connect to cloud storage, or replace files in your working folder.

import csv
import tempfile
from datetime import date
from decimal import Decimal
from pathlib import Path

import duckdb


def read_csv(db, path):
    return db.sql(r"""
        SELECT * FROM read_csv(?, header=true, auto_detect=false,
            delim=',', quote='"', escape='"', nullstr='\N',
            dateformat='%Y-%m-%d',
            columns={'account_id': 'VARCHAR', 'joined_on': 'DATE',
                     'balance': 'DECIMAL(10,2)', 'note': 'VARCHAR'})
    """, params=[str(path)])


def run_demo():
    rows = [
        ["00123", "2026-11-01", "12.50", ""],
        ["00456", "2026-11-02", r"\N", "priority, EU"],
        ["00789", "2026-11-03", "0.00", r"\N"],
        ["00001", "2026-11-04", "-2.10", "café\nsecond line"],
    ]
    expected = [
        ("00001", date(2026, 11, 4), Decimal("-2.10"), "café\nsecond line"),
        ("00123", date(2026, 11, 1), Decimal("12.50"), ""),
        ("00456", date(2026, 11, 2), None, "priority, EU"),
        ("00789", date(2026, 11, 3), Decimal("0.00"), None),
    ]
    with tempfile.TemporaryDirectory() as folder, duckdb.connect() as db:
        root = Path(folder)
        source, columnar, exported = [root / n for n in
                                      ("input.csv", "data.parquet", "output.csv")]
        with source.open("w", encoding="utf-8", newline="") as stream:
            writer = csv.writer(stream)
            writer.writerow(["account_id", "joined_on", "balance", "note"])
            writer.writerows(rows)
        typed = read_csv(db, source)
        assert typed.order("account_id").fetchall() == expected
        typed.write_parquet(str(columnar), compression="zstd")
        restored = db.read_parquet(str(columnar))
        assert restored.columns == typed.columns
        assert restored.types == typed.types
        assert restored.order("account_id").fetchall() == expected
        restored.write_csv(str(exported), header=True, sep=",", na_rep=r"\N")
        assert read_csv(db, exported).order("account_id").fetchall() == expected
    print("4 rows verified: CSV -> Parquet -> CSV")


if __name__ == "__main__":
    run_demo()

Run it normally, without Python's -O option, which would disable the assertions:

python roundtrip.py

Expected output:

4 rows verified: CSV -> Parquet -> CSV

Understand what the checks establish

The first comparison checks the interpreted CSV values against explicit Python strings, dates, decimals, and missing values. After the Parquet write, the program checks column names, types, and the ordered rows. It then writes a new CSV with the same missing-value marker and reads it with the original contract.

The reader uses explicit DuckDB CSV options, including auto_detect=false and a complete column schema. The Python export uses na_rep for the missing-value marker, as documented in the relational API. SQL and Python option names are not always identical.

The comparison sorts by the unique fixture ID so physical row order is not part of the promise. It checks equivalent interpreted records, not byte-identical CSV files. Quoting, line endings, or decimal formatting can change while the agreed values remain the same.

This tiny fixture does not establish a compression ratio, a speedup, or compatibility with your spreadsheet, warehouse, or another Parquet reader. It supplies known headers; a production import also needs to validate incoming header names and order. Extend the sample with your difficult values and run it through the real receiving tool. If the destination is PostgreSQL, the Python ETL pipeline guide covers validation and loading as a separate workflow.

Measure the query you will actually run

DuckDB can read selected Parquet columns and push suitable filters into the scan. Skipping data depends on the file metadata and layout, as explained in its Parquet reader documentation. A query reading nearly every value has a different opportunity from one that needs a few columns and a narrow date range.

Compare representative files with the same records, query, reader version, hardware, and result checks. CSV can also be compressed, for example with gzip. Record both formats' compression settings, whether conversion time is included, and whether the cache is warm. Measure output size, elapsed time, peak memory, and any remote-transfer costs relevant to the job. A one-time export and a daily query deserve different cost calculations.

File layout also matters. Too many tiny files add overhead; row-group size trades off compression, memory, parallelism, and filtering. The DuckDB Parquet guidance explains these choices. Tune them from the workload instead of treating one file size or row-group setting as correct everywhere.

Keep exchange and analytical copies deliberate

A practical arrangement is a CSV export for a partner and a Parquet copy for recurring internal analysis. Name the authoritative source and make the derived copies reproducible. Record the source snapshot, schema version, transformation, row count, and checks performed.

Decide how a correction or deletion reaches each copy. Adding a field, changing a decimal scale, or redefining a timestamp needs a compatibility decision, even when the files still open. Avoid silently combining exports that use different meanings for the same column.

Neither format alone supplies database transactions, access control, or a complete retention policy. If the workload needs coordinated updates across many files, assess the surrounding database or table-management system rather than expecting the extension to provide those guarantees.

Place the files and query compute with Hivenet

Object storage with Hivenet provides an S3-compatible place for datasets and application files. Your query engine or processing application remains a separate component. Check its connector, credentials, supported operations, and transfer behavior before treating a local example as a remote integration.

Compute with Hivenet provides CPU infrastructure for customer-operated transformation and query jobs. Choose capacity from the actual scan, memory use, schedule, and recovery needs. The four-row example here is a correctness check, not a Hivenet VM benchmark.

Before choosing the operating setup, bring a representative dataset, its schema, the receiving tools, the recurring queries, and the update schedule. Those details make the format and infrastructure decisions concrete.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.