
DuckDB can query local Parquet files with SQL without first loading them into a persistent database. That makes it useful for a reporting job on a CPU VM: select the columns you need, filter records, calculate a result, and write a smaller file for the next step.
This guide builds two small Parquet files, summarizes paid orders by region, and checks that the exported result matches the query. It also examines a query plan and explains what that plan does not prove. The examples passed 18 checks on Windows with CPython 3.12.14 and DuckDB 2.0.0 on November 12, 2026. These are correctness checks, not a Linux or Hivenet VM performance benchmark.
DuckDB runs inside the Python process in this example. There is no database server to provision, and the analysis reads the Parquet files directly. The temporary database connection holds working state; the input files and exported result remain on disk after it closes.
This fits tasks such as checking a daily export, reducing a set of event files to a report, or preparing an analytical result. If your actual goal is a repeatable pipeline that loads records into PostgreSQL, our Python ETL guide covers that different workflow.
Parquet stores data by column. DuckDB can push selected columns and suitable filters into its Parquet scan, as described in the Parquet reader documentation. Whether that saves substantial work depends on the files, their metadata, and the query. A file format alone does not guarantee a fast job.
Use Python 3.11 or newer in an activated virtual environment, following the DuckDB Python requirements. Pin the version to reproduce this example:
python -m pip install duckdb==2.0.0
Create a working folder and save the next three files there: fixture.sql, summary.sql, and run_demo.py. The example requires only DuckDB and the Python standard library. It generates synthetic data locally; it does not download business records or connect to cloud storage.
Save this as fixture.sql. The ten orders include two unknown amounts, a genuine zero, and a canceled order. Amounts use an exact decimal type rather than binary floating point. Both files have the same schema.
CREATE TABLE orders_source (
order_id INTEGER, region VARCHAR,
amount DECIMAL(10, 2), status VARCHAR
);
INSERT INTO orders_source VALUES
(1, 'eu', 12.50, 'paid'),
(2, 'us', 20.00, 'paid'),
(3, 'eu', NULL, 'paid'),
(4, 'eu', 0.00, 'paid'),
(5, 'eu', 7.25, 'canceled'),
(6, 'eu', 30.00, 'paid'),
(7, 'us', 5.00, 'paid'),
(8, 'eu', 2.50, 'paid'),
(9, 'apac',9.00, 'paid'),
(10,'eu', NULL, 'paid');
COPY (SELECT * FROM orders_source WHERE order_id <= 5)
TO 'orders/part-1.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);
COPY (SELECT * FROM orders_source WHERE order_id > 5)
TO 'orders/part-2.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);
DROP TABLE orders_source;
The temporary source table exists only to generate the fixture. It is dropped before the analysis, so the later query must read the files. In a real job, you would start with existing Parquet data and skip this generation step.
Save this as summary.sql. The glob combines the two input files, the filter removes the canceled order, and the grouping produces one row per region.
SELECT region,
count(*) AS paid_rows,
count(amount) AS known_amounts,
sum(amount) AS known_total
FROM read_parquet('orders/*.parquet')
WHERE status = 'paid'
GROUP BY region
ORDER BY region
count(*) counts all paid rows. count(amount) counts rows whose amount is known. sum(amount) adds the known amounts. This preserves the distinction between a zero and missing data instead of silently turning an unknown amount into zero. See DuckDB's NULL semantics when adapting the query.
The total is therefore a known total, not a claim that the report contains every paid amount. If all amounts in a group are NULL, the sum remains NULL. Decide whether that should fail a production report or appear with an explicit completeness flag.
Save this as run_demo.py. It creates a fresh output directory, runs the fixture and query, writes summary.parquet, reads that file back, and captures a plan for a narrower query.
import argparse
import os
from pathlib import Path
import duckdb
SOURCE = Path(__file__).resolve().parent
def main():
parser = argparse.ArgumentParser()
parser.add_argument("output", nargs="?", default="duckdb-parquet-demo")
args = parser.parse_args()
output = Path(args.output).resolve()
# Refuse an existing directory rather than overwrite its files.
output.mkdir()
os.chdir(output)
Path("orders").mkdir()
summary = (SOURCE / "summary.sql").read_text(encoding="utf-8").strip()
projection = """
SELECT order_id, amount
FROM read_parquet('orders/*.parquet')
WHERE region = 'eu' AND status = 'paid' AND amount >= 10
ORDER BY order_id
"""
with duckdb.connect(":memory:") as con:
con.execute((SOURCE / "fixture.sql").read_text(encoding="utf-8"))
rows = con.execute(summary).fetchall()
print("region | paid_rows | known_amounts | known_total")
for region, count, known, total in rows:
print(f"{region} | {count} | {known} | {total:.2f}")
con.execute(
"COPY (" + summary + ") TO 'summary.parquet' "
"(FORMAT PARQUET, COMPRESSION ZSTD)"
)
restored = con.execute(
"SELECT * FROM read_parquet('summary.parquet') ORDER BY region"
).fetchall()
if restored != rows:
raise RuntimeError("Export does not match the query result")
plan = con.execute("EXPLAIN " + projection).fetchone()[1]
Path("plan.txt").write_text(plan, encoding="utf-8")
print("Round trip: matched")
print("Filtered orders:", con.execute(projection).fetchall())
print("Saved: summary.parquet and plan.txt")
if __name__ == "__main__":
main()
From the folder containing the three files, run:
python run_demo.py
The script creates duckdb-parquet-demo beneath the current working directory. It refuses to reuse an existing output directory, so a second run needs a fresh name as its positional argument. It does not delete or replace an earlier run.
Captured output from the tested environment:
region | paid_rows | known_amounts | known_total
apac | 1 | 1 | 9.00
eu | 6 | 4 | 45.00
us | 2 | 2 | 25.00
Round trip: matched
Filtered orders: [(1, Decimal('12.50')), (6, Decimal('30.00'))]
Saved: summary.parquet and plan.txt
The EU result contains six paid rows but only four known amounts, totaling 45.00. Across all regions there are nine paid rows, seven known amounts, and a known total of 79.00. Those values were checked independently of the SQL result. The two input Parquet files also remained byte-for-byte unchanged after analysis.
The export uses COPY TO with Zstandard compression. Its location is outside orders/, so the next input glob does not accidentally read the report as another source file. Keep raw inputs and derived outputs separate in a recurring job.
fetchall() is convenient for this three-row report. For a large result, export from the query directly rather than materializing all rows as Python objects just to write them out.
The script also selects the IDs and amounts of paid EU orders worth at least 10. Its EXPLAIN output was:
╭─ Order By ────────────────╮
│ Order By: order_id ASC │
│ ~2 rows │
╰─────────────┬─────────────╯
╭─ Projection ┴─────────────╮
│ Projections: order_id, │
│ amount │
│ ~2 rows │
╰─────────────┬─────────────╯
╭─ Read Parquet ────────────╮
│ Function: READ_PARQUET │
│ Projections: amount, │
│ order_id │
│ Filters: region = 'eu', │
│ status = 'paid', │
│ amount >= 10.00 │
│ ~2 rows │
╰───────────────────────────╯
The Parquet scan shows the selected output columns and the region, status, and amount predicates. This is evidence that the planner places those predicates in the scan. It is not a measurement of bytes avoided, row groups skipped, elapsed time, or the benefit of another CPU core.
Plain EXPLAIN displays a plan without running the query; the separate query execution in the script returns orders 1 and 6. The row estimates shown with a tilde are estimates, even when they happen to match this fixture. Use execution profiling and representative data before making performance claims.
A glob is convenient, but it selects whatever matches at execution time. For a reproducible report, record the input file list and control when producers finish writing it. Including the same file twice can count its rows twice; this was checked with a separate small fixture. SQL does not infer that two paths represent an accidental duplicate.
Inspect schemas before combining exports. If a new file adds a column, union_by_name = true can combine columns by name and fill an absent column with NULL. Our separate schema-change check verified that behavior. It does not decide whether a renamed field has the same meaning or whether incompatible types should be accepted. The Parquet tips explain the option and row-group tradeoffs.
The tested reader raises an error when the input glob matches nothing. Keep that failure visible when an empty delivery would make the report misleading. A successful query over the wrong set of files is harder to detect than an explicit missing-input error.
A CPU instance from Compute with Hivenet can host this kind of local analysis without a GPU. Treat the VM as a place to run and measure the workload, not a guarantee that every Parquet dataset fits a particular configuration.
Start with representative file sizes and record elapsed time, peak process memory, temporary-disk use, and the output size. Leave room for the operating system, inputs, output, and temporary work. Keep the original input data recoverable before treating a VM disk as the only copy.
DuckDB's workload-tuning guide covers thread count, row groups, and spilling to disk. More threads do not always improve a query, and support for larger-than-memory work does not guarantee that every combination of joins and aggregates will finish. This example did not test spilling or CPU scaling.
For the first production job, keep the scope as narrow as the example: known inputs, explicit columns and filters, a checked aggregate, and a separate output file. Add scheduling, remote object storage, or a downstream database only when the reporting task requires them.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.