← Blog
A small lavender abacus with cream beads and one orange bead on a pale background.
Published on
2026-11-17

Polars vs pandas: choose a Python data tool

Choose pandas when its existing code, integrations, and data model already serve your workload well. Evaluate Polars when a substantial part of the job consists of dataframe transformations that you can express as filters, joins, and aggregations, especially when execution time or memory has become a constraint.

The useful Polars vs pandas comparison starts with one job and its expected results. A migration can change missing-value handling, joins, types, and ordering. A faster run is little help if the output means something different.

This guide includes an original local example tested on Windows with Python 3.12.14, pandas 3.1.1, and Polars 2.1.0. All 13 tests passed, including 30 deterministic input variants. The example checks behavior and result equivalence. It does not benchmark either library or test a cloud deployment.

Compare the work before comparing the libraries

Write down what the program actually does: the input format and size, columns needed, joins performed, expected output, and libraries that consume it. Then identify where time and memory go. A slow download, a Python loop, and an expensive join call for different changes.

DecisionpandasPolars
Existing applicationOften the smaller change when surrounding code expects pandas objects.Requires checking the expressions and interfaces that would change.
Rows and labelsIndexes and label alignment are part of its data model.Uses explicit columns rather than a pandas-style row index.
ExecutionOrdinary dataframe operations execute eagerly.Offers eager operations and lazy query plans.
Memory pressureConsider selecting fewer columns, suitable types, and chunking compatible work.Consider lazy scans and streaming where the actual operations support it.
Migration checksPreserve the behavior your application relies on.Verify joins, missing values, types, ordering, and downstream consumers.

The Polars migration guide explains these conceptual differences. Treat a rewrite as a change in execution and data semantics, rather than replacing each method with a similarly named method.

Keep types and execution choices explicit

Polars uses a columnar representation based on Arrow. pandas also supports Arrow-backed types and operations, so “pandas uses NumPy, Polars uses Arrow” is an incomplete basis for choosing between them. Record the actual data types and backend used in your comparison.

Polars can parallelize supported operations internally. That does not mean every Python function you place inside a transformation becomes parallel or efficient. Prefer library expressions where they fit, and measure the operations your job performs. Adding processes around an already parallel workload can introduce contention and extra copies.

A lazy query describes work before executing it. Polars can inspect that plan and apply optimizations. In the example below, collect() executes the plan. Because the input is already in memory, this is not a demonstration of skipping columns or rows during a file scan.

Run the same small report in both libraries

Our synthetic report totals positive order amounts by customer segment. Its rules are deliberate: omit rows with a missing customer ID, omit missing or nonpositive amounts, retain unknown customer IDs under unmapped, and reject repeated non-null keys in the customer lookup. This is a selected-positive-amount report, not net revenue accounting.

Amounts are small integer cents, avoiding floating-point comparisons in this fixture. The inputs are trusted Python tuples. A production import still needs its own validation of units, ranges, identifiers, and accepted values. For that broader workflow, see the Python ETL pipeline guide.

In a fresh Python environment, install the tested versions:

python -m pip install pandas==3.1.1 polars==2.1.0

Save this as dataframe_demo.py:

"""Compare a small, trusted, in-memory fixture. This is not a benchmark."""
import pandas as pd
import polars as pl

ORDERS = [(10, 1200), (10, 800), (20, None), (None, 500),
          (30, 200), (20, -100), (20, 700)]
CUSTOMERS = [(10, "team"), (20, "solo")]
EXPECTED = [("solo", 700), ("team", 2000), ("unmapped", 200)]


def pandas_summary(orders, customers):
    sales = pd.DataFrame(orders, columns=["customer_id", "cents"]).astype(
        {"customer_id": "Int64", "cents": "Int64"})
    lookup = pd.DataFrame(customers, columns=["customer_id", "segment"]).astype(
        {"customer_id": "Int64", "segment": "string"})
    accepted = sales.loc[
        sales["customer_id"].notna() & sales["cents"].notna()
        & sales["cents"].gt(0)]
    joined = accepted.merge(
        lookup.loc[lookup["customer_id"].notna()], on="customer_id",
        how="left", validate="many_to_one")
    joined["segment"] = joined["segment"].fillna("unmapped")
    result = joined.groupby("segment", as_index=False)["cents"].sum()
    return sorted((str(s), int(c)) for s, c in result.itertuples(index=False, name=None))


def polars_query(orders, customers):
    sales = pl.DataFrame(orders, schema={"customer_id": pl.Int64,
                                       "cents": pl.Int64}, orient="row")
    lookup = pl.DataFrame(customers, schema={"customer_id": pl.Int64,
                                           "segment": pl.String}, orient="row")
    return (
        sales.lazy()
        .filter(pl.col("customer_id").is_not_null()
                & pl.col("cents").is_not_null() & (pl.col("cents") > 0))
        .join(lookup.lazy().filter(pl.col("customer_id").is_not_null()),
              on="customer_id", how="left", validate="m:1")
        .with_columns(pl.col("segment").fill_null("unmapped"))
        .group_by("segment").agg(pl.col("cents").sum())
    )


def polars_summary(orders, customers):
    result = polars_query(orders, customers).collect()
    return sorted((s, int(c)) for s, c in result.select("segment", "cents").rows())


def null_join_defaults():
    left_pd = pd.DataFrame({"key": pd.Series([None], dtype="Int64")})
    right_pd = pd.DataFrame({"key": pd.Series([None], dtype="Int64"),
                             "label": ["matched"]})
    left_pl = pl.DataFrame({"key": [None]}, schema={"key": pl.Int64})
    right_pl = pl.DataFrame({"key": [None], "label": ["matched"]},
                            schema={"key": pl.Int64, "label": pl.String})
    return (left_pd.merge(right_pd, on="key", how="left")["label"].iloc[0],
            left_pl.join(right_pl, on="key", how="left")["label"][0])


if __name__ == "__main__":
    a = pandas_summary(ORDERS, CUSTOMERS)
    b = polars_summary(ORDERS, CUSTOMERS)
    assert a == b == EXPECTED
    print("pandas:", a)
    print("Polars:", b)
    print("Default null-key joins (pandas, Polars):", null_join_defaults())

Run it:

python dataframe_demo.py

The tested output is:

pandas: [('solo', 700), ('team', 2000), ('unmapped', 200)]
Polars: [('solo', 700), ('team', 2000), ('unmapped', 200)]
Default null-key joins (pandas, Polars): ('matched', None)

Both implementations produce 2,900 cents across the same three groups. The result is sorted before comparison because the order of grouped rows is not part of this report's meaning. The test suite also checks empty inputs, missing labels, unmatched IDs, rejected duplicate lookup keys, and unchanged source tuples.

Check the join that can change your results

The last output line comes from a separate one-row example with a missing key on each side. pandas merge matches null keys. A Polars join does not match null keys by default; its nulls_equal option changes that behavior.

Neither default decides your business rule. In our report, a missing customer ID is excluded before the join in both implementations. A non-null ID absent from the lookup is retained as unmapped. Those cases must remain distinct when you migrate the job.

Lookup cardinality matters too. A repeated customer key can multiply rows and inflate a total. The example asks both libraries to enforce a many-to-one join, and the tests confirm that a repeated non-null lookup key raises an error. Compare values and cardinality before treating an apparently successful run as correct.

Use lazy execution and streaming with a specific question

Polars provides a way to inspect a query plan. That helps you understand the operations being executed and investigate whether a proposed rewrite expresses the intended work. We inspected the example's plan, but did not measure an optimization speedup.

Polars streaming can execute suitable work in batches. Some operations may fall back to in-memory execution. A streaming setting alone therefore does not establish that a job will stay within a chosen memory limit.

This example uses the ordinary collect() call and an in-memory source. It does not request the streaming engine. Test your complete query, including joins and validation requirements, before choosing a smaller machine based on a streaming assumption.

Measure a representative job before claiming a winner

First improve obvious inefficiencies in the current implementation. The pandas scaling guide covers loading less data, using efficient types, and splitting suitable work into chunks. Chunking needs care when a join or aggregation depends on rows across chunks.

For a fair performance comparison, use the same input, selected columns, types, output requirements, and correctness checks. Record library versions, hardware, thread settings, and whether the measurement includes reading and writing files. Time the completed result, including Polars collection, rather than just constructing a lazy plan.

Run several repetitions and distinguish cold reads from warm cached runs. Measure peak process memory with the same method for both implementations. A dataframe's reported size does not include every temporary buffer or the rest of the process. Keep file generation outside the timed section unless generating files is part of the job you intend to compare.

These measurements can support a decision about your workload. Our tiny correctness fixture cannot establish a general speed ranking, a memory-saving percentage, or a recommended VM size.

Decide whether the migration earns its cost

Keep pandas when the current job meets its deadline and memory budget, and changing it would disrupt useful integrations without a measured benefit. Try Polars on a bounded transformation when you can state the expected output and test its edge cases. That makes it easier to assess both the runtime benefit and the work required to maintain the new code.

A migration test should include the consumer of the result. A downstream library may expect a pandas index, particular nullable types, or a specific sort order. Conversion between formats or dataframe implementations adds another step whose behavior and cost belong in the comparison.

Turn the result into a CPU workload decision

Once the job is correct, use its measured duration, peak memory, input volume, and run frequency to evaluate CPU compute with Hivenet. Bring the measured job to the capacity decision instead of choosing an instance from a library name.

Input datasets and completed outputs can be stored separately with Hivenet Object Storage. Include transfer time and credentials management in that design. This example does not test a remote storage connection, and object storage does not remove the memory needed while a local dataframe job runs.

Start with one representative transformation. If its results hold and its operating requirements are understood, you have a concrete basis for keeping pandas, adopting Polars, or changing the compute capacity around the existing job.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.