Python multiprocessing: CPU batch jobs with a process pool

Python multiprocessing runs work in separate processes so a CPU-heavy Python job can use more than one processor core. It is useful when a batch can be divided into independent pieces, each with enough computation to justify starting workers and passing data between them.

Start with a correct serial implementation. Then compare its results with a small process pool before changing worker counts or choosing a larger machine. This guide follows that sequence with a reproducible numerical batch, then shows how to report individual task failures with ProcessPoolExecutor.

When a process pool is a good fit

Look for work that spends time calculating rather than waiting: independent numerical calculations, Python-heavy transformations, or validation across separate batches. If each task needs another task's changing state, first reconsider how to divide the work.

On conventional CPython builds, the global interpreter lock limits simultaneous execution of Python bytecode by threads. Processes are a useful option for CPU-heavy Python code. Threads can still suit waiting on network or disk operations, and native libraries may release the lock. Python also has optional free-threaded builds, so check the interpreter and libraries you actually run. The Python threading documentation explains this distinction.

A pool on one machine does not provide a durable job queue across servers. If work must survive application restarts, support retries, or run on multiple machines, review the separate design decisions in our Celery worker concurrency guide.

A complete Python multiprocessing example

Save the following as batch.py. It counts primes in four half-open ranges: the lower bound is included and the upper bound is excluded. The deliberately simple trial-division algorithm provides independent CPU work with small, checkable results. It is a teaching fixture, not a recommendation for high-performance prime generation.

These examples were executed as files on Windows with CPython 3.12.14. They use only the standard library and explicitly select spawn. Linux and Hivenet VM execution were not part of the validation.

"""Small CPU batch example. Run as a file, not in a notebook cell."""
import multiprocessing as mp
from math import isqrt


def count_primes(bounds):
    start, stop = bounds
    if start > stop:
        raise ValueError("start must not exceed stop")
    total = 0
    for n in range(max(2, start), stop):
        if all(n % d for d in range(2, isqrt(n) + 1)):
            total += 1
    return total


def parallel_counts(jobs, workers=2, chunksize=1):
    if workers < 1 or chunksize < 1:
        raise ValueError("workers and chunksize must be positive")
    ctx = mp.get_context("spawn")
    with ctx.Pool(processes=workers) as pool:
        return pool.map(count_primes, jobs, chunksize=chunksize)


def main():
    jobs = [(0, 100), (100, 200), (200, 300), (300, 400)]
    serial = [count_primes(job) for job in jobs]
    parallel = parallel_counts(jobs, workers=2)
    assert parallel == serial
    for job, count in zip(jobs, parallel):
        print(f"{job}: {count}")
    print(f"total: {sum(parallel)}")
    print(f"matches serial: {parallel == serial}")


if __name__ == "__main__":
    main()

From the directory containing the file, run:

python batch.py

The captured output is:

(0, 100): 25
(100, 200): 21
(200, 300): 16
(300, 400): 16
total: 78
matches serial: True

The serial and parallel runs both return 78 primes below 400. The validation also compared the worker function with an independently implemented sieve, rather than relying only on two calls to the same function. Empty batches, repeated jobs, reversed input order, several chunk sizes, and worker exceptions were checked.

The parent builds the job list and gathers results. Each worker receives two integers and returns one integer; no task edits another task's data. In a file-processing workload, the same design can pass a file path and return a small summary. Open the file inside the worker rather than passing an open file handle. For the surrounding extraction and loading steps, see our Python ETL pipeline guide.

Why the main guard and spawn matter

With spawn, a child starts a fresh interpreter and imports the main module. The if __name__ == "__main__": guard keeps that import from launching another pool. Place worker functions at module level and run these examples as scripts, rather than pasting them into an interactive session.

Start-method defaults depend on the operating system and Python version. Python 3.14 changed the POSIX default from fork to forkserver where supported. Explicit get_context("spawn") makes the choice clear for this example without changing the application's global default. See the Python 3.14 start-method documentation before adapting code that assumes Linux always uses fork.

Arguments and results must cross process boundaries. Python's pickle documentation explains which objects can be serialized and why functions must be importable by name. Use ordinary data and module-level functions here; lambdas, nested functions, open files, and active database connections are poor things to submit as jobs. Do not load serialized data from an untrusted source.

Map, chunksize, and result ordering

In this example, Pool.map blocks until it has the results and returns them in input order. That is why the final zip pairs each range with the right count even if workers finish in a different sequence. The multiprocessing reference describes the pool APIs, including alternatives for incremental results.

chunksize=1 sends one range per scheduled chunk. Increasing it bundles several jobs together. Larger chunks can reduce dispatch overhead, but a worker assigned an expensive chunk may finish well after the others. Choose using representative task sizes; four tiny ranges cannot establish a useful production setting.

The with block ensures the pool is released. Here the blocking map has completed before the block ends. A Pool context exits by terminating its workers, so do not submit asynchronous work and leave the block while expecting it to finish. For explicit graceful Pool shutdown, stop new submissions with close(), then wait with join().

Use ProcessPoolExecutor to inspect each task

If you need a result or an error for each submitted job, a future gives you a convenient handle. Save this second file as executor_demo.py next to batch.py. One range is deliberately invalid.

"""Report each task result, including one deliberate invalid range."""
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor
from batch import count_primes


def main():
    jobs = [(0, 100), (10, 1), (100, 200)]
    ctx = mp.get_context("spawn")
    with ProcessPoolExecutor(max_workers=2, mp_context=ctx) as executor:
        futures = [executor.submit(count_primes, job) for job in jobs]
        for job, future in zip(jobs, futures):
            try:
                count = future.result()
            except ValueError as exc:
                print(f"{job}: ERROR {exc}")
            else:
                print(f"{job}: {count}")


if __name__ == "__main__":
    main()

Run the second script:

python executor_demo.py

The captured output keeps the failed job visible while reporting the valid jobs:

(0, 100): 25
(10, 1): ERROR start must not exceed stop
(100, 200): 21

future.result() raises the task's exception in the parent. This example catches the expected ValueError only; it does not silently convert every infrastructure error into a successful run. It reports in submission order, which can make reporting wait behind a slow first task. Use as_completed with an explicit future-to-job mapping when completion-order reporting is preferable.

The executor context waits for submitted work during shutdown. A timeout while waiting for a future does not stop its running computation, and canceling a future cannot cancel work already running. Design bounded tasks and an explicit retry or termination policy if deadlines matter. The concurrent.futures reference documents those behaviors. Avoid calling executor or future methods from a function submitted to the same process pool.

Choose worker counts from measurements

Two workers are enough to demonstrate the mechanics. They are not a sizing recommendation. For your workload, keep the input fixed and compare a serial run with a few explicit worker counts. Record total elapsed time including startup, peak memory, output correctness, and failures. Repeat runs; do not promote the best single result into a capacity estimate.

Watch for three common limits. Very small jobs can cost more to dispatch than to compute. Large arguments or return values increase serialization and memory costs. Numerical libraries that create their own threads can multiply the total runnable work when placed inside several processes. Check library thread settings before increasing the pool again.

For a large job source, avoid copying this example's list-of-all-futures pattern unchanged. Bound outstanding work and consume results as you go. Python 3.14 added buffersize to executor map for limiting submitted work whose results have not been yielded; it is not available in the tested Python 3.12 runtime. See the Python 3.14 executor documentation.

Run the workload on a CPU VM

CPU instances with Compute with Hivenet provide an isolated VM with SSH access for workloads such as data processing and background jobs. A process pool runs inside that VM; it does not automatically expand across instances.

Hivenet counts each vCPU as a hardware thread, so do not treat the vCPU count as a count of independent physical cores. Choose a small instance that meets the memory requirement, establish a serial baseline there, then test a bounded pool. Keep Python and dependency versions in the run record. The local examples in this guide establish correctness, not a speedup or a Hivenet capacity benchmark.