Celery with Redis: deploy a background worker on a CPU VM

Celery with Redis lets a Python application send work to a separate worker process. Celery defines and executes tasks; Redis carries the queued messages and can also store task results. This guide builds a private, single-VM example for developers who need to move work out of an API process.

The example uses Docker Compose, a command-line producer, one Celery worker, and Redis. It calculates the sum of squared integers, returns a task ID, and lets you check the result. It has no public endpoint and uses no customer data. Start with FastAPI background tasks: when to use Celery if you are still deciding whether a queue is necessary.

Validation status, November 2, 2026: Fifteen local Python checks passed on Windows with Python 3.12.14, Celery 5.6.3, and redis-py 6.4.0. Python compilation and Docker Compose configuration validation also passed. No container image was built or run, and no Redis broker, Celery worker process, Linux VM, or Hivenet deployment was exercised. Startup, network isolation, persistence, shutdown, and recovery procedures below remain untested.

How the producer, broker, worker, and result backend fit together

The producer sends a task name and arguments to the broker. A worker reads the message and runs the matching Python function. A result backend records state and return values when configured. These are separate responsibilities, even when one Redis server performs both storage roles.

Here, Redis database 0 holds broker data and database 1 holds results. That separates key spaces, not memory, durability, or failure risk: both databases still belong to one Redis process on one VM. The result backend is useful for this exercise but is optional for tasks whose outputs are stored elsewhere. See the Celery Redis configuration reference.

A task ID identifies a submission. It does not prove the work finished. If a connection fails while publishing, the producer may not know whether the broker accepted the message. A real application needs a durable job record and a reconciliation policy when that uncertainty matters.

Prepare a private Linux VM

Use a disposable Linux VM with Docker Engine, the Compose plugin, and OpenSSL installed. Confirm SSH access and enough disk space for images, logs, and Redis data. The Docker-on-Compute guide covers the surrounding VM workflow. These Linux instructions have not been executed for this article.

Use a new directory and the seven files below. Run commands from that directory. The Compose project name is celery-cpu-demo; use a VM without another project of that name. The worker starts with concurrency 1 to keep the exercise small. This is a starting configuration, not a capacity recommendation.

Define the Celery application and task

Save this as tasks.py. The broker and backend addresses use redis, the Compose service name. A container's localhost would refer to that same container, not the Redis service.

import os
from urllib.parse import quote

from celery import Celery

password = quote(os.environ["REDIS_PASSWORD"], safe="")
app = Celery(
    "cpu_jobs",
    broker=f"redis://:{password}@redis:6379/0",
    backend=f"redis://:{password}@redis:6379/1",
)
app.conf.update(
    task_serializer="json",
    result_serializer="json",
    accept_content=["json"],
    result_accept_content=["json"],
    result_expires=3600,
    task_track_started=True,
    worker_prefetch_multiplier=1,
    broker_connection_retry_on_startup=True,
)


def validate_numbers(numbers):
    if not isinstance(numbers, list) or not 1 <= len(numbers) <= 10_000:
        raise ValueError("Provide a list of 1 to 10,000 integers.")
    if any(type(n) is not int or abs(n) > 1_000_000 for n in numbers):
        raise ValueError("Each value must be an integer from -1,000,000 to 1,000,000.")
    return numbers


@app.task(acks_late=True)
def sum_squares(numbers):
    values = validate_numbers(numbers)
    return sum(n * n for n in values)

The task accepts 1 to 10,000 integers, each between −1,000,000 and 1,000,000. It rejects booleans and non-integers and has no external side effects. Repeating it with the same input produces the same result. These limits bound this demonstration's calculation; they do not establish production throughput or protect an arbitrary future API's raw request size.

JSON is the accepted message and result format. Results expire after one hour, so the backend is not a permanent job history. The password is read from the environment and URL-encoded before inclusion in connection URLs. Do not log or share those URLs.

Add a producer and task-status command

Save this as producer.py. It validates input before submission and prints a task ID only after the publish call returns. A submission exception exits the command without a success message; it is not evidence that no message reached Redis.

import argparse
import json

from tasks import app, sum_squares, validate_numbers


def submit(numbers):
    validate_numbers(numbers)
    result = sum_squares.apply_async(args=[numbers], retry=False)
    return {"task_id": result.id, "state": "submitted"}


def status(task_id):
    result = app.AsyncResult(task_id)
    state = result.state
    response = {"task_id": task_id, "state": state}
    if state == "SUCCESS":
        response["result"] = result.result
    elif state == "FAILURE":
        response["error"] = "Task failed; inspect the worker logs."
    elif state == "PENDING":
        response["note"] = "No result recorded: queued, unknown, or expired."
    return response


def main():
    parser = argparse.ArgumentParser()
    commands = parser.add_subparsers(dest="command", required=True)
    add = commands.add_parser("submit")
    add.add_argument("numbers", type=int, nargs="+")
    check = commands.add_parser("status")
    check.add_argument("task_id")
    args = parser.parse_args()
    output = submit(args.numbers) if args.command == "submit" else status(args.task_id)
    print(json.dumps(output))


if __name__ == "__main__":
    main()

PENDING does not distinguish a waiting task from an unknown ID or an expired result. The status command makes that ambiguity visible. A customer-facing status API would need authentication, access checks for each job, bounded connection timeouts, and its own durable state model. This command is only for a trusted operator on the VM.

Package the worker with Docker

Save the package requirements as requirements.txt:

celery[redis]==5.6.3
redis==6.4.0

Save this as Dockerfile. The application runs as a non-root user. The local checks used the pinned Celery and Redis client versions, but the Linux image and its dependency resolution have not been tested. The Python image tag can change; record and pin image digests after validating your own build.

FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt \
    && useradd --uid 10001 --create-home worker
COPY tasks.py producer.py ./
USER worker
CMD ["celery", "-A", "tasks:app", "worker", "--loglevel=INFO", "--concurrency=1"]

Save this allowlist as .dockerignore. Only application files enter the build context; the password file and local tests stay out.

*
!requirements.txt
!tasks.py
!producer.py
!Dockerfile

Save these exclusions as .gitignore so credentials and temporary Python files do not enter a new Git repository:

.env
.env.*
__pycache__/
.pytest_cache/

Configure Redis and Celery with Docker Compose

Save this as compose.yaml. Redis has no published host port. The services share an internal Docker network, and the producer runs only when explicitly requested through the tools profile.

name: celery-cpu-demo

x-app: &app
  build: .
  environment:
    REDIS_PASSWORD: ${REDIS_PASSWORD:?Create .env first}
  networks: [jobs]
  depends_on:
    redis:
      condition: service_healthy
  logging:
    driver: json-file
    options:
      max-size: "10m"
      max-file: "3"

services:
  redis:
    image: redis:8.10.3-alpine
    environment:
      REDIS_PASSWORD: ${REDIS_PASSWORD:?Create .env first}
    command:
      - sh
      - -c
      - exec redis-server --appendonly yes --appendfsync everysec --requirepass "$${REDIS_PASSWORD}"
    volumes: [redis-data:/data]
    networks: [jobs]
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", 'REDISCLI_AUTH="$${REDIS_PASSWORD}" redis-cli ping | grep -q PONG']
      interval: 5s
      timeout: 3s
      retries: 10
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
  worker:
    <<: *app
    restart: unless-stopped
    stop_grace_period: 60s
  producer:
    <<: *app
    profiles: [tools]
    entrypoint: ["python", "producer.py"]
    command: ["--help"]

networks:
  jobs:
    internal: true
volumes:
  redis-data:

The service_healthy dependency waits for the Redis health check during startup. It does not continually restart the worker when Redis becomes unhealthy. See Docker's startup-order documentation. The worker has a container restart policy and a 60-second stop grace period, neither of which proves that an interrupted task will be recovered.

The internal network setting is intended to isolate this stack from external networks. There are no outbound API calls in the example. If a future task needs external access, design that path deliberately and verify the resulting network boundaries. Do not expose Redis port 6379 to make the producer reachable.

A password adds authentication, but environment variables and process arguments remain visible to administrators with Docker or host access. This is a trusted, single-host exercise using a shared Redis password. For a broader deployment, use a managed secret mechanism, Redis ACLs, and TLS when traffic crosses an untrusted boundary. Consult the Redis security guidance.

Create the local password file

On the disposable Linux VM, create .env with a fresh random password. This command overwrites that file, so use the new exercise directory and run it once. Keep the file private and out of backups or sharing systems that are not intended for secrets.

umask 077
printf 'REDIS_PASSWORD=%s\n' "$(openssl rand -hex 32)" > .env
docker compose config --quiet

Use --quiet for configuration validation. Plain docker compose config can print the expanded password. The local editorial check used a placeholder value and did not create a deployment credential.

Start the worker and submit one job

The following is the unexecuted container procedure. Build and start the worker and Redis, then inspect their state before submitting work:

docker compose up --build -d redis worker
docker compose ps
docker compose logs --tail=100 worker
docker compose --profile tools run --rm producer submit 1 2 3

The expected submission response contains a task ID and state: submitted. Copy the actual ID into the next command. Do not use the placeholder literally:

docker compose --profile tools run --rm producer status TASK_ID

For input 1 2 3, a successful run should report SUCCESS and result 14. That is a mathematical expectation, not captured worker output. If the state is FAILURE, inspect the worker logs; if it remains PENDING, check the ID, worker connection, task registration, and result expiry before assuming the job is queued.

If startup fails, check that .env exists, Redis becomes healthy, and the worker can authenticate. An “unregistered task” error suggests a task-name or code-version mismatch between producer and worker. Preserve relevant logs without copying credentials or real task payloads into a public issue.

Plan for retries, interruptions, and duplicate execution

This pure calculation uses acks_late=True, so normal acknowledgment follows execution. Celery still acknowledges certain child-process termination cases even with late acknowledgment. The task_reject_on_worker_lost option changes that behavior but can create repeated failure loops; it is not enabled here. Read Celery's task acknowledgment and retry guidance before adapting the sample.

No automatic task retry is configured, and the producer disables automatic publish retries. If your real task calls a service, retry only appropriate transient failures with bounded attempts, delay, and timeouts. Design repeated execution before retrying side effects such as payments, emails, or file creation.

Redis's default visibility timeout for Celery is one hour. An unacknowledged message can be delivered again after that interval; long-running jobs or distant ETA/countdown tasks need particular care. Increasing the timeout can delay recovery. It does not give exactly-once execution. Review the Redis transport caveats when changing this behavior.

What Redis persistence covers

The named volume stores Redis data outside the container's writable layer. Append-only persistence with appendfsync everysec reduces the loss window, but a crash can still lose recent writes. A volume is not a backup and cannot protect against loss of the VM or its disk. Both the broker and result backend share that risk here. See the Redis persistence reference.

Before using this design for consequential work, test recovery with disposable inputs. Stop the worker gracefully while work is active; separately test a worker interruption, Redis restart, and host restart. Record which submitted IDs finish, fail, repeat, or become unknown. This small calculation may complete too quickly to observe interruption, so use a controlled test task and never infer recovery from a successful normal run.

Celery documents warm shutdown on TERM, which waits for active tasks. A container stop timeout can still force termination. None of these interruption cases was tested for this article.

Connect the worker to CPU capacity

The useful CPU question is how much work must finish within a given time, including time waiting in the queue. Measure a representative task's CPU use, peak memory, and duration before increasing concurrency. Leave room for Redis, the operating system, and any colocated API. The tiny arithmetic task above is a functional example, not a sizing benchmark.

Use the vCPU vs. physical core guide to interpret advertised CPU resources, and the CPU VM buying guide to consider memory, storage, and access requirements. A worker and API on separate VMs can be managed independently, but that adds networking and operating work.

For the next sizing decision, see how to choose Celery worker concurrency and CPU capacity. It separates CPU time, queue delay, and memory per process, with hypothetical calculations and a proposed test procedure.

For an API entry point, return to the FastAPI deployment guide. The existing API example does not automatically become durable by adding this worker: submission, job records, authorization, and failure reconciliation still need an explicit design.

Hivenet's CPU VM documentation describes general compute and background-service uses. Confirm the current configuration, storage behavior, networking, and availability with the team before planning a deployment. This guide contains no Hivenet runtime, capacity, or performance result. You can discuss a CPU deployment with the job type, expected volume, and recovery requirements.

Stop the exercise without deleting its data volume

When finished, use the command below. It removes the project's containers and network while leaving its named volume. Do not add --volumes unless you intend to delete the stored queue and results. Stopping containers does not delete or stop billing for a cloud VM; manage that separately in your provider's controls.

docker compose down

‍