Web scraping infrastructure: queues, retries, and storing results

Web scraping infrastructure is the set of services that schedules collection, limits requests, runs extraction, validates records, and stores results. For recurring jobs, start with a bounded queue, a small worker pool, a shared request budget, and a result store that can accept repeated work safely. Adding workers before those controls are in place can multiply failures and duplicate output.

This guide follows a collection job from admission to storage. It includes a downloadable Python lab that tests queue and recovery behavior against an owned local HTTP fixture. Use the Playwright web scraping guide for the browser job itself; this article covers the infrastructure around that job.

Test scope, November 12, 2026: All 15 tests passed on Windows with Python 3.12.14 and SQLite 3.53.1. The lab exercised two worker threads, real loopback HTTP requests, a child process exiting after a claim, and an injected transactional write failure. It did not run Playwright, Celery, Redis, Docker, a Linux VM, or a Hivenet deployment. It is a single-host pipeline demonstration, not a distributed crawler or capacity benchmark.

Separate the stages of your web scraping pipeline

A scheduler decides what needs collecting and when. A URL frontier records candidates discovered from a seed list or links. A queue holds admitted jobs until workers can run them. Extraction turns page content into structured output; validation decides whether that output is acceptable; storage makes accepted records available to downstream systems.

These are responsibilities, not a requirement to deploy six services. A small collector can keep queue state and records in one database. Split components when measurement or failure isolation justifies the extra operating work. If the source offers an authorized API or export, evaluate that before building browser automation. Static HTML may only need an HTTP client; a headless browser is useful when the required content depends on page execution or interaction.

For browser workers, measure the whole browser process tree and bound active contexts. The Playwright memory usage guide explains that measurement separately. A queue limit cannot prevent a single browser job from exhausting memory.

Define a job and its output before adding concurrency

Use a job key that represents the work you intend to repeat. In this lab it combines the source origin, entity ID, collection window, and extractor version. Repeated submission of that same key is ignored, including after completion. A new collection window creates a new job so yesterday's success does not suppress today's refresh.

URL deduplication and record deduplication solve different problems. Two URLs can describe the same entity, while one URL can change over time. Keep a source-specific entity key and decide which URL variations are equivalent. Do not strip query parameters without checking whether they change the requested data.

Contract	Lab rule
Job identity	Origin + entity + collection window + fixture-v1 extractor label
Outstanding work	At most 20 queued or leased jobs
Attempt	One claim; three claims maximum, including expired leases
Record	entity_id, integer value from 0 to 1,000,000, and finite observed_at Unix time
Freshness check	Observation time is no more than 300 seconds old and is not in the future
Latest result	One row per origin and entity; only a newer observation replaces it

The fixture supplies its own observation timestamp. A real extraction adapter should record collection time itself and keep the source's revision timestamp separately when available. A recent fetch does not prove that the underlying data is recent. These schema and timing rules are demonstration policies, not universal requirements for scraped data.

Bound admission and share the request budget

Backpressure means refusing or delaying new work when the next stage cannot keep up. Count leased jobs as well as waiting jobs toward the queue's capacity. Otherwise, workers can empty the visible queue into their own memory while the scheduler keeps filling it.

Track the age of the oldest waiting job alongside queue depth. A short queue can still contain a stalled job. When a full queue rejects admission, keep the candidate in a durable scheduler or frontier so it can be reconsidered later. The finite lab raises an explicit capacity error; it does not include a recurring scheduler.

A request delay inside each worker is not a shared per-domain limit. Four workers with independent delays can still send four requests together. Coordinate the request budget across every worker that contacts the same source, and define what happens when that coordination is unavailable. Pausing requests is a clear starting policy.

The lab has one owned origin. Its two threads share a lock that remains held through the HTTP response, then waits at least 0.2 seconds before the next request. This deliberately serializes requests to that origin. A separate SQLite gate spaces job claims, but claim timing alone would not guarantee network request timing. Neither mechanism is a tested multi-host rate limiter.

Browser navigation complicates this boundary: one page can load scripts, images, frames, and API responses. A navigation budget is not automatically a budget for all HTTP requests. When adding Playwright, test subrequests, redirects, session state, and cleanup against the actual target policy. Stop on access refusals and investigate permissions, source limits, and extraction failures before increasing concurrency or introducing proxy rotation.

Classify failures before retrying them

Retry logic needs an attempt limit, a delay, and a rule for which failures are temporary. A timeout might justify another attempt. A record with the wrong schema needs investigation. Repeating a broken selector against the same page can spend the entire request budget without producing usable data.

Failure	Response in the lab
Transport error or HTTP 502, 503, 504	Retry, with waits of one then two seconds; stop after attempt three
Other non-200 response, including redirects	Fail for inspection; no redirect following
Malformed, stale, or oversized output	Fail without retrying
Worker disappears after claiming	Reclaim after the 15-second lease expires, within the attempt budget
Database write fails	Stop the worker; retain the leased job for recovery after storage is repaired

For external sources, add a reviewed response policy for rate-limit signals and any server-provided waiting instructions. The fixture does not implement that policy. Production backoff commonly also needs jitter to spread retries across workers; the lab uses fixed delays so its behavior is easy to inspect.

Keep exhausted jobs in a failed state with a bounded reason and enough context to investigate. This serves the same operational purpose as a dead-letter queue: work has stopped automatically and now needs a decision. Replay only after addressing the cause, with a deliberate new attempt budget. A failed row is not permission for an infinite retry loop.

Make result writes safe to repeat

A worker can finish extraction and lose its connection before the coordinator hears about it. Another worker may then repeat the job. Design storage so that this uncertainty does not create duplicate records or overwrite newer data with an older response.

The sample writes the result and changes the job to done in the same SQLite transaction. It first checks that the worker still owns an unexpired lease and that its attempt number is current. An old worker cannot finish a job after a later attempt has taken over. The following function is included in the download:

def complete(db, job, record, now):
    validate(record, job, now)
    with transaction(db):
        if not current(db, job, now):
            return False
        db.execute('''INSERT INTO results VALUES(?,?,?,?)
          ON CONFLICT(origin,entity) DO UPDATE SET
          observed_at=excluded.observed_at,body=excluded.body
          WHERE excluded.observed_at > results.observed_at''',
                   (job['origin'], job['entity'], record['observed_at'], json.dumps(record)))
        db.execute("UPDATE jobs SET state='done',lease=NULL,error=NULL WHERE job_key=?", (job['job_key'],))
    return True

The SQL update accepts only a newer observation for the entity. A completed older observation can therefore leave the latest stored record unchanged. Here, done means the valid result was considered under that rule, not that every attempt produced a new row.

Python's SQLite transaction documentation explains the explicit transaction controls used by the helper. SQLite permits only one write transaction at a time. Keep transactions short and keep network requests outside them. This file-backed design is for one host; moving the file to a network share is not a distributed queue design.

If the result store and queue are separate systems, a database commit and a broker acknowledgment are not one atomic operation. Make the destination write idempotent and reconcile uncertain completion. An outbox or another durable handoff may be appropriate, but it needs its own tests. This lab does not establish exactly-once execution or atomic writes to an external store.

Run the bounded reference example

Download the collection pipeline lab, extract it into a new directory, and use Python 3.12 or later. It uses only the standard library. Run these commands from the extracted directory:

python -m unittest -v
python demo.py --output demo-output

The output directory must not already exist. The program starts a fixture on 127.0.0.1, submits six candidates containing one duplicate, and runs two worker threads. It contacts no external website and launches no browser. Each response is capped at 64 KiB. The socket timeout is two seconds; the worker loop checks a 30-second deadline between iterations. These are controlled-fixture limits, not a hard cancellation guarantee for a slow-streaming external server.

The captured run produced five distinct jobs, eight HTTP requests, and two stored records:

good     done    1 attempt
retry    done    2 attempts
bad      failed  1 attempt
stale    failed  1 attempt
exhaust  failed  3 attempts

The three failures are expected: invalid data, an old observation, and a source that keeps returning HTTP 503. The observed minimum gap between fixture requests was about 0.203 seconds in this run. That confirms this run's bounded request behavior; it is not a throughput estimate for web scraping at scale.

Inspect demo-output/summary.json and demo-output/pipeline.sqlite. The automated checks also cover duplicate completion, full-queue rejection, concurrent claims, stale-worker rejection, and older observations arriving after newer ones. One child process exits immediately after claiming work; the test advances the lease clock before reclaiming it. A database trigger injects a failure between the result write and completion update, then verifies rollback and recovery after the trigger is removed. Neither check simulates losing the host or its disk.

Choose storage and retention around the consumer

A latest-record table suits consumers that need the current value for each entity. Historical analysis needs a separate append-only observation history, with collection window and extractor version attached. Object storage can hold bounded raw responses or larger immutable batches, while a relational database can serve records that need constraints and queries. Choose the storage shape from the access pattern rather than treating every collection as a data lake.

Set separate retention rules for raw content, normalized records, job history, and logs. Saving all raw HTML indefinitely increases storage and review work. Keep only material you have a reason and permission to retain, with a deletion policy and restricted access where needed.

The sample's purge_results function removes old snapshots but retains job keys. Deleting result data therefore does not cause the same collection-window job to run again. The lab has a finite input set; a recurring service also needs a bounded job-history policy. If you delete deduplication keys, state how long duplicate suppression lasts and how older submissions are rejected or reconciled.

For the next stage, the Python ETL pipeline guide covers validating and loading a daily CSV into PostgreSQL. Its input schema differs from this lab's JSON record. Mapping fields, timestamps, and rejected rows is a separate integration step; the examples are not connected automatically.

Move to a broker or remote workers after testing the boundaries

The Celery and Redis CPU VM guide explains producer, broker, worker, and result-backend responsibilities. Its sample calculates numbers on a private Docker network. Browser collection needs a separately designed outbound access path and a tested extraction task. Do not paste the pipeline into that stack and assume its networking or recovery behavior carries over.

In Celery, late acknowledgment changes when a normal task message is acknowledged, but some child-process termination cases are still acknowledged. The task_reject_on_worker_lost option changes that behavior and requires care around repeated failures. Consult Celery's task acknowledgment and retry documentation, then test your actual broker settings, worker termination, visibility timeout, and destination writes. The SQLite lease used here is not a model of all Celery delivery behavior.

Before increasing worker count, record queue age, attempts per completed job, validation failures, request status by source, result-write latency, and memory per active browser. Add a freshness measure that reflects the consumer's deadline. A low HTTP error rate can hide a pipeline that successfully saves the wrong fields after a layout change.

Use the CPU VM buying guide when you have measured the worker and can describe its memory, storage, and networking needs. Compute with Hivenet can be evaluated for self-managed workers; you remain responsible for the queue, extraction logic, source access, data handling, and recovery. This article makes no managed crawler, capacity, or deployment-performance claim.

Web scraping infrastructure questions

Do small scraping jobs need a queue?

A finite one-off job may be easier to run synchronously. A queue becomes useful when collection is recurring, workers must recover unfinished work, or producers can submit faster than extraction and storage can finish. Keep a capacity limit and a durable record of accepted work.

Does retrying guarantee that every job runs once?

No. Recovery can repeat execution, especially when completion is uncertain. Use stable job keys, bounded retries, and idempotent result writes. The sample prevents duplicate rows for its latest-record table, but does not promise exactly-once execution.

What should be tested before distributed web crawling?

Test aggregate source limits across workers, duplicate delivery, interrupted work, stale completions, malformed output, and failed writes. Then test the actual multi-host queue, browser adapter, network boundaries, and storage recovery. Passing the local lab establishes only the behaviors and failure cases described above.