Playwright web scraping with Python: build a repeatable browser job

Playwright web scraping lets a Python program collect data after a browser has rendered a page and run its JavaScript. The useful unit of work is a validated result you can produce again: open the page, wait for the required content, extract records, handle pagination, and close the browser even when something fails.

This walkthrough is for developers turning a browser script into a recurring data job. It uses an owned, synthetic catalog with two pages, delayed content, duplicate products, and invalid prices. You will run a headless browser, inspect accepted and rejected records, and test what happens when the job cannot finish.

Test limits, November 11, 2026: All 17 sample tests passed on Windows with Python 3.12.14, Playwright 1.64.1, and Chromium 156.0.8078.4. The demo returned three accepted records and four rejected rows. Linux/macOS, Docker build and runtime, scheduling, production websites, and Hivenet deployment were not tested. These checks are not a performance or memory benchmark.

Choose browser automation when the data needs it

Playwright is a browser automation library with a Python API. It supports Chromium, Firefox, and WebKit; this example installs and tests Chromium only. A headless browser runs without a visible browser window, but still executes the page's scripts.

Before launching a browser for every URL, check whether the data you are authorized to collect is already available through an API, an export, or the initial HTML. Those routes may remove the need to render the page at all. Browser automation becomes useful when the result depends on JavaScript, a page interaction, or rendered state that the simpler route does not provide.

Our fixture deliberately inserts its product elements after the initial HTML loads. Fetching that HTML alone will not return those rendered elements. The browser job waits for the fixture's explicit readiness marker before collecting them.

Use the supplied local catalog first. It contains invented products and prices, requires no account, and makes no third-party requests. The sample accepts only its own 127.0.0.1 origin. Adapting it to another site requires a separate review of the site's access rules, data permissions, request limits, and technical behavior.

Download the sample and define the output

Download the Playwright web scraping sample and extract it. The archive includes the source, a test suite, the checked example output, a README, and a Dockerfile for a later packaging review.

• fixture.py starts the local catalog on an available loopback port.
• scrape.py collects records, validates fields, handles bounded retries, and writes a completed snapshot.
• demo.py starts the fixture, runs the job, and stops the fixture afterward.
• test_scrape.py exercises successful collection and controlled failures.

Each accepted record needs a stable ID, a nonempty name, and a nonnegative USD price with exactly two decimal places. Prices stay decimal strings so the output does not introduce binary floating-point rounding. Missing or malformed values go into a rejection report; the program does not guess a replacement.

{
  "id": "a1",
  "name": "Notebook",
  "price_usd": "12.50"
}

The fixture repeats product a2 on its second page. This example keeps the first valid record for an ID and rejects later occurrences as duplicates. That is an explicit snapshot policy. A changing catalog may need a different rule for conflicting values, observation times, and freshness.

Install Playwright and its browser

From the extracted sample directory, create a Python 3.12 virtual environment. On Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python -m playwright install chromium

On Linux or macOS, activate the environment with source .venv/bin/activate instead. If activation is unavailable, invoke the environment's Python executable directly. The package is pinned to playwright==1.64.1; do not silently substitute a different release when reproducing the checked run.

Installing the Python package and installing the browser binaries are separate steps. On a supported Linux host, the browser also needs its system dependencies. Follow Playwright's browser-installation guidance for that host; the Linux and macOS setup commands were not exercised in this Windows test.

Run the complete local job:

python demo.py --output output/catalog.json

The checked run printed:

3 accepted; 4 rejected; 2 pages; 2 attempts
Chromium 156.0.8078.4; wrote output\catalog.json

The script starts the local server itself. You do not need to expose a port publicly or keep another terminal running. Open output/catalog.json to inspect the three accepted products and the raw rejected rows with their reasons.

Wait for dynamic content before extracting data

A navigation milestone does not establish that the data you need is ready. page.goto() returns a response that lets the job inspect the HTTP status. The sample waits for domcontentloaded, then separately waits for #catalog[data-ready="true"].

The fixture sets that attribute only after inserting its product rows and next-page link. This gives the job an observable condition instead of an arbitrary delay. On another site, define a condition that actually represents complete usable content; the presence of the first product alone may be too early.

Inside the sample's page-reading function, extraction uses this sequence after the HTTP response has been checked:

await page.locator('#catalog[data-ready="true"]').wait_for(
    state="attached", timeout=timeout_ms
)
rows = await page.locator("[data-item]").evaluate_all("""nodes => nodes.map(node => ({
    id: node.getAttribute('data-item'),
    name: node.querySelector('.name')?.textContent ?? null,
    price: node.querySelector('.price')?.textContent ?? null
}))""")
next_link = page.get_by_role("link", name="Next page", exact=True)
next_href = await next_link.get_attribute("href") if await next_link.count() else None

The CSS selectors describe this fixture's structure. The next-page link uses its accessible role and name. Playwright locators provide ways to find elements, while evaluate_all() runs the small extraction function over the matching nodes. Its result still needs validation in Python.

Do not treat auto-waiting as a promise that every field is complete. Here, the explicit readiness wait precedes the bulk extraction. Missing price elements become null in the raw data and are rejected by the validation step rather than crashing an unrelated selector operation.

Bound pagination, retries, and job duration

A repeatable scraper needs a stopping rule for both successful and unsuccessful work. The sample follows the next-page link sequentially and records each visited URL. It stops when there is no next link; it fails if a URL repeats, pagination leaves the fixture origin, or the page limit is reached before pagination ends.

• Page limit: three pages. Reaching the limit with another page pending is a failure, not a successful partial dataset.
• Attempts: two per page. HTTP 5xx responses and navigation/readiness timeouts can trigger a retry. There is a short bounded delay before the second attempt.
• Navigation and readiness: each has a 3,000 ms timeout. They are separate waits.
• Job deadline: run_job() requests cancellation after 30 seconds. Cleanup may take additional time.

HTTP 4xx responses stop this example without retrying, including 404 and 429. A real service needs its own response-specific policy, including how to honor Retry-After and enforce a shared request budget. Increasing retries indiscriminately can add load while delaying a result that will never become usable.

The reference job also uses request routing to allow only its fixture origin and blocks service workers in its browser context. This keeps the lab self-contained. It is not a replacement for network isolation or a security design for arbitrary websites.

The complete exception handling is in read_page() and collect() in the archive. Keep those functions together when reproducing the job; a copied extraction fragment does not include its retry, pagination, or cleanup behavior.

Validate records before replacing the output

The sample validates fields before adding a record to the accepted set. The second page deliberately contains a duplicate ID, a malformed price, an empty name, and a missing price. Its rejection reasons are:

duplicate_id
invalid_price
missing_name
invalid_price

A successful HTTP request therefore does not count as a usable record. The accepted count is three, even though the two pages contain seven raw rows. Inspect both counts when checking whether a scraping change has improved the job.

The writer runs only after collection finishes. It writes one temporary JSON file beside the destination and replaces the destination with that completed snapshot. If collection fails, the previous snapshot remains. A rerun replaces the same output file instead of appending the same products again.

This is a single-writer file pattern. It has no concurrent-writer coordination, retained history, or tested crash/power-loss recovery. If another system needs to consume the output, define a schema version, observation time, retention rules, and a delivery contract before treating the file as an interface.

Close pages and the browser on failure too

The job opens one browser instance and one browser context, then creates a page for each attempt. A finally block closes each page. Outer cleanup closes the context and browser after collection, including the tested failure paths.

Playwright's context-close API closes the pages belonging to that context. Closing explicitly also makes the lifetime of the resource visible in the code. Do not rely on a script eventually exiting to clean up a long-running worker.

A browser context separates session state such as cookies and storage. It does not provide an operating-system security boundary for untrusted pages. Keep browser execution, sandboxing, credentials, and network access in the deployment review.

Start with this sequential job before creating multiple pages concurrently. Measure total browser-process memory, job duration, accepted output, and failures together. More vCPUs do not establish how many browser jobs fit in RAM. The existing vCPU vs. core guide explains the hardware distinction; browser-specific concurrency still needs a workload test.

Run the checks before changing the fixture

Run the supplied tests from the sample directory:

python -m unittest -v

The 17 passing tests include six contract/file checks and eleven browser checks. The browser checks cover delayed rendering and pagination, validation and duplicates, a recoverable 503, exhausted retries, a non-retried 404, a missing readiness signal, a pagination cycle, the page cap, an external next-page link, identical rerun output, page cleanup, browser disconnection, and deadline cancellation.

Those checks establish behavior against this controlled fixture. They do not establish correctness on a different website or sustained throughput. After changing selectors or the output contract, add fixtures for the new cases and rerun the suite. Keep expected records available so a script that exits successfully but returns incomplete data is caught.

Package the job for a VM after the local run

The archive includes this version-matched Dockerfile for packaging review:

FROM mcr.microsoft.com/playwright/python:v1.64.1-noble
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt \
    && mkdir -p /output && chown pwuser:pwuser /output
COPY --chown=pwuser:pwuser fixture.py scrape.py demo.py ./
USER pwuser
CMD ["python", "demo.py", "--output", "/output/catalog.json"]

The official Playwright Docker guidance says its image includes browsers and system dependencies; the Python package must be installed separately. Keep the image and package versions aligned. The image is intended for testing and development, and its documentation cautions against using it to visit untrusted sites.

The Dockerfile switches to pwuser, and the Python code requests the Chromium sandbox. Neither choice proves that the sandbox works on a particular Linux host. Review the documented seccomp/user-namespace requirements, resource limits, and host compatibility before running it. Do not disable the sandbox to make a failed launch disappear.

Docker build and execution remain untested for this package. The default output path is inside the container; configure a writable persistent output mount before removing the container. No production-site crawling procedure is implied by this local fixture.

For the host setup, use Run Docker on a Compute VM. For a CPU-only workload evaluation, the vCPU VM buying guide covers configuration and operating responsibilities. Confirm the selected environment through Hivenet's vCPU documentation before deploying.

Once the local job works, compare CPU instance configurations using a representative workload and measured memory requirements. Keep the accepted-record count, retry rate, runtime, and complete cost in the comparison. Hivenet provides the compute environment; the browser job and its operating controls remain yours to implement.