← Blog
A lavender balance scale with two ivory pans, each holding an orange block.
Published on
2026-11-18

Nginx load balancer: upstreams, retries, and failover

An Nginx load balancer sends requests to a group of application servers instead of forwarding every request to one backend. A useful first configuration should make that behavior observable: you should be able to identify which backend answered, see a failed attempt in the logs, and check what happens when a backend returns.

This guide uses open-source Nginx 1.31.8 and two small Python backends on loopback addresses. The configuration and 31 functional checks ran on Windows with Python 3.12.14. Nginx documents limitations in its native Windows version, so these results establish behavior, not Linux capacity or production availability.

Choose application instances that can serve the same requests

Load balancing works best when either backend can handle the next request correctly. Decide where sessions, uploaded files, and shared application state will live. Sending a user to a second instance does not make the first instance's local files or in-memory session available there.

Our backends differ only in their identifying name. The default policy uses round-robin selection with equal weights. The Nginx load-balancing guide describes other methods, but this example does not benchmark least connections, weighted pools, or sticky sessions. If you are still choosing the proxy software, start with Caddy vs Nginx.

The entire demonstration runs on one machine. It can reveal a stopped backend process, but it cannot survive loss of that machine. Keep this distinction in mind before interpreting successful failover as high availability.

Create two backends whose responses you can identify

Save this as backend.py in a new demonstration directory. It uses Python's standard library and listens only on 127.0.0.1. The slow and dropped-response paths are deliberate test controls; this is not a production application server.

import argparse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import socket
import threading
import time


class Backend(ThreadingHTTPServer):
    allow_reuse_address = True
    daemon_threads = False

    def __init__(self, name, port):
        self.name = name
        self.received = []
        self.lock = threading.Lock()
        super().__init__(("127.0.0.1", port), Handler)


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def log_message(self, *args):
        pass

    def reply(self, status, data):
        body = json.dumps(data).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.send_header("Connection", "close")
        self.end_headers()
        try:
            self.wfile.write(body)
        except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError):
            pass
        self.close_connection = True

    def do_GET(self):
        with self.server.lock:
            self.server.received.append(("GET", self.path))
        if self.path.startswith("/read/slow"):
            time.sleep(1.4)
        status = 503 if self.path.startswith("/read/status503") else 200
        self.reply(status, {"backend": self.server.name, "path": self.path})

    def do_POST(self):
        size = int(self.headers.get("Content-Length", "0"))
        if not 0 <= size <= 8192:
            self.reply(413, {"error": "body too large"})
            return
        self.rfile.read(size)
        with self.server.lock:
            self.server.received.append(("POST", self.path))
        if self.path.startswith("/write/drop"):
            self.close_connection = True
            self.connection.shutdown(socket.SHUT_RDWR)
            return
        self.reply(200, {"backend": self.server.name, "accepted": True})


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("name", choices=["A", "B"])
    parser.add_argument("port", type=int)
    args = parser.parse_args()
    with Backend(args.name, args.port) as server:
        try:
            server.serve_forever()
        except KeyboardInterrupt:
            pass

In separate terminals, run python backend.py A 19011 and python backend.py B 19012. Each GET response identifies its backend. The POST handler records a received request in memory; its drop path then closes the connection without returning a response. No real payment, database update, or other business write occurs.

Keep the ports private and use a directory containing only this example. The automated checks stop and restart their own backend instances; do not run the manual instances at the same time as those checks.

Define the Nginx upstream group and request policies

Save this complete file as nginx.conf and create a logs directory beside it. Both backend addresses belong to app_pool. The public-facing socket in this demonstration is also loopback-only, on port 18083.

worker_processes 1;
pid logs/nginx.pid;
error_log logs/error.log info;

events { worker_connections 128; }

http {
    log_format upstream_json escape=json
        '{"uri":"$uri","method":"$request_method",'
        '"status":$status,"upstream":"$upstream_addr",'
        '"upstream_status":"$upstream_status",'
        '"upstream_seconds":"$upstream_response_time"}';
    access_log logs/access.log upstream_json;

    upstream app_pool {
        server 127.0.0.1:19011 max_fails=1 fail_timeout=3s;
        server 127.0.0.1:19012 max_fails=1 fail_timeout=3s;
    }

    server {
        listen 127.0.0.1:18083;
        server_name localhost;
        client_max_body_size 8k;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Connection close;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_connect_timeout 500ms;
        proxy_send_timeout 1s;
        proxy_read_timeout 1s;

        location = /healthz {
            access_log off;
            return 204;
        }

        location /read/ {
            limit_except GET { deny all; }
            proxy_pass http://app_pool;
            proxy_next_upstream error timeout;
            proxy_next_upstream_tries 2;
        }

        location /write/ {
            limit_except POST { deny all; }
            proxy_pass http://app_pool;
            proxy_next_upstream off;
        }

        location / { return 404; }
    }
}

Use the Nginx executable with an explicit prefix for this directory. For example, nginx -p /absolute/path/lb-demo/ -c nginx.conf -t checks the file. On Windows, use a path such as C:/path/lb-demo/, with forward slashes and quotes if needed. After the check succeeds, run the same command without -t to start this instance. Always keep the same prefix for later control commands.

The read route permits GET and HEAD, and can try a second backend for an error or timeout. The write route permits POST and disables passing the request to another backend. Our configuration explicitly closes upstream HTTP connections to keep the fixture simple; it does not demonstrate connection-pool tuning. Body size and timeout values are small demonstration limits, not production recommendations.

max_fails=1 and fail_timeout=3s govern passive failure accounting and temporary exclusion. Consult the upstream module reference when adapting these controls to your application and worker topology. A single-worker local run does not verify shared state across multiple proxy workers.

Check distribution using responses and upstream logs

Save this as probe.py and run python probe.py while both backends and Nginx are running.

import http.client
import json


def request(path, method="GET", body=None):
    connection = http.client.HTTPConnection("127.0.0.1", 18083, timeout=5)
    try:
        connection.request(method, path, body=body)
        response = connection.getresponse()
        return {"status": response.status,
                "body": response.read().decode("utf-8")}
    finally:
        connection.close()


if __name__ == "__main__":
    for number in range(6):
        result = request("/read/demo/" + str(number))
        print(json.dumps(result))

In the recorded six-request run, the backend names were A, B, A, B, A, B. Each received three sequential requests. This observation is useful for checking the configuration, but does not promise an exact split for every burst of concurrent requests or every production workload.

The access log records the final client status alongside upstream addresses, upstream statuses, and upstream response times. After backend B stopped, one successful client request had this recorded upstream result:

{
  "uri": "/read/failover/0",
  "method": "GET",
  "status": 200,
  "upstream": "127.0.0.1:19012, 127.0.0.1:19011",
  "upstream_status": "504, 200",
  "upstream_seconds": "0.505, 0.002"
}

The first connection attempt timed out on this Windows run; the second backend returned 200. Looking only at the final client status would hide that failure. Other network failure modes can produce different statuses. Keep upstream details in operational logs rather than exposing internal addresses through response headers. This log format uses the path without the query string, although sensitive data can still appear in a path.

Test failure, temporary exclusion, and recovery separately

Stopping B caused subsequent reads to finish through A. After the failed attempt, the observed requests inside the failure window attempted only A. We restarted B, waited beyond the configured exclusion interval, and verified that both backend names appeared again.

This is passive handling driven by application traffic. It does not run periodic background probes or verify all application dependencies. The local /healthz route returns directly from Nginx; our all-backends-down check still received 204 from it while a read through the pool failed. Use a separate end-to-end check for application readiness.

ScenarioObserved result
Six sequential readsA, B, A, B, A, B; all returned 200
Backend B stoppedOne failed connection attempt followed by a successful attempt on A
Temporary exclusion and recoveryOnly A attempted during observed exclusion; B returned after the interval
Both backends delayedTwo 504 attempts; final 504
Backend-generated 503Returned without retry under the chosen policy
Accepted POST with dropped responseOne backend receipt, one upstream attempt, final 502
All backends stoppedRead returned 502; proxy-local health still returned 204

All 31 checks passed, including syntax validation, method restrictions, log evidence, backend shutdown, and closed fixture listeners. The suite preserves its access logs and sends a graceful quit command only to its own Nginx prefix. For a manual run, stop Nginx with the same -p and -c options plus -s quit, then stop each Python backend with Ctrl+C.

Match retries and timeouts to the operation

Retries are part of application semantics. In our write test, one backend accepted the POST and then dropped the response. The client received 502, and Nginx made only one upstream attempt because the write route uses proxy_next_upstream off. A failed response therefore did not mean the request had never reached the application.

Disabling proxy retries does not stop a caller from retrying. Operations that must not run twice need an application-level duplicate-handling design, such as a durable idempotency key with defined reuse rules. Do not enable broad replay simply to make gateway errors disappear.

The proxy module reference defines retry conditions. Our read policy includes errors and timeouts, not backend-generated HTTP 503 responses; the 503 test returned that status without a second attempt. A retry also cannot repair a response that is already partly delivered to the client.

A read timeout concerns gaps between upstream reads, not a deadline for the whole response. In our controlled slow-response case, both backends exceeded the one-second wait before response headers, producing two 504 attempts and a final 504. A two-attempt limit does not by itself provide an end-to-end latency budget. Review client, proxy, and application deadlines together, especially for uploads and streaming responses.

Plan the deployment beyond this local example

Before placing backends on separate hosts, define network access, TLS termination, process supervision, logs, readiness checks, and how traffic will reach a healthy proxy. A second application process does not remove the single Nginx process, host, or shared dependency as a failure point. Test those failures independently.

This setup also does not perform inference-aware scheduling. For model-serving workloads, the vLLM load-balancing guide covers the different concerns around inference backends and Hivenet Router. Avoid transferring conclusions from this small HTTP fixture to token generation or long-lived streams.

Use the API hosting guide to assess the operating model. Compute with Hivenet is an option for CPU instances running your applications and proxy. Your team manages Nginx, backend processes, network rules, certificates, and monitoring. This example makes no claim about a managed load-balancer service, multi-host resilience, or cloud performance.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.