← Blog
A lavender paper-strip recorder with an ivory roll and an orange button on a pale lavender background.
Published on
2026-11-17

Python logging: levels, handlers, JSON, and exceptions

Python logging records what an application did, which operation failed, and the context needed to investigate. Start with a named logger and an explicit severity threshold. Add structured fields when you need to find all events for one job or request. The useful outcome is a short trail you can follow during a failure.

This guide builds a small example using Python's standard library. It covers levels, handlers, JSON output, exceptions, and duplicate messages, with a deliberately failing batch job you can run locally. The examples were tested on Python 3.12.14.

Start with a named logger

Save the following as basic_example.py and run python basic_example.py. No extra package is required.

import logging

logger = logging.getLogger(__name__)

if __name__ == "__main__":
    logging.basicConfig(
        level=logging.INFO,
        format="%(levelname)s %(name)s %(message)s",
    )
    logger.debug("Detailed input inspection")
    logger.info("Processed %s rows", 3)

The observed output is INFO __main__ Processed 3 rows on standard error. The debug event is filtered out. A logger created with __name__ follows the module name when imported; a directly executed script uses __main__. Keep that identity so you can distinguish your worker from an HTTP client or database library.

Use print() for the result a command-line user requested, such as exported data. Use logging for operational events around that result. Python's logging HOWTO explains the distinction and the standard severity levels.

Choose levels that describe the outcome

Choose a level by what happened to the operation. An expected validation rejection might deserve an informational event in one service; the same rejection could be an error in a nightly import that was supposed to succeed. Define that policy consistently across your application.

LevelExample in a batch applicationOperational question
DEBUGA detailed checkpoint while diagnosing parsingWhat happened inside this step?
INFOA job started or completedIs expected work progressing?
WARNINGA retry was needed, but the job can continueDoes this pattern need attention?
ERRORA job failed and produced no usable resultWhich operation needs recovery?
CRITICALThe service cannot initialize a required componentCan this process keep serving its purpose?

In the first example, INFO includes informational events and higher severities. A small job normally needs a start, a completion or failure, and enough context to connect them. Logging every row makes the important events harder to find and increases storage volume. Add detail for a specific investigation, then reduce it again.

Configure handlers and JSON output once

The logger creates an event. A handler sends it to a destination. A formatter determines the representation. In this example, a root handler writes JSON to standard output, and the module logger sends its events there through propagation. Both thresholds are explicit. An independently configured child logger still has to pass the handler's threshold.

Save this as logging_demo.py. Run it separately from the first script with python logging_demo.py.

import json
import logging
from datetime import datetime, timezone
from logging.config import dictConfig

logger = logging.getLogger(__name__)


class JsonFormatter(logging.Formatter):
    def format(self, record):
        event = {
            "time": datetime.fromtimestamp(
                record.created, timezone.utc
            ).isoformat(timespec="milliseconds"),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        for key, expected_type in (("job_id", str), ("rows", int)):
            value = getattr(record, key, None)
            if type(value) is expected_type:
                event[key] = value
        if record.exc_info:
            event["exception"] = self.formatException(record.exc_info)
        return json.dumps(event, ensure_ascii=True)


def configure_logging(level="INFO"):
    # Application startup only: this replaces logging configuration.
    dictConfig({
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {"json": {"()": JsonFormatter}},
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "stream": "ext://sys.stdout",
                "formatter": "json",
                "level": level,
            }
        },
        "root": {"handlers": ["console"], "level": level},
    })


def run_demo():
    context = {"job_id": "job-17"}
    logger.debug("Inspecting fixture")
    logger.info("Job started", extra=context)
    rows = ["12", "7", "invalid"]
    try:
        values = [int(row) for row in rows]
    except ValueError:
        logger.exception("Job rejected", extra=context)
        return 1
    logger.info("Job completed", extra={**context, "rows": len(values)})
    return 0


if __name__ == "__main__":
    configure_logging()
    raise SystemExit(run_demo())

The example intentionally fails while parsing invalid. Our run produced two JSON lines: an INFO event with the message Job started, then an ERROR event with Job rejected, the same job_id, and a ValueError traceback. The process exited with status 1. The debug message and completion event were absent.

Each record contains a UTC timestamp, severity, logger name, and message. Optional job_id and rows fields are included only with the expected types. The formatter escapes embedded newlines, so the traceback stays inside one physical JSON line. A consumer can parse that line and display the traceback across multiple lines.

dictConfig expresses the handler and formatter connections in one place. Call this configuration at application startup, before work begins. Repeating it in our isolated fixture left one root handler, but it replaces logging configuration and is unsuitable as an import-time action in a reusable library. The setting disable_existing_loggers=False does not promise that third-party handlers will disappear or that their previous setup remains unchanged.

Record exceptions without hiding the failure

Inside an exception handler, logger.exception() records an error with traceback context. The demo uses it where the job is rejected and returns a failing exit status. Logging does not itself retry a job, undo partial writes, or tell a scheduler that the work failed. Choose those behaviors separately.

Assign responsibility for the failure event. If a helper logs an exception and re-raises it, and its caller logs the same exception again, two records may describe one failure even when the handlers are configured correctly. A useful arrangement is for low-level functions to raise errors and for the job or request boundary to record the outcome with its identifier.

For the fixture, job-17 connects the start and rejection. In a real service, choose a non-sensitive identifier that remains stable across retries. Keep the business result separate from diagnostic text: a successful log write does not establish that the data import succeeded.

Choose structured fields deliberately

The JSON formatter builds a small output dictionary instead of serializing every attribute on the log record. Our tests supplied synthetic API-key, password, and payload fields through extra; none appeared in the emitted JSON. Missing optional context and fields with the wrong types were also handled without losing the event.

That allowlist has a specific limit. The formatter still writes the message and exception text. A second test deliberately put synthetic secrets in those locations, and both remained visible. The allowed job_id could also contain sensitive text if the caller supplies it. This example is a field-selection pattern, not a general redaction system.

Keep credentials, authorization headers, raw request bodies, and personal content out of logging calls. Review error messages from libraries before forwarding them to a shared destination. Decide who can read the logs and how long they should remain available. Test those decisions using representative synthetic inputs before collecting production traffic.

A fixed text formatter containing %(job_id)s requires that field on each record it formats. We reproduced a formatting failure when it was missing. The JSON example avoids that dependency by checking optional fields individually. Do not reuse reserved record names such as message for custom context.

Fix duplicate or missing messages at the right layer

Our duplicate-message test attached a handler to the child logger while leaving the root handler in place. One call produced two records. Removing the child handler returned the output to one record. The logging reference documents propagation and why handlers at multiple points in the hierarchy can repeat an event.

SymptomFirst checkUseful correction
One call produces two identical recordsHandlers on both the child and an ancestorChoose one output path; usually keep the root handler
INFO events never appearEffective logger level and handler levelSet both thresholds deliberately
Changing basicConfig appears to do nothingWhether root handlers already existInspect the application's existing configuration
Only records without context disappearRequired fields in the formatterProvide defaults or make the fields optional
Output exists locally but not in the collectorThe selected stream and collector configurationTrace one known event through each stage

By default, basicConfig() leaves an already configured root logger alone. Its force=True option removes and closes existing root handlers. Use that only when your application owns the configuration; it can disrupt a notebook, test runner, or hosting framework. For an intentionally independent logger with its own handler, disabling propagation can be appropriate. Doing that without a working destination can instead hide events.

Choose where the logs will go

Standard output is a practical starting point for this job because the surrounding process manager can capture it. Our JSON demo writes there explicitly; the earlier basic example uses standard error. If a command reserves standard output for machine-readable results, keep operational logs on standard error or another destination.

When running in a container, use the Docker logs guide to retrieve output and configure retention. If writing a local file, the standard library offers rotating file handlers. Rotation needs a size or time policy and a retention limit; it does not create a backup or an alerting service.

This example is single-process. Do not copy it into several workers and give each an independent file handler for the same file: that shared-file arrangement is not supported by the standard logging implementation. The Python logging cookbook describes a queue-based design with one writer. We did not implement or test that design here. For the separate question of scheduling CPU work, see Python multiprocessing.

Verify the behavior before running continuously

The local suite passed 16 checks on Python 3.12.14. It exercised severity thresholds, repeated configuration, exception context, optional fields, JSON parsing, Unicode and newline handling, a duplicate-handler failure and repair, and the deliberate limits of field selection. It also ran both scripts as separate processes and checked the failing job's exit status.

Those are application-level checks. They do not test log delivery during an outage, concurrent workers, disk rotation, collector performance, or a cloud deployment. Before leaving a service running, verify that a known test event reaches its final destination and that a full disk or unavailable collector has an acceptable effect on the application.

Once the application is diagnosable locally, choose how to operate it. Our API hosting comparison explains the responsibilities of a VM, managed platform, and serverless deployment. Compute with Hivenet provides CPU infrastructure for applications and workers; the vCPU guide explains instance configurations. On a customer-operated VM, plan your own application supervision, log collection, retention, and alerts.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.