
Python logging records what an application did, which operation failed, and the context needed to investigate. Start with a named logger and an explicit severity threshold. Add structured fields when you need to find all events for one job or request. The useful outcome is a short trail you can follow during a failure.
This guide builds a small example using Python's standard library. It covers levels, handlers, JSON output, exceptions, and duplicate messages, with a deliberately failing batch job you can run locally. The examples were tested on Python 3.12.14.
Save the following as basic_example.py and run python basic_example.py. No extra package is required.
import logging
logger = logging.getLogger(__name__)
if __name__ == "__main__":
logging.basicConfig(
level=logging.INFO,
format="%(levelname)s %(name)s %(message)s",
)
logger.debug("Detailed input inspection")
logger.info("Processed %s rows", 3)
The observed output is INFO __main__ Processed 3 rows on standard error. The debug event is filtered out. A logger created with __name__ follows the module name when imported; a directly executed script uses __main__. Keep that identity so you can distinguish your worker from an HTTP client or database library.
Use print() for the result a command-line user requested, such as exported data. Use logging for operational events around that result. Python's logging HOWTO explains the distinction and the standard severity levels.
Choose a level by what happened to the operation. An expected validation rejection might deserve an informational event in one service; the same rejection could be an error in a nightly import that was supposed to succeed. Define that policy consistently across your application.
| Level | Example in a batch application | Operational question |
|---|---|---|
| DEBUG | A detailed checkpoint while diagnosing parsing | What happened inside this step? |
| INFO | A job started or completed | Is expected work progressing? |
| WARNING | A retry was needed, but the job can continue | Does this pattern need attention? |
| ERROR | A job failed and produced no usable result | Which operation needs recovery? |
| CRITICAL | The service cannot initialize a required component | Can this process keep serving its purpose? |
In the first example, INFO includes informational events and higher severities. A small job normally needs a start, a completion or failure, and enough context to connect them. Logging every row makes the important events harder to find and increases storage volume. Add detail for a specific investigation, then reduce it again.
The logger creates an event. A handler sends it to a destination. A formatter determines the representation. In this example, a root handler writes JSON to standard output, and the module logger sends its events there through propagation. Both thresholds are explicit. An independently configured child logger still has to pass the handler's threshold.
Save this as logging_demo.py. Run it separately from the first script with python logging_demo.py.
import json
import logging
from datetime import datetime, timezone
from logging.config import dictConfig
logger = logging.getLogger(__name__)
class JsonFormatter(logging.Formatter):
def format(self, record):
event = {
"time": datetime.fromtimestamp(
record.created, timezone.utc
).isoformat(timespec="milliseconds"),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
}
for key, expected_type in (("job_id", str), ("rows", int)):
value = getattr(record, key, None)
if type(value) is expected_type:
event[key] = value
if record.exc_info:
event["exception"] = self.formatException(record.exc_info)
return json.dumps(event, ensure_ascii=True)
def configure_logging(level="INFO"):
# Application startup only: this replaces logging configuration.
dictConfig({
"version": 1,
"disable_existing_loggers": False,
"formatters": {"json": {"()": JsonFormatter}},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout",
"formatter": "json",
"level": level,
}
},
"root": {"handlers": ["console"], "level": level},
})
def run_demo():
context = {"job_id": "job-17"}
logger.debug("Inspecting fixture")
logger.info("Job started", extra=context)
rows = ["12", "7", "invalid"]
try:
values = [int(row) for row in rows]
except ValueError:
logger.exception("Job rejected", extra=context)
return 1
logger.info("Job completed", extra={**context, "rows": len(values)})
return 0
if __name__ == "__main__":
configure_logging()
raise SystemExit(run_demo())
The example intentionally fails while parsing invalid. Our run produced two JSON lines: an INFO event with the message Job started, then an ERROR event with Job rejected, the same job_id, and a ValueError traceback. The process exited with status 1. The debug message and completion event were absent.
Each record contains a UTC timestamp, severity, logger name, and message. Optional job_id and rows fields are included only with the expected types. The formatter escapes embedded newlines, so the traceback stays inside one physical JSON line. A consumer can parse that line and display the traceback across multiple lines.
dictConfig expresses the handler and formatter connections in one place. Call this configuration at application startup, before work begins. Repeating it in our isolated fixture left one root handler, but it replaces logging configuration and is unsuitable as an import-time action in a reusable library. The setting disable_existing_loggers=False does not promise that third-party handlers will disappear or that their previous setup remains unchanged.
Inside an exception handler, logger.exception() records an error with traceback context. The demo uses it where the job is rejected and returns a failing exit status. Logging does not itself retry a job, undo partial writes, or tell a scheduler that the work failed. Choose those behaviors separately.
Assign responsibility for the failure event. If a helper logs an exception and re-raises it, and its caller logs the same exception again, two records may describe one failure even when the handlers are configured correctly. A useful arrangement is for low-level functions to raise errors and for the job or request boundary to record the outcome with its identifier.
For the fixture, job-17 connects the start and rejection. In a real service, choose a non-sensitive identifier that remains stable across retries. Keep the business result separate from diagnostic text: a successful log write does not establish that the data import succeeded.
The JSON formatter builds a small output dictionary instead of serializing every attribute on the log record. Our tests supplied synthetic API-key, password, and payload fields through extra; none appeared in the emitted JSON. Missing optional context and fields with the wrong types were also handled without losing the event.
That allowlist has a specific limit. The formatter still writes the message and exception text. A second test deliberately put synthetic secrets in those locations, and both remained visible. The allowed job_id could also contain sensitive text if the caller supplies it. This example is a field-selection pattern, not a general redaction system.
Keep credentials, authorization headers, raw request bodies, and personal content out of logging calls. Review error messages from libraries before forwarding them to a shared destination. Decide who can read the logs and how long they should remain available. Test those decisions using representative synthetic inputs before collecting production traffic.
A fixed text formatter containing %(job_id)s requires that field on each record it formats. We reproduced a formatting failure when it was missing. The JSON example avoids that dependency by checking optional fields individually. Do not reuse reserved record names such as message for custom context.
Our duplicate-message test attached a handler to the child logger while leaving the root handler in place. One call produced two records. Removing the child handler returned the output to one record. The logging reference documents propagation and why handlers at multiple points in the hierarchy can repeat an event.
| Symptom | First check | Useful correction |
|---|---|---|
| One call produces two identical records | Handlers on both the child and an ancestor | Choose one output path; usually keep the root handler |
| INFO events never appear | Effective logger level and handler level | Set both thresholds deliberately |
| Changing basicConfig appears to do nothing | Whether root handlers already exist | Inspect the application's existing configuration |
| Only records without context disappear | Required fields in the formatter | Provide defaults or make the fields optional |
| Output exists locally but not in the collector | The selected stream and collector configuration | Trace one known event through each stage |
By default, basicConfig() leaves an already configured root logger alone. Its force=True option removes and closes existing root handlers. Use that only when your application owns the configuration; it can disrupt a notebook, test runner, or hosting framework. For an intentionally independent logger with its own handler, disabling propagation can be appropriate. Doing that without a working destination can instead hide events.
Standard output is a practical starting point for this job because the surrounding process manager can capture it. Our JSON demo writes there explicitly; the earlier basic example uses standard error. If a command reserves standard output for machine-readable results, keep operational logs on standard error or another destination.
When running in a container, use the Docker logs guide to retrieve output and configure retention. If writing a local file, the standard library offers rotating file handlers. Rotation needs a size or time policy and a retention limit; it does not create a backup or an alerting service.
This example is single-process. Do not copy it into several workers and give each an independent file handler for the same file: that shared-file arrangement is not supported by the standard logging implementation. The Python logging cookbook describes a queue-based design with one writer. We did not implement or test that design here. For the separate question of scheduling CPU work, see Python multiprocessing.
The local suite passed 16 checks on Python 3.12.14. It exercised severity thresholds, repeated configuration, exception context, optional fields, JSON parsing, Unicode and newline handling, a duplicate-handler failure and repair, and the deliberate limits of field selection. It also ran both scripts as separate processes and checked the failing job's exit status.
Those are application-level checks. They do not test log delivery during an outage, concurrent workers, disk rotation, collector performance, or a cloud deployment. Before leaving a service running, verify that a known test event reaches its final destination and that a full disk or unavailable collector has an acceptable effect on the application.
Once the application is diagnosable locally, choose how to operate it. Our API hosting comparison explains the responsibilities of a VM, managed platform, and serverless deployment. Compute with Hivenet provides CPU infrastructure for applications and workers; the vCPU guide explains instance configurations. On a customer-operated VM, plan your own application supervision, log collection, retention, and alerts.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.