Article

Structured logging with Python and Django: from log soup to useful events

by Gary Worthington, More Than Monkeys

If you have ever SSH’d onto a box and grepped through a log file at 2 a.m., you already know why logging matters.

What most teams do not realise is that string logs are quietly taxing everything downstream: observability, incident response, audit, even product analytics. You can only get so far with “search for this substring and hope”.

Structured logging fixes that by turning logs into machine readable events instead of vaguely formatted paragraphs.

This post walks through how to do proper structured logging in Python, using Django as the concrete example. It assumes you already know basic Python logging and how Django settings work. The aim is to help you move your team from “log soup” to “queryable events” without rewriting the whole app.

What do we mean by “structured logging”?

Unstructured log:

[2025-11-15 10:01:03] INFO views.checkout: Order 1234 completed for user 42, total=19.99 GBP, provider=stripe

It is readable, but all the interesting bits are inside an English sentence. Your log system has to guess what is important.

Structured log (conceptually a JSON object):

{
  "timestamp": "2025-11-15T10:01:03.123Z",
  "level": "INFO",
  "logger": "shop.views.checkout",
  "event": "checkout_completed",
  "order_id": 1234,
  "user_id": 42,
  "basket_total": 19.99,
  "currency": "GBP",
  "payment_provider": "stripe",
  "request_id": "a1b2c3",
  "path": "/checkout/complete",
  "method": "POST"
}

In practice, your logging stack will output this on a single line, but thinking of it as a JSON object is what matters.

Now your log backend can answer questions such as:

  • “Show me error rates by payment provider.”
  • “Find all failed checkouts for user 42 in the last hour.”
  • “Graph checkout_success / checkout_failed per minute, grouped by tenant_id.”

The key idea is:

Log events with fields, not paragraphs.

In Python terms, that means you want your logging output to be structured (usually JSON), and you want a disciplined pattern for how you log events throughout the codebase.

Core ingredients

For a Django app, a solid structured logging setup usually has:

JSON output
Logs are serialised as JSON so tools like CloudWatch, Loki, Elastic, Datadog or whatever you use can parse fields.

Consistent event shape
Each log line has a clear event name and a small, predictable set of fields. Developers should be able to guess them.

Context propagation
Things like request_id, user_id, tenant_id, environment get attached automatically for every log line in a request.

Minimal friction for developers
Logging an event should feel like calling a normal logger, not wrestling a small framework.

We will do all of this with the standard logging module plus a JSON formatter. No magic.

Step 1: Add a JSON formatter in plain Python

I will use python-json-logger because it is lightweight and plays nicely with the standard library.

Install it:

pip install python-json-logger

Basic example:

from __future__ import annotations

import logging
from typing import Any, Dict
from pythonjsonlogger import jsonlogger

def configure_root_logger() -> None:
    """
    Configure the root logger to output JSON to stdout.
    This is a simple baseline that you can reuse in Django or scripts.
    """
    handler = logging.StreamHandler()
    formatter = jsonlogger.JsonFormatter(
        "%(asctime)s %(levelname)s %(name)s %(message)s "
        "%(request_id)s %(user_id)s %(tenant_id)s"
    )
    handler.setFormatter(formatter)
    root_logger = logging.getLogger()
    root_logger.setLevel(logging.INFO)
    root_logger.handlers.clear()
    root_logger.addHandler(handler)

def example_usage() -> None:
    """
    Emit a structured log line using the configured root logger.
    """
    logger = logging.getLogger("example.checkout")
    payload: Dict[str, Any] = {
        "order_id": 1234,
        "basket_total": 19.99,
        "currency": "GBP",
        "payment_provider": "stripe",
    }
    logger.info("checkout_completed", extra=payload)

if __name__ == "__main__":
    configure_root_logger()
    example_usage()

Example output (pretty printed here for clarity, but usually emitted on a single line):

{
  "asctime": "2025-11-15 10:01:03,456",
  "levelname": "INFO",
  "name": "example.checkout",
  "message": "checkout_completed",
  "request_id": null,
  "user_id": null,
  "tenant_id": null,
  "order_id": 1234,
  "basket_total": 19.99,
  "currency": "GBP",
  "payment_provider": "stripe"
}

The exact timestamp will differ, but the important bit is that:

  • Each field is a separate JSON key.
  • Your domain data (order_id, basket_total, currency, payment_provider) is first class, not buried in a string.
  • Context fields like request_id, user_id, tenant_id are present and ready to be wired up properly in Django in the later steps.

Step 2: Wire JSON logging into Django

Django uses the standard logging module under the hood. You configure it via the LOGGING setting in settings.py.

Here is a minimal JSON logging setup for Django:

# settings.py
from __future__ import annotations

import logging
from typing import Any, Dict
from pythonjsonlogger import jsonlogger
LOG_LEVEL: str = "INFO"

class RequestContextFilter(logging.Filter):
    """
    Attach request-scoped context to log records.
    This version is a placeholder. We will wire it up with real context in the next section.
    """
    def filter(self, record: logging.LogRecord) -> bool:
        """
        Modify the LogRecord in place, then return True so the record is logged.
        """
        # Defaults so the formatter always has these fields.
        if not hasattr(record, "request_id"):
            record.request_id = None
        if not hasattr(record, "user_id"):
            record.user_id = None
        if not hasattr(record, "tenant_id"):
            record.tenant_id = None
        return True

LOGGING: Dict[str, Any] = {
    "version": 1,
    "disable_existing_loggers": False,
    "filters": {
        "request_context": {
            "()": "path.to.settings.RequestContextFilter",
        },
    },
    "formatters": {
        "json": {
            "()": jsonlogger.JsonFormatter,
            "fmt": (
                "%(asctime)s %(levelname)s %(name)s %(message)s "
                "%(request_id)s %(user_id)s %(tenant_id)s %(path)s %(method)s"
            ),
        },
    },
    "handlers": {
        "console_json": {
            "class": "logging.StreamHandler",
            "filters": ["request_context"],
            "formatter": "json",
        },
    },
    "root": {
        "handlers": ["console_json"],
        "level": LOG_LEVEL,
    },
    "loggers": {
        "django": {
            "handlers": ["console_json"],
            "level": LOG_LEVEL,
            "propagate": False,
        },
        "django.request": {
            "handlers": ["console_json"],
            "level": LOG_LEVEL,
            "propagate": False,
        },
        "myapp": {
            "handlers": ["console_json"],
            "level": LOG_LEVEL,
            "propagate": False,
        },
    },
}

Test it quickly in a Django shell:

import logging

logger = logging.getLogger("myapp.checkout")
logger.info(
    "checkout_completed",
    extra={"order_id": 1234, "basket_total": 19.99, "currency": "GBP"},
)

Example output:

{
  "asctime": "2025-11-15 10:05:12,789",
  "levelname": "INFO",
  "name": "myapp.checkout",
  "message": "checkout_completed",
  "request_id": null,
  "user_id": null,
  "tenant_id": null,
  "path": null,
  "method": null,
  "order_id": 1234,
  "basket_total": 19.99,
  "currency": "GBP"
}

You still need request context (request_id, user_id, and so on). That is where middleware and ContextVars come in.

Step 3: Add request scoped context

You want every log line during a request to carry the same request_id, and ideally user_id and tenant_id if you have multi tenancy.

3.1 Context storage

Use contextvars so your logs work correctly even when you introduce async views or background tasks.

Create a small module, for example myapp/logging_context.py:

# myapp/logging_context.py
from __future__ import annotations

from contextvars import ContextVar
from typing import Any, Dict, Optional

_request_id: ContextVar[Optional[str]] = ContextVar("request_id", default=None)
_user_id: ContextVar[Optional[str]] = ContextVar("user_id", default=None)
_tenant_id: ContextVar[Optional[str]] = ContextVar("tenant_id", default=None)

def set_request_context(
    request_id: str,
    user_id: Optional[str],
    tenant_id: Optional[str],
) -> None:
    """
    Store per-request context in ContextVars.
    This should be called once at the beginning of each request.
    """
    _request_id.set(request_id)
    _user_id.set(user_id)
    _tenant_id.set(tenant_id)

def clear_request_context() -> None:
    """
    Clear per-request context after the request is finished.
    """
    _request_id.set(None)
    _user_id.set(None)
    _tenant_id.set(None)

def get_request_context() -> Dict[str, Optional[str]]:
    """
    Retrieve the current request context as a dict.
    Suitable for attaching to log records.
    """
    return {
        "request_id": _request_id.get(),
        "user_id": _user_id.get(),
        "tenant_id": _tenant_id.get(),
    }

3.2 Middleware to populate context

Now add a middleware that sets this up for each request.

# myapp/middleware.py
from __future__ import annotations

import typing as t
import uuid
from django.http import HttpRequest, HttpResponse
from django.utils.deprecation import MiddlewareMixin
from .logging_context import clear_request_context, set_request_context

class RequestContextMiddleware(MiddlewareMixin):
    """
    Django middleware that initialises logging context for each request.
    It generates a request_id, captures the authenticated user (if any),
    and optionally attaches a tenant_id from your own logic.
    """

    def process_request(self, request: HttpRequest) -> None:
        """
        Called on each request before the view.
        Generates a request_id and stores user / tenant context.
        """
        request_id: str = str(uuid.uuid4())
        user_id: t.Optional[str] = None
        tenant_id: t.Optional[str] = None
        if request.user.is_authenticated:
            user_id = str(request.user.pk)
        # Example: derive tenant from a header or subdomain.
        # Replace with your actual multi-tenant logic.
        host: str = request.get_host()
        if host.startswith("acme."):
            tenant_id = "acme"
        elif host.startswith("contoso."):
            tenant_id = "contoso"
        set_request_context(request_id, user_id, tenant_id)
        # Sometimes it is useful to expose request_id back to the client.
        request.request_id = request_id

    def process_response(
        self,
        request: HttpRequest,
        response: HttpResponse,
    ) -> HttpResponse:
        """
        Attach request_id to the response (for clients) and clear context.
        """
        request_id: str = getattr(request, "request_id", "")
        if request_id:
            response["X-Request-ID"] = request_id
        clear_request_context()
        return response

Add the middleware fairly high up in MIDDLEWARE so other middleware and your views can rely on the context:

MIDDLEWARE = [
    "myapp.middleware.RequestContextMiddleware",
    # ...
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    # etc
]

Once this is in place, any log line inside a request will pick up the context. For example:

logger.info("some_event")

might end up as:

{
  "asctime": "2025-11-15 10:10:41,123",
  "levelname": "INFO",
  "name": "myapp.some_view",
  "message": "some_event",
  "request_id": "1e3f7d2c-9d8a-4c5b-9aec-752b7ef2c0a3",
  "user_id": "42",
  "tenant_id": "acme",
  "path": "/some/path",
  "method": "GET"
}

3.3 Update the logging filter to use the context

Finally, update the RequestContextFilter in settings.py so it pulls from logging_context:

# settings.py
from myapp.logging_context import get_request_context

class RequestContextFilter(logging.Filter):
    """
    Attach request-scoped context (request_id, user_id, tenant_id) to log records.
    """
    def filter(self, record: logging.LogRecord) -> bool:
        """
        Modify the LogRecord in place, then return True so the record is logged.
        """
        ctx = get_request_context()
        record.request_id = ctx.get("request_id")
        record.user_id = ctx.get("user_id")
        record.tenant_id = ctx.get("tenant_id")
        return True

You now have:

  • JSON logs.
  • Every log line during a request tagged with request_id, user_id, and tenant_id.

At this point you can already do things like:

  • request_id:1e3f7d2c-9d8a-4c5b-9aec-752b7ef2c0a3 to see the full trace of one request.
  • Filters by tenant_id:acme to understand noisy tenants.

Step 4: Logging application events, not just errors

Developers tend to treat logging as something you do “when things go wrong”. That is not wrong, but it is incomplete.

Structured logging is most powerful when you log key domain events in a consistent way.

For example, a checkout view:

# myapp/views.py
from __future__ import annotations

import logging
from typing import Any, Dict
from django.http import HttpRequest, HttpResponse, JsonResponse
from django.views.decorators.http import require_POST
from .models import Order
from .services.checkout import complete_checkout
logger = logging.getLogger(__name__)

@require_POST
def checkout_complete(request: HttpRequest) -> HttpResponse:
    """
    Handle completion of a checkout and log a structured event.
    On success, return HTTP 200 with order details. On failure, raise an error
    that will be logged with full context by Django's exception handling.
    """
    order: Order = complete_checkout(request)
    event_payload: Dict[str, Any] = {
        "event": "checkout_completed",
        "order_id": order.pk,
        "basket_total": float(order.total),
        "currency": order.currency,
        "payment_provider": order.payment_provider,
    }
    logger.info("checkout_completed", extra=event_payload)
    return JsonResponse(
        {
            "order_id": order.pk,
            "request_id": getattr(request, "request_id", None),
        }
    )

Example log output:

{
  "asctime": "2025-11-15 10:15:30,654",
  "levelname": "INFO",
  "name": "myapp.views",
  "message": "checkout_completed",
  "request_id": "3c8a1b2d-1f94-4a33-9a9d-16b54b5d0f12",
  "user_id": "42",
  "tenant_id": "acme",
  "path": "/checkout/complete",
  "method": "POST",
  "event": "checkout_completed",
  "order_id": 5678,
  "basket_total": 49.95,
  "currency": "GBP",
  "payment_provider": "stripe"
}

Example HTTP response shown to the client:

{
  "order_id": 5678,
  "request_id": "3c8a1b2d-1f94-4a33-9a9d-16b54b5d0f12"
}

In a log backend, you can then graph event:checkout_completed by tenant_id and payment_provider without touching the code again.

Step 5: Ship logs to something sensible

The details depend on your infrastructure, but the main principles are the same:

Write logs to stdout or stderr in the container or process.
Docker, Kubernetes, ECS and similar environments assume this.

Use JSON as the single format on the wire.
Once the log pipeline expects JSON, everyone’s life is simpler.

Do parsing and indexing in the log system, not in the app.
Django should not care whether you use CloudWatch, Loki, Elastic or a home grown stack.

A typical production log line might end up looking like this (pretty printed for readability):

{
  "timestamp": "2025-11-15T10:20:01.001Z",
  "level": "INFO",
  "service": "checkout-api",
  "environment": "production",
  "logger": "myapp.views",
  "message": "checkout_completed",
  "request_id": "3c8a1b2d-1f94-4a33-9a9d-16b54b5d0f12",
  "user_id": "42",
  "tenant_id": "acme",
  "path": "/checkout/complete",
  "method": "POST",
  "event": "checkout_completed",
  "order_id": 5678,
  "basket_total": 49.95,
  "currency": "GBP",
  "payment_provider": "stripe",
  "region": "eu-west-2"
}

In reality, this will be emitted as a single JSON line, which is what log collectors expect.

Common mistakes and how to avoid them

A few things I see repeatedly when teams bolt structured logging onto existing systems.

1. Treating structured logging as “just add JSON”

If you simply take existing log messages and wrap them in JSON, you get the worst of both worlds. Verbose text and no consistent fields.

Instead, do this:

  • Establish the set of common fields you want on all logs (request_id, user_id, tenant_id, environment, service, maybe version).
  • Decide on a short list of domain events per bounded context (user_registered, checkout_completed, subscription_cancelled).
  • Encourage developers to log those events with small, typed payloads.

2. Dumping entire objects into logs

This is how you accidentally log passwords and card numbers.

Bad:

logger.info("request_body", extra={"body": request.body}

Example of what you do not want:

{
  "message": "request_body",
  "body": "cardNumber=4111111111111111&expiry=12/25&cvv=123"
}

Better:

logger.info(
    "checkout_request_received",
    extra={
        "basket_total": float(basket.total),
        "item_count": basket.item_count,
    },
)

Example output:

{
  "message": "checkout_request_received",
  "basket_total": 49.95,
  "item_count": 3
}

You control the fields. You avoid PII. Compliance folks sleep better.

3. Logging at the wrong level

If everything is INFO, you cannot filter quickly. If everything is ERROR, your alerts are useless.

A simple scheme that works well in most Django apps:

  • DEBUG: noisy internal details, only in dev and test.
  • INFO: normal behaviour and domain events, always safe to keep.
  • WARNING: unusual, possibly problematic behaviour that is still handled gracefully.
  • ERROR: actual failures where the user saw an error or a fallback.
  • CRITICAL: system is basically unusable. You probably want a pager for this.

Example:

logger.warning(
    "payment_provider_slow",
    extra={"provider": "stripe", "elapsed_ms": 1250},
)

Example output:

{
  "asctime": "2025-11-15 10:22:44,210",
  "levelname": "WARNING",
  "name": "myapp.payments",
  "message": "payment_provider_slow",
  "request_id": "a9a3fdbe-7e2a-4b64-a05e-4d850e4a8d0b",
  "user_id": "42",
  "tenant_id": "acme",
  "provider": "stripe",
  "elapsed_ms": 1250
}

In Django specifically, unhandled exceptions already get logged at ERROR by django.request. Make sure your handlers are wired to capture those.

4. Letting every developer invent their own log shape

If one view logs checkout_complete and another logs checkoutCompleted and a third logs CHECKOUT_COMPLETED, your queries will be a mess.

Treat event names as part of your public API:

  • Lowercase with underscores is fine: checkout_completed.
  • Document them for the team.
  • Review them in pull requests like you review URL structure or database schema.

How to roll this into an existing project

You probably have a live Django app with a fair number of logs already. You do not need a big bang migration.

A practical order of attack:

Introduce JSON output and basic context

  • Add python-json-logger and the JSON formatter in LOGGING.
  • Add the request context middleware and filter.
    This is mostly infrastructure and configuration.

Before:

logger.info("checkout completed for order %s", order.pk)

After:

logger.info(     
  "checkout_completed",     
  extra={"event": "checkout_completed", "order_id": order.pk}, 
)

Upgrade one area of the app at a time
Pick a small, high value slice. For example, your checkout flow or login flow. Replace free form log lines with structured events and payloads. Confirm with your log backend that fields show up as expected.

Set some conventions
Write a short page in your internal docs:

  • What fields are always present.
  • Event naming rules.
  • Examples of good and bad logs, with realistic output.

Bake it into code review
When someone touches code that logs, reviewers should ask:

  • Are we logging a structured event?
  • Are we sending any sensitive data?
  • Will this be useful when debugging incidents or analysing behaviour?

Measure something meaningful with it
Use the new structured logs to build a small dashboard:

  • Checkout success rate per hour.
  • Login failure rate by tenant.
  • Error count per endpoint.

Once the team sees real insight, you will not have to convince anyone to use structured logging again.

Final thoughts

Structured logging is not glamorous, but it is one of those engineering practices that quietly upgrades your entire stack.

You get:

  • Faster debugging, because you can slice and dice by context rather than reading walls of text.
  • Better observability, because logs and metrics line up around shared fields such as request_id and tenant_id.
  • Easier compliance and audit, because your evidence is machine readable rather than whatever someone happened to type.

Most importantly, it is achievable with the tools you already have: Python’s logging module, Django’s LOGGING setting, and a bit of discipline.

Start small. Pick one part of the system, wire up structured logs properly, and prove the value. After you have found the error that used to take an hour in about thirty seconds, structured logging tends to sell itself.

Gary Worthington is a software engineer, delivery consultant, and fractional CTO who helps teams move fast, learn faster, and scale when it matters. He writes about modern engineering, product thinking, and helping teams ship things that matter.

Through his consultancy, More Than Monkeys, Gary helps startups and scaleups improve how they build software, from tech strategy and agile delivery to product validation and team development.

Visit morethanmonkeys.co.uk to learn how we can help you build better, faster.

Follow Gary on LinkedIn for practical insights into engineering leadership, agile delivery, and team performance.