Tracing a Python API with OpenTelemetry

The report always sounds the same: “the app is slow sometimes.” Not on every request, not for every user, and not reproducible on demand — which means print statements and a debugger attached to your local machine are useless, because the slow request already happened on a production replica you weren’t watching, and by the time you’re looking, the moment is gone. Logs help a little, if the request happened to touch a code path that logs something, but logs from a FastAPI handler, a Celery worker, and a database connection pool are three separate streams with no shared thread tying them back to the one request that was slow.

That’s the actual problem distributed tracing solves, and it’s narrower and more concrete than “observability” as a buzzword suggests. A trace is one request’s causal timeline, stitched together across every service and hop it touched, with real durations attached to each step. OpenTelemetry is the vendor-neutral standard for producing that timeline — instrument once, export anywhere (Jaeger, Tempo, Honeycomb, Datadog) — and the part that takes real engineering care isn’t installing the library, it’s making sure trace context survives every boundary a request crosses, especially the one between a synchronous API handler and an asynchronous background task.

You’ll learn:

• The concrete difference between a log line, a metric, and a trace span, and why you need all three, not just one
• How OpenTelemetry’s trace, span, and context model actually fits together
• How to auto-instrument a FastAPI app and get spans for every request with a few lines of setup
• Why trace context breaks by default at a Celery task boundary, and how to propagate it manually so a worker’s span nests correctly under its originating request
• How to pair structured logging with trace IDs so every log line is one click away from its full request timeline
• How sampling actually works, and why “trace everything” is a decision you’ll regret at scale

Table of Contents

1. The Basics
2. The Complete Architecture
3. Core Layers Explained
4. End-to-End Walkthrough
5. Special Cases
6. Scaling & Production Challenges
7. Code Examples
8. Common Pitfalls
9. Production Best Practices

The Basics

Logs, Metrics, and Traces Are Different Tools

A log is a discrete event with a message and some context, useful for “what happened” at a single point in time. A metric is an aggregated number over time — request count, p99 latency, error rate — useful for “is the system healthy right now” at a glance, but it throws away per-request detail. A trace is neither — it’s the connected timeline of one specific request as it moves through every service, function, and I/O call it touches, with a duration attached to each step. Metrics tell you that something is slow, traces tell you where, and structured logs correlated to that same trace tell you why at the specific step that was slow.

Why This Specifically Matters for Python APIs

• Async and background work fragment the timeline by default. A FastAPI handler that kicks off a Celery task, or awaits a downstream HTTP call, splits one logical request into multiple physical execution contexts — without explicit propagation, nothing connects them back together.
• “It’s slow sometimes” is unreproducible by construction. You can’t attach a debugger to a production replica after the fact. A trace captured at the time is the only artifact that still exists once the request is over.
• Multiple services means multiple log streams. Grepping several log files for a matching timestamp is a bad way to reconstruct one request’s path — a shared trace ID isn’t.

The Complete Architecture

 client
   │  traceparent: 00-<trace_id>-<span_id>-01
   ▼
┌─────────────┐   span: http.request        ┌───────────────┐
│  FastAPI      │──────────────────────────►│  span: db.query │
│  (root span)  │                            └───────────────┘
└──────┬───────┘
       │ task.delay() — trace context injected into task headers
       ▼
┌─────────────┐
│ Celery worker │  span: celery.task (child of the FastAPI root span)
└─────────────┘
       │
       ▼
   OTLP exporter → Collector → Jaeger / Tempo / vendor backend

The guiding principle: a trace is a tree of spans, every span has exactly one parent (except the root), and what makes the tree connected across process boundaries is a small piece of context — a trace ID and the current span ID — carried explicitly across every boundary that isn’t a plain in-process call. HTTP carries it in the traceparent header (the W3C Trace Context standard); a message queue like Celery has no such built-in header, so your code injects and extracts it manually.

Core Layers Explained

1. Traces, Spans, and Context

What it is: A span represents one unit of work — an HTTP request, a database query, a function call — with a name, a start and end time, attributes (key-value metadata), and a reference to its parent span. A trace is the full set of spans sharing one trace ID. Context is the small bundle (trace ID + current span ID + trace flags) propagated so a new span knows which trace and parent to attach to.

Why it matters: Nothing about “which trace does this span belong to” is automatic across a process or thread boundary — it’s carried explicitly, either in-process (via Python’s contextvars) or across the wire (the traceparent HTTP header, or manually injected headers for anything that isn’t HTTP).

Production tip: Don’t try to build this from scratch — OpenTelemetry’s Python SDK handles contextvars propagation automatically for anything using its instrumentation, including through async/await. The place you have to think about it explicitly is any boundary the SDK doesn’t already know about, which is exactly the Celery case below.

2. Auto-Instrumenting FastAPI

What it is: opentelemetry-instrumentation-fastapi wraps your app’s request handling so every incoming request automatically gets a root span, with attributes for the HTTP method, route, and status code — no manual span creation required for the basic case.

Why it matters: This gets you real request-level tracing for free — every request shows its actual duration and route, often enough alone to spot that /reports/export is the slow endpoint before you’ve written a single manual span.

Production tip: Auto-instrumentation covers the request/response boundary. It won’t automatically know that “step 3 inside this handler, which calls a third-party API” is worth its own span — for that, wrap the specific slow-looking code in a manual span so it shows up as its own named segment in the trace, not folded into the parent’s total duration.

3. Structured Logging Correlated to Trace IDs

What it is: Structured logging means emitting logs as key-value data (typically JSON) instead of freeform strings — logger.info("order created", order_id=123) instead of an interpolated sentence. Correlating it to tracing means injecting the active trace ID and span ID into every log record automatically.

Why it matters: A structured log with a trace ID attached lets you go from “this log line looks suspicious” to “show me the entire request this happened in” in one click, in any backend that links logs to traces (Grafana with Loki+Tempo, Datadog, Honeycomb). Without it, that log line is an isolated point with no path back to its context.

Production tip: Use structlog or python-json-logger with a processor that pulls the current span’s trace ID and span ID out of OpenTelemetry’s context on every log call — this is a few lines of setup that pays for itself the first time you’re debugging something that spans three services.

4. Propagating Spans Across a Celery Task Boundary

What it is: Celery tasks run in a completely separate process (the worker), receiving only whatever data was serialized into the task’s message — by default, just the function arguments, with no trace context attached at all. A task’s span, if you create one, starts as a brand-new, disconnected root trace by default, with no link back to the request that queued it.

Why it matters: This is the single most common gap in a Python service’s tracing setup — the API side looks fully instrumented, the worker side looks fully instrumented, yet the trace UI shows two unrelated traces instead of one connected timeline, because nothing carried the context across the queue.

Production tip: Inject the current trace context into the task’s headers when you call .delay()/.apply_async(), and extract it at the start of the task before starting its span — opentelemetry-instrumentation-celery handles a good chunk of this automatically via Celery’s signal hooks, but it’s worth understanding the manual injection/extraction shown below, since you’ll need the same pattern for any other non-HTTP boundary (Kafka, SQS, a raw Redis queue) without a library doing it for you. How well this propagation pattern fits also depends on which broker and worker model you’re running — see this comparison of Python task queue options for how Celery’s architecture compares to the alternatives here.

5. Exporters and the Collector

What it is: The SDK produces spans; an exporter ships them somewhere. You can export directly to a backend, or — the more common pattern — export via OTLP (OpenTelemetry Protocol) to a Collector, a separate process that batches, filters, and forwards to one or more real backends (Jaeger, Tempo, a vendor’s ingestion endpoint).

Why it matters: Exporting directly from every instance couples every service to that backend’s ingestion format and rate limits. Routing through a Collector decouples them — you can swap backends, add sampling rules, or fan out to two destinations, all in Collector config, without touching application code.

Production tip: Start with a local Collector even in development — one Docker container, and your export config (OTEL_EXPORTER_OTLP_ENDPOINT) never has to change between local, staging, and production; only the Collector’s own output config does.

End-to-End Walkthrough

Trace a single slow request from a POST /reports endpoint that queues a Celery task:

1. Client sends POST /reports. No traceparent header is present, so FastAPI’s instrumentation starts a brand-new trace and root span, generating a fresh trace ID.
2. Handler validates the request (a fast child span, a few milliseconds) and calls generate_report.delay(report_id).
3. Before the message is sent to the broker, the current trace context (trace ID + this span’s ID) is injected into the Celery message headers.
4. The handler returns 202 Accepted immediately — the root span for the HTTP request ends here, but the trace isn’t finished, since the queued task hasn’t run yet.
5. A Celery worker picks up the task, possibly seconds later on a different machine entirely. Before running the task function, it extracts the trace context from the message headers and starts a new span as a child of the original request’s span — same trace ID, correctly nested.
6. The task runs, including a database query that takes 4 of the task’s 4.3 total seconds — visible as its own child span with a duration that immediately identifies it as the bottleneck.
7. The trace backend renders one connected waterfall: the fast HTTP handler, then (after a queue gap you can also measure) the worker’s task span, then the slow query nested inside it — the exact hop that was slow, with a number attached, instead of a guess.

Special Cases

Async endpoint instrumentation. FastAPI’s async def handlers work correctly with auto-instrumentation because the SDK propagates context via contextvars, which asyncio respects across await boundaries within the same task. You do need special handling for asyncio.create_task() calls that spawn genuinely separate tasks, since a new task doesn’t inherit the parent’s context unless passed explicitly.

Background tasks via BackgroundTasks. FastAPI’s built-in BackgroundTasks run in-process after the response is sent, still within the same contextvars scope as the request — context propagates automatically here, unlike the Celery case, because there’s no process or serialization boundary involved.

Fire-and-forget tasks with no caller waiting. Some Celery tasks are queued by a cron-like scheduler, not a live request — there’s no meaningful “parent request” to link to. Starting a fresh root trace for these is correct; an artificial parent span just adds noise.

Third-party HTTP calls. opentelemetry-instrumentation-requests (or httpx) auto-instruments outbound calls and injects the traceparent header into the outbound request automatically, so if the downstream service is also instrumented, the trace continues across that boundary too.

Scaling & Production Challenges

Sampling: “trace everything” doesn’t survive real traffic. At meaningful volume, exporting a span for every request generates enormous data volume and cost for a lot of boring, fast, successful requests you’ll never look at. Head-based sampling (deciding at the trace’s start, e.g. 10% of requests) is simple but can miss the rare slow outlier you actually care about. Tail-based sampling (deciding after the trace completes, keeping anything slow or erroring regardless of sample rate) needs a Collector configured to buffer and decide, but it’s the pattern that actually keeps the traces you need.

Cardinality in span attributes. Attaching a raw user ID or a full request body as a span attribute feels convenient until your backend’s cost or index blows up from effectively unbounded unique values. Keep high-cardinality data in structured logs and reserve span attributes for a bounded set of dimensions you’ll actually filter or group by.

Clock skew across services. A trace’s waterfall view depends on every service’s clock being reasonably in sync — NTP drift can make a child span appear to start before its parent, which is confusing but not actually a tracing bug.

Collector as a new operational dependency. Once it’s in the telemetry path, its own availability and backpressure behavior become something to monitor — configure it to drop spans under overload rather than block or crash the application it’s collecting from.

Code Examples

Instrumenting a FastAPI app with automatic tracing and OTLP export:

# pip install opentelemetry-distro opentelemetry-exporter-otlp \
#             opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-celery

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

resource = Resource.create({"service.name": "reports-api"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

tracer = trace.get_tracer(__name__)


@app.post("/reports")
async def create_report(report_id: str):
    with tracer.start_as_current_span("validate-report-request"):
        pass  # validation logic here

    generate_report.delay(report_id)
    return {"status": "queued"}

Manually propagating trace context across the Celery boundary:

from celery import Celery
from celery.signals import before_task_publish, task_prerun
from opentelemetry import trace, context
from opentelemetry.propagate import inject, extract

celery_app = Celery("worker", broker="redis://localhost:6379/0")
tracer = trace.get_tracer(__name__)


@before_task_publish.connect
def inject_trace_context(headers=None, **kwargs):
    # Runs in the caller (the FastAPI process) — attach the current
    # trace context onto the outgoing task's message headers.
    if headers is None:
        return
    inject(headers)


@celery_app.task(name="generate_report")
def generate_report(report_id: str):
    # This task's own tracing setup extracts the injected context
    # from the task's request headers (wired via task_prerun in a
    # real setup, or automatically by opentelemetry-instrumentation-celery)
    # and starts this span as a child of the original request's span.
    with tracer.start_as_current_span("generate_report") as span:
        span.set_attribute("report.id", report_id)
        _run_slow_query(report_id)

Correlating structured logs to the active trace with structlog:

import structlog
from opentelemetry import trace


def add_trace_context(logger, method_name, event_dict):
    span = trace.get_current_span()
    ctx = span.get_span_context()
    if ctx.is_valid:
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict


structlog.configure(
    processors=[add_trace_context, structlog.processors.JSONRenderer()]
)

log = structlog.get_logger()
log.info("report_generated", report_id="r_123")
# {"event": "report_generated", "report_id": "r_123",
#  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", ...}

A minimal Collector config for local development:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlphttp/jaeger:
    endpoint: http://jaeger:4318

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/jaeger]

Common Pitfalls

Mistake: assuming Celery tasks inherit trace context automatically. Without explicit injection/extraction, every task starts a disconnected root trace, and the UI shows unrelated fragments instead of one timeline. Solution: use opentelemetry-instrumentation-celery or the manual inject/extract pattern above.

Mistake: putting high-cardinality data (raw user IDs, full payloads) into span attributes. This blows up ingestion cost and index size in most backends. Solution: keep that data in structured logs correlated by trace ID; keep span attributes to a bounded, filterable set.

Mistake: tracing every request at 100% sample rate in production. This is enormous, mostly boring data volume that costs real money. Solution: use tail-based sampling so slow and erroring requests are always kept, while routine fast ones are sampled down.

Mistake: logs and traces living in two disconnected systems with no shared ID. You end up manually correlating by eyeballing timestamps across two UIs. Solution: inject trace ID and span ID into every structured log line.

Mistake: instrumenting the API but never the worker (or vice versa). Half a trace is still a broken trace — you can see the request was slow but not why, or why but not what request it belonged to. Solution: instrument every process that touches the request’s lifecycle.

Production Best Practices

• Instrument the request/response boundary first with auto-instrumentation — the highest-value, lowest-effort step, and what most “why is this slow sometimes” investigations start from.
• Treat every non-HTTP boundary (queues, pub/sub, cron) as a manual propagation point until proven otherwise.
• Correlate structured logs to trace IDs from day one, not after the first painful cross-service debugging session makes the gap obvious.
• Use tail-based sampling once you’re past toy traffic volumes — full sampling doesn’t scale financially or practically.
• Route through a Collector rather than exporting directly from every service. It’s the seam that lets you change backends or sampling policy without redeploying application code.

Wrapping Up

A trace is only as useful as the boundaries it survives crossing, and in a typical Python service the boundary that breaks first is the one between a synchronous request handler and whatever asynchronous work it kicks off — a Celery task, a queue, a fire-and-forget call. Auto-instrumentation gets the easy 80% for free; the remaining 20%, propagating context across exactly those boundaries, is where “we have tracing” turns into “we can find the slow hop in a request nobody can reproduce.”

If your current setup has FastAPI instrumented and Celery running unconnected traces next to it, that gap — not a missing dashboard, not a missing exporter — is almost certainly why the trace UI hasn’t answered your last few “why was this slow” questions.