FastAPI BackgroundTasks vs Celery: Picking the Right One

BackgroundTasks looks like it solves the same problem Celery does — run some work after the response goes out, without making the user wait for it. It’s built into FastAPI, needs no extra infrastructure, and the code to use it is three lines. That simplicity is exactly why teams reach for it to send confirmation emails, kick off a report generation job, or write an audit log entry, and it works right up until a deploy rolls mid-request, or the process gets OOM-killed, or the pod autoscaler decides to terminate an instance — and the task that was “in the background” simply never happened, with no error, no retry, and no record that it was ever supposed to run.

That failure mode isn’t a BackgroundTasks bug. It’s BackgroundTasks doing exactly what it was designed to do — run a coroutine in the same process, after the response, with no persistence layer behind it. The actual mistake is using it for work that needed the guarantees only a real task queue provides. This post lays out what BackgroundTasks genuinely is, the specific conditions under which it silently drops work, what Celery adds to fix that, and a concrete framework for choosing between them instead of defaulting to whichever one you set up first.

You’ll learn:

• What BackgroundTasks actually does under the hood, and why it runs in the same process as the request
• The specific failure scenarios — deploys, crashes, autoscaling — where BackgroundTasks silently loses work
• What Celery adds structurally: a broker, persistence, retries, and independent worker processes
• A decision framework based on durability requirements, retry needs, and expected task duration
• How to combine both — BackgroundTasks for the truly fire-and-forget work, Celery for everything that needs a guarantee
• Working code for both, including a Celery retry policy and a FastAPI endpoint that dispatches to it

Table of Contents

1. The Basics
2. How BackgroundTasks Actually Works
3. Where BackgroundTasks Silently Drops Work
4. What Celery Adds
5. The Decision Framework
6. Hybrid Patterns
7. Code Examples
8. Lighter-Weight Alternatives to Celery
9. Common Pitfalls
10. Production Best Practices

The Basics

Two Different Tools Solving Two Different Problems

BackgroundTasks is a Starlette feature FastAPI exposes directly: a way to schedule a function to run after the HTTP response has already been sent, inside the same worker process that handled the request. Celery is a distributed task queue: a separate system with its own worker processes, a message broker (Redis or RabbitMQ, typically) sitting between your app and those workers, and a results/state backend that survives the app process being restarted.

The surface-level similarity — “run this later, don’t block the response” — hides a much bigger structural difference: BackgroundTasks has no persistence layer at all. If the process that scheduled the task dies before the task finishes, the task is gone. Celery’s broker persists the task the moment it’s enqueued, independently of whether your API process is even still running by the time a worker picks it up.

Why This Distinction Gets Missed

Most tutorials introduce BackgroundTasks with genuinely fire-and-forget examples — logging, a cache warm — where losing the occasional task is harmless. Teams then reuse the same pattern for things that aren’t harmless to lose: sending a password reset email, charging a payment, generating a document a user is waiting on. The API looks identical in both cases. The failure consequence does not.

How BackgroundTasks Actually Works

A BackgroundTasks object is injected into a route, functions are registered on it, and FastAPI runs them after the response is sent — but still inside the same async event loop, in the same worker process:

from fastapi import FastAPI, BackgroundTasks

app = FastAPI()

def send_confirmation_email(email: str):
    # runs after the response has already gone out
    email_client.send(email, "Thanks for signing up")

@app.post("/signup")
async def signup(email: str, background_tasks: BackgroundTasks):
    create_user(email)
    background_tasks.add_task(send_confirmation_email, email)
    return {"status": "created"}

The user gets their response the instant create_user finishes — they don’t wait on send_confirmation_email at all. That’s the entire value proposition, and it’s genuinely useful. But note what’s not happening: there’s no queue, no broker, no separate worker, and nothing tracking whether send_confirmation_email ever actually completed. It’s a scheduled coroutine call within the same process lifecycle as the request that triggered it — closer to asyncio.create_task with slightly nicer ergonomics than to anything resembling a job queue.

If send_confirmation_email is itself synchronous and blocking, it also runs inside the same threadpool-or-event-loop constraints covered in Why Your FastAPI Endpoint Blocks the Event Loop — a slow background task can still degrade the rest of the process’s throughput if it isn’t written to cooperate with the loop. A synchronous function registered with add_task runs in Starlette’s threadpool exactly like a def route would, while an async def function registered the same way runs directly on the event loop — so a background task that blocks without awaiting can stall in-flight requests, not just other background work, which is easy to miss precisely because the task looks decoupled from the request/response cycle at a glance.

Exception handling is similarly easy to misjudge. If a registered task raises, FastAPI doesn’t crash the response — it already went out — but by default the exception is only visible if your logging setup captures unhandled exceptions from background tasks specifically; a bare try/except around the route itself never sees it, because by the time the task runs, the route function has already returned.

Where BackgroundTasks Silently Drops Work

Every one of these is a normal, expected event in a production deployment — not an edge case:

Deploys. A rolling deploy sends SIGTERM to the old process. If a background task is mid-flight (or hasn’t started yet because it’s queued behind other work in the same event loop) when the process’s grace period expires and it gets SIGKILLed, that task is gone, permanently, with nothing logged about it.

Autoscaling scale-down. The same mechanism as a deploy — an orchestrator (Kubernetes, ECS) decides to terminate an instance under low load, and any BackgroundTasks work in flight on that instance disappears with it.

Process crashes and OOM kills. An unrelated memory leak or an unexpectedly large request elsewhere in the same process can get the whole worker OOM-killed, taking every pending background task down with it — even tasks that had nothing to do with whatever caused the crash.

No retry on failure. If send_confirmation_email raises an exception — the email provider times out, say — BackgroundTasks doesn’t retry it. By default the exception is swallowed (visible only if you’ve wired up exception logging on it), and the task is simply never completed. There’s no dead-letter queue, no backoff, no second attempt.

No cross-process visibility. If you run multiple Uvicorn/Gunicorn workers, a BackgroundTasks job scheduled on worker 2 has no relationship to worker 1 at all — you can’t inspect, retry, or cancel it from anywhere except the exact process that scheduled it, and that process’s own logs are the only record it existed.

None of this makes BackgroundTasks broken. It makes it a tool for work you can genuinely afford to lose occasionally — not a general-purpose job queue wearing a simpler API.

What Celery Adds

Celery’s architecture directly answers every gap above by inserting a durable, independent layer between “the request that triggered the work” and “the process that does the work”:

• A message broker (Redis/RabbitMQ) that persists the task. The moment .delay() is called, the task is serialized and written to the broker — independent of the FastAPI process’s lifecycle. If the API process dies a millisecond later, the task is already safe in the broker.
• Independent worker processes. Celery workers are separate processes (often separate containers or hosts entirely) that pull tasks off the broker. A deploy of your API doesn’t touch them; a deploy of your workers doesn’t touch your API.
• Built-in retries with backoff. A task can declare max_retries and a backoff policy, so a transient failure — a downstream API being briefly unavailable — gets retried automatically instead of silently dying.
• A results backend. Task state (pending, success, failure, and the return value) is queryable after the fact, from any process, which makes “did this actually complete” an answerable question instead of a log-grepping exercise.
• Scheduling and rate limiting. Periodic tasks (celery beat), rate limits per task type, and priority queues are first-class features, not something you’d have to hand-roll on top of BackgroundTasks.

The tradeoff is real infrastructure: a broker to run and monitor, worker processes to deploy and scale independently, and a genuinely more complex mental model — task serialization, idempotency, and broker connection handling are all now your problem in a way they weren’t with BackgroundTasks.

The Decision Framework

Three questions settle almost every case:

1. Can you afford to lose this task entirely, silently, with no retry? If yes — a metrics ping, a best-effort cache warm, an analytics event — BackgroundTasks is fine and adding Celery would be pure overhead. If no — anything involving money, anything a user is explicitly waiting on the result of, anything with a compliance or audit requirement — you need Celery’s durability.

2. Does the task need to survive a deploy or scale-down event? BackgroundTasks work only survives if the process that scheduled it stays alive long enough to finish it. If your deploy cadence is frequent (multiple times a day, common with CI/CD) and tasks can run longer than your graceful-shutdown window, BackgroundTasks will lose work on a predictable cadence, not a rare one.

3. Does the task need a retry policy, scheduling, or cross-process visibility? If a failure needs an automatic retry, if the task needs to run on a schedule independent of any request, or if you need to check task status from a different process (an admin dashboard, a separate API), that’s specifically what Celery’s broker and results backend are for — BackgroundTasks has no mechanism for any of the three.

Hybrid Patterns

Most mature FastAPI codebases use both, deliberately, rather than picking one tool for the whole app. A common and effective split: use BackgroundTasks for work that’s cheap to redo or genuinely disposable, and dispatch to Celery for anything with a durability or retry requirement — often from the very same endpoint.

@app.post("/orders")
async def create_order(payload: OrderCreate, background_tasks: BackgroundTasks):
    order = await create_order_record(payload)

    # Disposable — fine to lose occasionally, no retry needed
    background_tasks.add_task(log_analytics_event, "order_created", order.id)

    # Durable — must survive a deploy, needs a retry policy
    send_order_confirmation.delay(order.id)

    return order

The rule of thumb: if you’d be upset to discover in a post-mortem that a task silently never ran, it belongs in Celery. If you’d shrug it off, BackgroundTasks is the right amount of tooling.

Code Examples

A minimal Celery setup that mirrors the framework above — retries with exponential backoff for a genuinely important task:

# celery_app.py
from celery import Celery

celery_app = Celery(
    "worker",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1",
)

@celery_app.task(
    bind=True,
    max_retries=5,
    default_retry_delay=10,  # seconds, before backoff kicks in
)
def send_order_confirmation(self, order_id: int):
    try:
        order = fetch_order(order_id)
        email_client.send(order.customer_email, render_receipt(order))
    except EmailProviderTimeout as exc:
        # exponential-ish backoff: 10s, 20s, 40s...
        raise self.retry(exc=exc, countdown=10 * (2 ** self.request.retries))

# app/orders/router.py
from app.workers.celery_app import send_order_confirmation

@router.post("/")
async def create_order_endpoint(payload: OrderCreate):
    order = await create_order(payload)
    send_order_confirmation.delay(order.id)
    return order

.delay(order.id) returns immediately — it just enqueues the task on the broker — so the endpoint’s response time is unaffected, while the actual guarantee of completion now lives in Celery’s broker and retry policy rather than in the FastAPI process. This kind of module boundary — a workers/ package the router dispatches into without owning any of its internals — fits naturally into the domain-driven layout covered in FastAPI Project Structure That Survives Growth.

Lighter-Weight Alternatives to Celery

Celery isn’t the only durable option, and for an async-first FastAPI codebase, it’s often not the most natural fit — Celery’s worker model predates asyncio and treats async tasks as a bolted-on feature rather than a first-class design.

arq is a Redis-backed queue built specifically for asyncio from the ground up. Task functions are async def, workers run their own event loop, and there’s no synchronous-to-async bridging to think about:

# worker.py
async def send_order_confirmation(ctx, order_id: int):
    order = await fetch_order(order_id)
    await email_client.send_async(order.customer_email, render_receipt(order))

class WorkerSettings:
    functions = [send_order_confirmation]
    redis_settings = RedisSettings(host="localhost")

# dispatching from FastAPI
redis = await create_pool(RedisSettings(host="localhost"))
await redis.enqueue_job("send_order_confirmation", order.id)

Dramatiq and RQ sit in a similar space — simpler setup than Celery, Redis as the broker, and a smaller feature surface (no celery beat-equivalent scheduling in RQ’s core, for instance) in exchange for less operational overhead.

The decision framework above still applies regardless of which durable queue you pick — the question was never specifically “Celery or nothing,” it’s “does this task need broker-backed durability at all.” Celery remains the right default when you need its ecosystem — mature scheduling, routing, and monitoring tooling that the newer alternatives are still catching up on — but for a greenfield async FastAPI service, arq is worth evaluating first specifically because it avoids mixing a sync-first task runner into an otherwise async-first codebase.

Common Pitfalls

Mistake: using BackgroundTasks for anything involving payment or compliance. A silently dropped charge confirmation or audit log entry is a business problem, not a technical inconvenience. Solution: anything with a legal, financial, or audit requirement goes through Celery, no exceptions.

Mistake: assuming Celery tasks are automatically idempotent. A retried task runs the entire function again — if send_order_confirmation isn’t safe to run twice, a retry after a partial failure can double-send. Solution: design tasks to be idempotent (check-before-act, or use idempotency keys) whenever retries are enabled.

Mistake: putting genuinely long-running work in BackgroundTasks. A task that runs for minutes inside the same process as your API workers competes for the same resources as request handling. Solution: anything beyond a few seconds belongs in a separate worker pool, which is exactly what Celery provides.

Mistake: standing up Celery for a single, low-stakes task. The operational cost of a broker and worker fleet isn’t justified by one best-effort analytics call. Solution: default to BackgroundTasks until a concrete durability or retry requirement shows up.

Mistake: not monitoring Celery worker health. A crashed or backed-up worker fleet silently accumulates a growing backlog with no user-facing symptom until it’s severe. Solution: monitor queue depth and worker liveness (Flower, or your APM’s Celery integration) as a first-class production metric.

Production Best Practices

• Default to BackgroundTasks and upgrade deliberately. Don’t stand up Celery infrastructure before a concrete task actually needs the guarantees it provides.
• Make Celery tasks idempotent by default. Assume every task will be retried at least once, because it eventually will be.
• Set explicit retry limits and backoff, never unlimited retries. A task that retries forever against a permanently broken dependency just becomes a different kind of incident.
• Log task dispatch and completion separately. Knowing a task was scheduled is not the same as knowing it succeeded — instrument both.
• Treat queue depth as an alertable metric. A growing, unprocessed Celery backlog is one of the earliest signals of a downstream outage.

Wrapping Up

BackgroundTasks and Celery aren’t competing solutions to the same problem — they’re the right tool for two genuinely different durability requirements, and the mistake is picking based on setup effort instead of what happens when a deploy lands mid-task. If losing the work silently would be a real problem, that’s your answer regardless of how much simpler BackgroundTasks looks today. Most production FastAPI apps end up using both, on purpose, once they’ve been burned by exactly one dropped task too many.

Do you actually know what happens to your in-flight background tasks the next time you deploy?