Making Celery Tasks Idempotent

The support ticket usually reads the same way: “customer says they were charged twice for one order.” Someone pulls the logs, finds a single API call, a single charge_customer.delay(order_id) — and then finds the task ran twice anyway, twelve seconds apart, on two different workers. Nobody wrote a bug that duplicated the call. The system worked exactly as designed. That’s the part that trips people up: Celery task running twice isn’t a malfunction, it’s the documented, intended behavior of at-least-once delivery, and it will happen to every Celery deployment that runs long enough.

The fix isn’t finding and patching the “race condition” — there usually isn’t one to find, in the sense of a bug you can point at. The fix is accepting that any task can execute more than once for the same logical unit of work, and designing the task itself so that running it twice produces the same end state as running it once. That’s what idempotency actually means here: not “this never happens,” but “it’s safe when it does.”

You’ll learn:

• Why Celery task running twice is expected behavior, not a bug, once you understand acks_late and broker redelivery
• The real-world causes of celery duplicate execution — worker crashes, visibility timeouts, network partitions, and retries
• How to design idempotency keys that make duplicate task execution a no-op
• How to implement a distributed lock with Redis SET NX EX to prevent overlapping execution
• Database-level idempotency using unique constraints as a second line of defense
• How to handle inherently non-idempotent side effects like sending an email or calling a payment API
• How to actually test that a task is idempotent, not just assume it

Table of Contents

1. Why At-Least-Once Delivery Means Duplicate Execution
2. The Real Causes of Duplicate Execution
3. Idempotency Keys: The Core Pattern
4. Distributed Locks with SET NX EX
5. Database-Level Idempotency as a Second Line of Defense
6. Handling Non-Idempotent Side Effects
7. Testing for Idempotency
8. Common Pitfalls

Why At-Least-Once Delivery Means Duplicate Execution

Celery, like every mainstream Python task queue, defaults to at-least-once delivery when configured for durability. That phrase does a lot of work and most people skim past it: at-least-once means the system guarantees a task will be attempted one or more times — never zero — but explicitly does not guarantee exactly one execution. “At least” includes “more than.”

The mechanism is task_acks_late. With it disabled (Celery’s historical default), a worker acknowledges a message the moment it starts the task, before the function body has run. If the worker process dies mid-execution — OOM kill, deploy, hard crash — the message is already acknowledged and gone. The task silently never completes and never retries. That’s a data-loss risk, not a duplication risk.

Set task_acks_late = True and the trade flips: the broker only removes the message once the task returns successfully. If the worker dies after finishing the actual work but before that acknowledgment reaches the broker — network blip, process killed in the half-second window after the return statement — the broker has no way to know the work already happened. It redelivers the message to another worker. The side effect (the charge, the email, the row insert) happens again.

# celeryconfig.py
task_acks_late = True
task_reject_on_worker_lost = True
worker_prefetch_multiplier = 1

This is the correct default for anything where losing a task silently is worse than running it twice — which is most production work. But it means duplicate execution isn’t a corner case you might hit; it’s a scenario you’re explicitly opting into every time you choose “don’t lose tasks” over “never repeat tasks.” You cannot have both without additional application-level work, because no message broker can atomically couple “task finished” with “your side effect committed” across a network boundary.

The Real Causes of Duplicate Execution

Knowing the mechanism helps, but it helps more to know what actually triggers it in a live system, because “this basically never happens” is a common and wrong assumption:

• Worker crashes mid-task. OOM kills, SIGKILL from an orchestrator during a deploy, or a hard process crash after the side effect but before the ack.
• Broker visibility timeout expiring. With Redis as the broker, an in-flight (unacked) message becomes eligible for redelivery after visibility_timeout seconds, regardless of whether the original worker is still working on it. A task that legitimately takes longer than the configured timeout gets redelivered to a second worker while the first is still running it.
• Manual retries plus a partial success. A task raises an exception after its side effect already committed (e.g., the database write succeeded but a subsequent notification call timed out and raised), and autoretry_for retries the whole function — repeating the part that already succeeded, which is exactly why retry backoff and dead-letter queue design deserves the same deliberate treatment as the idempotency itself.
• Network partitions between worker and broker. The ack is sent but never received due to a transient network issue; from the broker’s perspective, that’s indistinguishable from a worker that died.
• Operator error. Someone manually replays a message from a dead-letter queue without checking whether it had already partially succeeded before failing.

None of these require a bug in your code. They’re properties of running distributed workers against a network you don’t fully control. Redis-backed Celery deployments hit the visibility timeout scenario often enough that it deserves its own dedicated treatment — see the section on visibility timeout redelivery for the mechanics of how a task can get picked up twice purely from a timeout being set shorter than the task actually runs.

Idempotency Keys: The Core Pattern

The standard fix is the same one payment APIs like Stripe popularized: attach a unique idempotency key to each logical unit of work, and before doing anything with side effects, check whether that key has already been processed.

import redis
from celery import shared_task

redis_client = redis.Redis(host="localhost", port=6379, db=2)

@shared_task(bind=True)
def charge_customer(self, idempotency_key: str, order_id: int, amount_cents: int):
    already_done = redis_client.get(f"idempotency:{idempotency_key}")
    if already_done:
        # Task already completed successfully — this is a redelivery, not new work.
        return already_done.decode()

    charge_id = payment_gateway.charge(order_id=order_id, amount_cents=amount_cents)

    # Only mark as done AFTER the side effect succeeds.
    redis_client.set(f"idempotency:{idempotency_key}", charge_id, ex=86400)
    return charge_id

Two details matter here more than the happy path. First, the idempotency key has to be generated by the caller, at the point where the logical decision to do the work is made — not inside the task itself. If you generate the key inside charge_customer, a redelivered task generates a fresh key every time and the whole check is pointless. Generate it once, upstream, typically as a UUID tied to the business event (f"order-charge-{order_id}" works if an order can only be legitimately charged once; a fresh UUID per checkout attempt works if retried checkouts are legitimately separate charges).

Second, the write to mark completion has to happen after the side effect succeeds, not before — marking it done first and then failing the actual charge would silently swallow real work that never happened. This ordering feels obvious written out, but it’s the exact bug that shows up in idempotency implementations that were added under deadline pressure.

Distributed Locks with SET NX EX

The idempotency key check above handles sequential redelivery — task ran, then ran again later. It doesn’t handle concurrent redelivery, where two workers pick up the same message at nearly the same instant and both pass the “already done?” check before either has written the completion marker. That’s a race condition the key alone can’t close; you need a lock.

Redis’s SET key value NX EX seconds is the standard building block: it sets a key only if it doesn’t already exist (NX), with an automatic expiry (EX) so a crashed worker doesn’t hold the lock forever.

import uuid
import redis
from celery import shared_task

redis_client = redis.Redis(host="localhost", port=6379, db=2)

def acquire_lock(lock_key: str, ttl_seconds: int = 30) -> str | None:
    token = str(uuid.uuid4())
    # NX: only set if not already held. EX: auto-expire so a dead worker
    # can't hold the lock forever.
    acquired = redis_client.set(lock_key, token, nx=True, ex=ttl_seconds)
    return token if acquired else None

def release_lock(lock_key: str, token: str) -> None:
    # Only release if we're still the holder — avoids releasing a lock
    # that expired and was re-acquired by someone else in the meantime.
    lua_script = """
    if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
    else
        return 0
    end
    """
    redis_client.eval(lua_script, 1, lock_key, token)

@shared_task(bind=True)
def charge_customer(self, idempotency_key: str, order_id: int, amount_cents: int):
    lock_key = f"lock:charge:{idempotency_key}"
    token = acquire_lock(lock_key, ttl_seconds=30)
    if token is None:
        # Someone else is already processing this exact unit of work.
        # Retry later rather than proceeding concurrently.
        raise self.retry(countdown=5, max_retries=10)

    try:
        already_done = redis_client.get(f"idempotency:{idempotency_key}")
        if already_done:
            return already_done.decode()

        charge_id = payment_gateway.charge(order_id=order_id, amount_cents=amount_cents)
        redis_client.set(f"idempotency:{idempotency_key}", charge_id, ex=86400)
        return charge_id
    finally:
        release_lock(lock_key, token)

The lock TTL needs to comfortably exceed the task’s worst-case execution time — if it expires while the task is still legitimately running, a second worker can acquire it and you’re back to a race, just a rarer one. The release_lock function’s compare-and-delete via a Lua script matters for the same reason: without it, a worker whose lock already expired (and was picked up by someone else) could delete the new holder’s lock on cleanup, defeating the whole point.

This lock-plus-key pattern generalizes well beyond payments: any task with an external, non-transactional side effect (sending a notification, calling a third-party API, writing to a system you don’t control) benefits from the same two-layer defense.

A Note on Single-Instance Redis Locks

The SET NX EX pattern above assumes a single Redis instance (or a primary you’re reading and writing against consistently), and that assumption is worth stating explicitly rather than glossing over. If Redis fails over to a replica before the lock key has replicated — Redis replication is asynchronous by default — a second worker can acquire the “same” lock against the new primary while the first worker still believes it holds it. For most task-queue use cases this is an acceptable, well-understood risk: the failure window is small, and the idempotency key check plus the database unique constraint downstream catch what the lock misses. If your workload genuinely cannot tolerate that gap — very high-value financial operations, for instance — look at the Redlock algorithm, which acquires the lock against a quorum of independent Redis instances rather than one, at the cost of meaningfully more operational complexity. For the overwhelming majority of Celery workloads, a single well-configured Redis instance with a sensible TTL is the pragmatic choice, and reaching for Redlock by default is usually solving a problem you don’t have yet.

Database-Level Idempotency as a Second Line of Defense

Redis-based locks and keys are fast but not infallible — Redis can restart, keys can expire slightly earlier than intended under memory pressure, and a lock’s TTL is always a guess about worst-case timing. The most robust idempotency guarantee, when the side effect is a database write, is a unique constraint the database enforces regardless of what the application layer got wrong.

CREATE TABLE charges (
    id SERIAL PRIMARY KEY,
    idempotency_key VARCHAR(255) UNIQUE NOT NULL,
    order_id INTEGER NOT NULL,
    amount_cents INTEGER NOT NULL,
    gateway_charge_id VARCHAR(255) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT now()
);

from sqlalchemy.exc import IntegrityError

@shared_task(bind=True)
def charge_customer(self, idempotency_key: str, order_id: int, amount_cents: int):
    charge_id = payment_gateway.charge(order_id=order_id, amount_cents=amount_cents)
    try:
        db.session.add(Charge(
            idempotency_key=idempotency_key,
            order_id=order_id,
            amount_cents=amount_cents,
            gateway_charge_id=charge_id,
        ))
        db.session.commit()
    except IntegrityError:
        # Row already exists for this key — a duplicate delivery raced us
        # here. The charge already succeeded and was recorded; treat this
        # as success, not failure.
        db.session.rollback()
    return charge_id

Notice this version still calls the payment gateway before checking for a duplicate — that’s a real gap, since a genuinely concurrent duplicate would still double-charge externally even though your own database stays consistent. That’s exactly why this is a second line of defense layered under the Redis lock, not a replacement for it: the lock prevents concurrent execution from reaching the external call twice in the first place, and the unique constraint catches the cases where the lock’s TTL assumption turns out to be wrong. Belt and suspenders, not either/or.

Handling Non-Idempotent Side Effects

Some side effects genuinely can’t be made idempotent at the point of execution — sending an email is the classic case, since there’s no “check first” step most transactional email providers expose that’s cheaper than just sending it. The idempotency key pattern still works here, but the burden shifts entirely onto the check-before-send guard, since there’s no external system to fall back on as a second check the way a unique constraint backs up a database write.

@shared_task(bind=True)
def send_order_confirmation(self, idempotency_key: str, order_id: int):
    lock_key = f"lock:email:{idempotency_key}"
    token = acquire_lock(lock_key, ttl_seconds=15)
    if token is None:
        raise self.retry(countdown=3, max_retries=5)

    try:
        sent_key = f"idempotency:email:{idempotency_key}"
        if redis_client.get(sent_key):
            return "already-sent"

        email_provider.send(to=get_order_email(order_id), template="order_confirmation")
        redis_client.set(sent_key, "1", ex=604800)
        return "sent"
    finally:
        release_lock(lock_key, token)

For genuinely irreversible external actions with no idempotency support on the provider’s side, the honest answer is that you’re managing risk, not eliminating it — the goal shifts to making duplicates rare and low-consequence (a duplicate confirmation email is an annoyance) rather than assuming you can drive the probability to zero for every possible side effect. Reserve the strictest locking and longest TTLs for the side effects where a duplicate is genuinely costly — payments, inventory decrements, irreversible state transitions — and accept looser guarantees for the ones where it’s merely annoying.

Testing for Idempotency

Idempotency bugs are easy to introduce and easy to miss in code review, because the happy-path test — call the task once, assert the expected result — passes identically whether or not the task is actually idempotent. The test that matters calls it twice:

def test_charge_customer_is_idempotent():
    key = str(uuid.uuid4())
    result_1 = charge_customer.apply(args=[key, order.id, 1000]).get()
    result_2 = charge_customer.apply(args=[key, order.id, 1000]).get()

    assert result_1 == result_2
    assert payment_gateway_mock.charge.call_count == 1  # not called twice
    assert Charge.query.filter_by(idempotency_key=key).count() == 1

A stronger version simulates the actual concurrent-redelivery scenario with threads or multiprocessing, invoking the task with the same key from two workers at once and asserting the external call still fires exactly once. That test is more expensive to write and slower to run, but it’s the only one that actually exercises the lock path rather than the sequential-key-check path — most idempotency bugs that reach production are concurrency bugs, not sequential logic bugs, and a sequential test alone will pass right over them.

Common Pitfalls

Mistake: generating the idempotency key inside the task. This defeats the entire pattern — a redelivered task with a freshly generated key looks like new work every time. Solution: generate the key once, at the point the business decision to do the work is made, and pass it into the task as an argument.

Mistake: marking work complete before the side effect actually succeeds. A crash between the marker write and the real work means the marker lies. Solution: always write the completion marker after the side effect succeeds, never before.

Mistake: relying on a lock with no expiry, or one shorter than worst-case task duration. A lock with no TTL turns a crashed worker into a permanent deadlock; a too-short TTL lets a second worker in while the first is still legitimately running. Solution: set the TTL generously above your task’s p99 duration, and monitor for tasks that regularly approach it.

Mistake: assuming the database unique constraint alone is sufficient. It catches duplicate writes, but by the time it fires, an external non-transactional call (a charge, an email) may have already happened twice. Solution: layer a Redis lock in front of the external call, and use the constraint as the second line of defense, not the only one.

Mistake: only testing the sequential case. The concurrent-redelivery race is the case that actually causes incidents, and a sequential “call it twice, one after another” test won’t catch a lock implemented incorrectly. Solution: write at least one test that fires the task concurrently under the same key.

Idempotency isn’t a feature you bolt onto a task after it’s caused an incident — it’s a design constraint you carry into every task from the start, the same way you’d assume a network call can time out. At-least-once delivery isn’t going away, and it isn’t a Celery limitation specifically — it’s what every durable task queue in the Python task queue landscape promises, because the alternative (guaranteed exactly-once) isn’t actually achievable across a network boundary without coordination costs nobody wants to pay. Build for duplicates from day one, and the 2 a.m. double-charge ticket turns into a non-event instead of an incident.