Celery + Redis Gotchas Nobody Warns You About

Redis is the default broker for most new Celery projects, and for good reason — it’s already in the stack for caching, it’s fast, and getting a worker running against it takes about four lines of config. What that easy setup hides is that Redis was never designed as a message broker in the way RabbitMQ was. Celery’s Redis transport is a compatibility layer built on top of Redis’s general-purpose data structures, and several of its behaviors — particularly around delivery guarantees — are emulated approximations of what a real AMQP broker does natively, not first-class features of Redis itself.

Most of the time that gap doesn’t matter. It matters a great deal the first time a task runs twice for no visible reason, or a worker sits at zero CPU while a queue backs up with thousands of pending messages, or a batch of tasks that should have fired next week simply never do. None of these are bugs in Celery. They’re the specific, well-documented consequences of a handful of settings that default to values tuned for a different scenario than the one you’re actually running — and once you know where to look, every one of them is fixable in a config file.

You’ll learn:

• Why Redis as a Celery broker only approximates real message-queue semantics, and where that approximation breaks
• How celery redis visibility timeout misconfiguration causes tasks to run twice on long-running work
• What celery prefetch multiplier actually controls, and why the default causes worker hoarding under mixed task durations
• Why tasks scheduled far in the future can silently vanish from the queue
• How Redis’s maxmemory-policy setting can quietly evict queued tasks under memory pressure
• A practical checklist for diagnosing celery tasks disappearing when logs don’t show an obvious cause

Table of Contents

1. Why Redis Isn’t a Real Message Broker
2. Visibility Timeout and Task Redelivery
3. Prefetch Multiplier and Worker Hoarding
4. Far-Future ETAs and the Reservation Problem
5. maxmemory-policy: The Setting That Eats Your Queue
6. Diagnosing Celery Tasks Disappearing
7. Common Pitfalls

Why Redis Isn’t a Real Message Broker

RabbitMQ speaks AMQP natively: acknowledgments, per-message redelivery, and consumer prefetch are protocol-level primitives the broker itself enforces. Redis has none of that built in — it’s a key-value store with list, set, and sorted-set data structures. Celery’s redis transport builds queue semantics on top of Redis lists (BRPOP for consuming) and uses a separate “unacked” sorted set, keyed by score, to track in-flight messages that haven’t been acknowledged yet.

That “unacked” set is the crux of almost every gotcha in this post. Since Redis has no native concept of “this consumer is still working on this message, don’t give it to anyone else,” Celery has to simulate it: when a worker pulls a task, Celery moves a copy into the unacked set with a timestamp. A periodic sweep (visibility_timeout seconds later) checks that set and, if a message hasn’t been acknowledged by then, assumes the worker died and puts it back on the main queue for redelivery. This is a reasonable design — but it’s a guess based on elapsed time, not a real signal from the worker about whether it’s still alive and working. Every one of the following gotchas comes from that guess being wrong in a specific, predictable way.

If your team is still deciding on a broker rather than locked into Redis already, it’s worth weighing this against the alternatives directly — see the broker comparison in choosing a Python task queue for how Redis, RabbitMQ, and SQS actually differ on delivery guarantees before you’re deep enough into production that switching is expensive.

Visibility Timeout and Task Redelivery

visibility_timeout (set via broker_transport_options) is how long Celery waits before assuming an unacknowledged message’s worker died and putting it back on the queue:

app.conf.broker_transport_options = {"visibility_timeout": 3600}  # seconds

The default is 3600 seconds (one hour) as of recent Celery versions — fine for most workloads, and actively dangerous for two specific cases: tasks that legitimately take longer than the configured timeout, and tasks configured with task_acks_late = True (which most production deployments should be running, per the earlier idempotency discussion).

Here’s the failure sequence: a task starts, takes 90 minutes to complete a large export, and the visibility timeout is set to 3600 seconds. At the one-hour mark, Celery’s sweep decides the worker must have died — no ack has been received — and redelivers the message to a second worker. Now two workers are running the same 90-minute export concurrently, and whichever one finishes second either overwrites the first’s output or duplicates whatever side effect the task performs. This is a textbook case of Celery task running twice that has nothing to do with an actual crash — the original worker was fine the entire time, just slower than the timeout assumed.

The fix is mechanical once you know to look for it: set visibility_timeout comfortably above your longest-running task’s worst-case duration, not its average.

# If your slowest task can take up to 2 hours, don't set this to 1 hour.
app.conf.broker_transport_options = {"visibility_timeout": 10800}  # 3 hours

The trade-off is real, not free: a longer visibility timeout also means a genuinely crashed worker’s task waits longer before being redelivered — if a worker dies one minute into a task with a 3-hour timeout, that task sits unprocessed for up to 3 hours before anyone (including Celery) notices and requeues it. There’s no single correct value; there’s a value that matches your actual task duration distribution, and it’s worth setting per-queue if your workload mixes fast and slow tasks rather than picking one global number that’s wrong for half of them.

Prefetch Multiplier and Worker Hoarding

worker_prefetch_multiplier controls how many tasks each worker process reserves from the queue at once, as a multiple of its concurrency. The default is 4 — meaning a worker with concurrency 8 will grab up to 32 tasks ahead of time and hold them in memory, working through its own local backlog before asking Redis for more.

This default was tuned for a world of many small, fast, roughly-uniform-duration tasks, where prefetching reduces the round-trip overhead of asking the broker for the next task one at a time. It actively hurts you the moment your task durations vary widely. Picture a queue with a mix of tasks that take 100ms and tasks that take 10 minutes: a worker prefetches 32 tasks, several of which happen to be the 10-minute kind, and now those tasks are sitting reserved in that worker’s local buffer — invisible to every other idle worker — while the assigned worker grinds through its queue serially. Other workers sit idle because, as far as Redis is concerned, that work is already spoken for.

This is the classic celery prefetch multiplier hoarding problem, and it looks exactly like the “tasks disappearing” symptom from the outside: the queue length reported by monitoring drops (the tasks were prefetched, so they’re technically “out” of the queue), but throughput doesn’t rise correspondingly, because they’re stuck waiting behind slow tasks on a worker that isn’t free to pick up new work.

# Global default — fine for short, uniform tasks
app.conf.worker_prefetch_multiplier = 4

# For queues with long-running or highly variable-duration tasks
app.conf.worker_prefetch_multiplier = 1

Setting worker_prefetch_multiplier = 1 means a worker only reserves one task per concurrent slot at a time — closer to true fair-dispatch behavior, at the cost of slightly more round trips to the broker per task, which is a cost worth paying for almost any workload with long-running tasks. If you’re mixing task durations in a single deployment, route the long-running tasks to a dedicated queue with its own worker pool set to prefetch_multiplier=1, and leave the default higher for a separate pool handling short, uniform tasks — mixing both patterns on one worker pool with one prefetch setting is what causes the hoarding in the first place.

Far-Future ETAs and the Reservation Problem

Scheduling a task for a specific future time — my_task.apply_async(eta=some_datetime) — works differently on Redis than it does conceptually: Celery doesn’t leave the message sitting untouched in the main queue until the ETA arrives. Depending on configuration and worker behavior, an ETA task can be reserved by a worker well ahead of its execution time and held in that worker’s memory, counting against its prefetch limit, until the scheduled moment arrives.

For a task scheduled a few minutes out, this is invisible. For a task scheduled days or weeks out — a reminder email, a subscription renewal check — it becomes a real problem: that reservation ties up a slot in the worker’s prefetch buffer for the entire waiting period, and if that worker restarts (a deploy, a crash, an autoscaling event) before the ETA arrives, the reserved task can be lost from that worker’s local state depending on exactly when and how the restart happens relative to Redis’s own tracking of it. The net effect looks identical to a task simply vanishing — nothing in the logs points at “this was scheduled correctly and then lost during a routine deploy,” because from the application’s perspective, apply_async succeeded and returned a task ID.

The practical mitigation is to not lean on Celery’s own ETA mechanism for anything scheduled more than an hour or so out. Use celery beat for genuinely periodic work, or — for one-off far-future scheduling — persist the intent in your own database table with a scheduled_for column and a periodic beat task that queries for due rows and enqueues them just-in-time, rather than handing a week-out ETA directly to the broker and trusting a worker to still be holding it, unchanged, seven days later.

# Fragile: a worker restart between now and next month can lose this
send_renewal_reminder.apply_async(args=[user_id], eta=next_month)

# Sturdier: durable intent in your own database, dispatched just-in-time
class ScheduledReminder(db.Model):
    user_id = db.Column(db.Integer, nullable=False)
    scheduled_for = db.Column(db.DateTime, nullable=False)
    dispatched = db.Column(db.Boolean, default=False)

@app.task
def dispatch_due_reminders():
    due = ScheduledReminder.query.filter(
        ScheduledReminder.scheduled_for <= datetime.utcnow(),
        ScheduledReminder.dispatched.is_(False),
    ).all()
    for reminder in due:
        send_renewal_reminder.delay(reminder.user_id)
        reminder.dispatched = True
    db.session.commit()

maxmemory-policy: The Setting That Eats Your Queue

This is the gotcha that causes the most confusion, because it isn’t a Celery setting at all — it’s a Redis server configuration most teams inherit from a caching use case and never revisit when the same Redis instance also becomes a Celery broker.

maxmemory-policy tells Redis what to do when it hits its configured memory ceiling. For a pure cache, allkeys-lru (evict the least-recently-used key, regardless of type) is a completely reasonable, common default — cached data is disposable by definition, and eviction is the intended behavior. The problem is that Celery’s queue data — the lists and sorted sets holding pending and unacked task messages — lives as ordinary Redis keys with no special protection. If that same Redis instance is shared between your application cache and your Celery broker, and memory pressure builds, allkeys-lru will evict queued-but-not-yet-executed task messages exactly as readily as it evicts a stale cache entry. The tasks aren’t failing. They aren’t erroring. They’re just gone — evicted the same way an old cache key would be, because as far as Redis is concerned, there’s no difference between the two.

# redis.conf — check this if Redis is shared between cache and Celery broker
maxmemory-policy allkeys-lru      # dangerous if this Redis also holds queue data

The fix is one of two approaches, both legitimate: run a separate, dedicated Redis instance for Celery with maxmemory-policy noeviction (Redis returns an error on writes when full instead of silently deleting data — which surfaces the capacity problem as a loud failure instead of a silent data loss), or, if a shared instance is unavoidable, use a separate logical database index (redis://localhost:6379/1 for Celery vs /0 for cache) and make sure maxmemory-policy is set to something eviction-safe for the queue’s use case given that Redis’s eviction policy applies instance-wide, not per logical database — the SELECT-style database separation only isolates keyspaces, not memory-pressure behavior.

# celeryconfig.py — dedicated logical DB, separate from application cache
app.conf.broker_url = "redis://localhost:6379/1"
app.conf.result_backend = "redis://localhost:6379/2"

# On the dedicated broker instance (or if sharing is unavoidable, understand
# this applies to the whole instance, not just the Celery-owned keys):
maxmemory-policy noeviction

This is worth checking even if you’ve never touched Redis config yourself — a shared Redis instance provisioned by infrastructure tooling or a managed cloud service often ships with an LRU-friendly default tuned for caching, on the reasonable assumption that whatever’s using it is a cache. If Celery later gets pointed at that same instance, nobody necessarily revisits that assumption.

Diagnosing Celery Tasks Disappearing

When tasks genuinely seem to vanish — no error, no dead-letter entry, no trace in the logs — work through these in order, since they cover the overwhelming majority of real cases:

1. Check maxmemory-policy on the actual broker instance. redis-cli config get maxmemory-policy — if it’s anything other than noeviction and this Redis handles anything besides Celery, this is the first suspect.
2. Check Redis memory usage against maxmemory. redis-cli info memory — if used_memory is close to maxmemory, eviction is actively happening even if you haven’t confirmed the policy yet.
3. Check whether the “disappeared” tasks were ETA/countdown tasks. If so, suspect the far-future reservation problem, especially if a worker restart or deploy happened during the waiting window.
4. Check prefetch settings against task duration variance. A queue that looks stalled with idle workers and a growing backlog is the hoarding pattern, not a disappearance — celery inspect active and celery inspect reserved will show tasks sitting reserved on specific workers instead of genuinely lost.
5. Check visibility_timeout against actual task durations if you’re seeing duplicates rather than losses — a task appearing to run twice is the mirror image of this same class of problem, not a separate one.

Most “vanishing” reports turn out to be one of the first three; most “duplicate” reports trace to the fourth. Very few are Celery bugs in the literal sense — almost all of them are default settings that were never adjusted for the specific shape of the workload actually running against them.

It’s also worth ruling out the boring explanations before diving into broker internals: confirm the task was actually enqueued in the first place (a silent serialization error on the producer side can make apply_async look successful while nothing reaches Redis at all), and confirm you’re inspecting the same Redis instance and logical database the worker is actually configured against — pointing redis-cli at the wrong host or DB index during an investigation is common enough, especially in setups where cache and broker share a host but use different database indexes, that it’s worth double-checking CELERY_BROKER_URL against whatever connection string your debugging session is using before concluding anything more exotic is at fault.

Common Pitfalls

Mistake: leaving visibility_timeout at its default with long-running tasks. Tasks that take longer than the timeout get redelivered and run twice. Solution: set it above your slowest task’s worst-case duration, not its average, and route long tasks to a queue with a timeout tuned for them specifically.

Mistake: using the default prefetch multiplier with mixed-duration tasks. Long tasks get hoarded in one worker’s buffer while others sit idle. Solution: set worker_prefetch_multiplier = 1 for queues with variable or long task durations, and isolate them from short, uniform-duration queues.

Mistake: scheduling far-future work with eta=/countdown= directly. Long-held reservations can be lost across worker restarts. Solution: use celery beat plus your own durable scheduling table for anything beyond roughly an hour out.

Mistake: sharing a cache-tuned Redis instance with Celery without checking maxmemory-policy. allkeys-lru evicts queued tasks exactly like stale cache entries. Solution: dedicate a Redis instance to the broker with noeviction, or at minimum confirm the shared instance’s eviction policy is safe for queue data.

Mistake: assuming “disappearing” and “duplicate” tasks are unrelated problems. They’re often the same root cause — a Redis-emulated delivery guarantee reacting to elapsed time rather than a real signal from the worker — showing up as opposite symptoms depending on timing. Solution: work through visibility timeout, prefetch, ETA handling, and maxmemory-policy as one connected checklist, not four unrelated leads.

None of these gotchas are Redis being unreliable, and none of them are Celery being buggy — they’re the visible seams of Redis playing a role it wasn’t originally built for, held together by settings that ship with defaults tuned for a generic workload rather than yours specifically. Once you’ve traced through visibility timeout, prefetch, far-future ETAs, and maxmemory-policy once, “tasks disappeared and nobody knows why” stops being a mystery and becomes a five-minute checklist.