Reranking in RAG: When It's Worth the Latency

“Add a reranker” has become the default answer to “our RAG answers aren’t great,” in roughly the same way “add an index” is the default answer to “our query is slow” — sometimes exactly right, sometimes solving a problem you don’t have while adding latency you’ll regret. A reranker doesn’t retrieve anything; it re-scores candidates your first-stage retrieval already found, using a model that’s slower but more precise than the embedding similarity that ranked them initially. If your first-stage retrieval isn’t finding the right chunk at all, reranking has nothing correct to promote, and you’ve added a few hundred milliseconds for no measurable improvement.

The actual decision isn’t “reranker: yes or no” — it’s a tradeoff you can measure directly: how much does a cross-encoder reranker improve precision at the top of your ranking, against how much latency it adds to every single query. This post covers how cross-encoder rerankers work mechanically, why they can afford to be more accurate than embedding similarity, and a reproducible way to measure the recall/precision gain against the latency cost on your own retrieval pipeline — with worked example numbers from a small test set to show the shape of that tradeoff, not a universal claim about how much any given reranker will help.

You’ll learn:

• Why cross-encoder rerankers can be more accurate than the embedding similarity that ranked candidates initially
• The architectural difference between a bi-encoder (used for first-stage retrieval) and a cross-encoder (used for reranking)
• How to measure a reranker’s actual precision gain on your own query set, with worked example numbers from a small test corpus
• The real latency cost of reranking, and where it comes from mechanically
• How to decide the candidate pool size and whether reranking belongs in the synchronous request path at all
• Cheaper alternatives to a full cross-encoder when full reranking latency isn’t acceptable

Table of Contents

1. What a Reranker Actually Does
2. Bi-Encoders vs. Cross-Encoders
3. Measuring the Gain: A Small Worked Benchmark
4. Where the Latency Actually Comes From
5. The Recall-vs-Latency Tradeoff, Concretely
6. Sizing the Candidate Pool
7. Self-Hosted Cross-Encoders vs. Hosted Reranking APIs
8. Instrumenting Reranking Latency in Production
9. Cheaper Alternatives to a Full Cross-Encoder
10. Deciding If Reranking Belongs in Your Pipeline

What a Reranker Actually Does

A standard RAG retrieval step is two conceptually separate jobs squeezed into one: find a reasonably good candidate set from a large corpus quickly, then pick the actually-best few from that candidate set accurately. First-stage retrieval — vector search, BM25, or hybrid search combining both — is optimized for the first job: it has to scan potentially millions of chunks and return a shortlist in single-digit milliseconds, which only works because it pre-computes each chunk’s embedding once and compares it to the query with a fast vector operation.

A reranker is built for the second job only. It takes the candidate set — typically 20 to 100 chunks — that first-stage retrieval already narrowed things down to, and re-scores each one against the query with a model that never runs at index time, only at query time, on this small already-narrowed set. Because it only has to be fast enough for tens of comparisons instead of millions, it can afford to be architecturally slower and more accurate per comparison.

Bi-Encoders vs. Cross-Encoders

The embedding model used for first-stage retrieval is a bi-encoder: it encodes the query and each document independently, into fixed vectors, entirely separately from each other. Similarity is computed afterward with a cheap operation (cosine distance) on two already-computed vectors. This independence is exactly what makes bi-encoders fast enough to index millions of documents ahead of time — the document embeddings are computed once, offline, and reused for every future query.

A cross-encoder throws that independence away on purpose. It takes the query and a single candidate document together, concatenated, as one input, and runs the full transformer over both jointly:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

pairs = [(query, candidate.text) for candidate in candidates]
scores = reranker.predict(pairs)  # one forward pass per (query, candidate) pair

ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)

Because the model attends across the query and the document simultaneously, it can pick up on interactions a bi-encoder’s independent-then-compared vectors structurally can’t represent — whether a specific phrase in the document actually answers the specific question asked, not just whether the two are topically similar. That’s the accuracy gain. The cost is that this can’t be precomputed: every query against every candidate requires a fresh forward pass, which is why cross-encoders are reserved for reranking a short candidate list rather than scanning a whole corpus.

Measuring the Gain: A Small Worked Benchmark

To make the tradeoff concrete, I measured a cross-encoder reranker (ms-marco-MiniLM-L-6-v2) against the same small 20-document, 47-query test set used in the chunking strategies benchmark — engineering runbooks and API reference pages, with hand-labeled gold spans per query. This is explicitly a small, illustrative worked example, not a large-scale published benchmark; the goal is to show the methodology and the shape of the effect, not to hand you a number to assume for your own corpus.

Pipeline: hybrid search (BM25 + vectors, RRF-fused) retrieves the top 30 candidates, and the cross-encoder reranks those 30 down to a final top 5.

The improvement here is entirely a reordering effect within a fixed candidate set — reranking cannot increase recall beyond what’s already present somewhere in the 30 candidates, since it never touches the corpus outside that set. The recall gain from 81% to 91% reflects cases where the correct chunk was retrieved into the top 30 by first-stage search but ranked outside the top 5 — the reranker’s job, and on this test set, roughly a third of the queries that first-stage retrieval “almost” got right (correct chunk present, just not in the top slots) were corrected by reranking. Precision improved for the same underlying reason: the top 5 slots more consistently contained genuinely relevant chunks instead of topically-adjacent-but-wrong ones.

Where the Latency Actually Comes From

The recall and precision gains above aren’t free, and understanding exactly where the latency goes is what makes the tradeoff decision concrete rather than a vague “reranking is slow” intuition. Reranking latency scales with the number of (query, candidate) pairs scored, because each pair is a separate forward pass through the model — there’s no batching shortcut that avoids the per-pair compute the way a precomputed embedding index does.

Rough, hardware-dependent orders of magnitude for a small cross-encoder like MiniLM: single-digit milliseconds per pair on a GPU, tens of milliseconds per pair on CPU. Reranking 30 candidates on CPU can land anywhere from roughly 150ms to 500ms depending on model size, sequence length, and batching efficiency; the same 30 candidates on a GPU with proper batching typically lands well under 100ms. Larger, more accurate cross-encoders (full-size models rather than distilled ones like MiniLM) push both numbers up further in exchange for another step of accuracy.

This is additive to whatever latency the first-stage retrieval and the LLM generation step already cost — reranking sits in the middle of the request path, not in parallel with either of them, since generation needs the reranked (not the raw) candidate set as its input.

The Recall-vs-Latency Tradeoff, Concretely

Putting the two measurements next to each other is the actual decision:

Reranking a smaller candidate pool costs less latency but recovers less of the possible gain, because fewer of the “almost right” cases (correct chunk present but outside the top slots) are captured in a smaller pool to begin with. This is the actual shape of the decision: it isn’t reranking-or-not, it’s how many candidates you’re willing to pay to rerank, traded directly against how much of the accuracy gain you need to capture.

Whether an extra 200-500ms is acceptable depends entirely on what’s downstream. A synchronous chat interface where a user is watching a loading spinner has a much lower latency budget than an async batch job generating a report overnight. A support ticket auto-triage system running against a queue has more room than a live customer-facing chatbot.

Sizing the Candidate Pool

The 30-candidate pool size used above isn’t a universal default — it’s a tuning knob with the same shape as the chunk-size tradeoff covered in the chunking strategies post: too small and the reranker never sees the correct chunk to promote it; too large and you’re paying reranking latency for candidates that had essentially no chance of being correct in the first place.

A practical way to size it: measure recall@k on first-stage retrieval alone at several values of k (10, 20, 30, 50) against your eval set. The k at which recall@k starts to plateau is roughly the ceiling on how much reranking can help — reranking a pool of 50 when recall@50 and recall@30 are nearly identical just adds 20 candidates’ worth of latency for no additional correct chunks to find.

Self-Hosted Cross-Encoders vs. Hosted Reranking APIs

There are two operationally different ways to run the cross-encoder step, and the choice affects the latency and reliability profile more than the accuracy one.

Self-hosted (a CrossEncoder model loaded into your own service, as in the code above) gives you full control over batching, hardware, and colocation with the rest of the pipeline — reranking can run on the same GPU box as embedding generation, avoiding a network hop entirely. The cost is operational: you own model updates, GPU provisioning, and scaling the reranking service under load, on top of everything else already running in a RAG pipeline.

Hosted reranking APIs (Cohere’s Rerank API and similar offerings from other vector-database and search vendors) remove that operational burden — a single HTTP call with the query and candidate texts returns a reranked order, no model hosting required. The tradeoff is a network round trip added to every query on top of the model’s own inference time, and per-call pricing that scales with the exact candidate-pool-size decision covered above — a larger candidate pool isn’t just more local compute, it’s a directly larger bill.

import cohere

co = cohere.Client(api_key="...")

response = co.rerank(
    query=query_text,
    documents=[c.text for c in candidates],
    top_n=5,
    model="rerank-english-v3.0",
)

Neither option changes the fundamental recall-vs-latency math from the sections above — it only changes where the latency comes from (local inference vs. network plus remote inference) and who’s responsible for keeping it fast. Measure both the same way: total wall-clock time added to the request, on your own network path, not a vendor’s quoted median.

Instrumenting Reranking Latency in Production

The tuning decisions above only stay correct if you keep measuring after reranking ships, since latency characteristics drift with candidate pool size, model updates, and load. A minimal instrumentation pattern around the reranking call:

import time
import logging

logger = logging.getLogger("rag.rerank")

def rerank_with_metrics(reranker, query: str, candidates: list, top_n: int = 5):
    start = time.perf_counter()
    pairs = [(query, c.text) for c in candidates]
    scores = reranker.predict(pairs)
    elapsed_ms = (time.perf_counter() - start) * 1000

    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:top_n]

    logger.info(
        "rerank_complete",
        extra={
            "candidate_count": len(candidates),
            "latency_ms": round(elapsed_ms, 1),
            "top_score": float(ranked[0][1]) if ranked else None,
        },
    )
    return ranked

Logging candidate_count alongside latency_ms matters specifically because it lets you separate a real regression (the same pool size suddenly taking longer — a hardware or model change) from an expected latency increase caused by a pool size change made somewhere else in the pipeline. Track this alongside recall@k on a rolling eval sample, not just latency alone — a reranker that got faster after a model swap but stopped correcting as many “almost right” cases is a regression even though the latency graph looks like an improvement.

Cheaper Alternatives to a Full Cross-Encoder

When full cross-encoder latency doesn’t fit the budget, there are intermediate options rather than an all-or-nothing choice:

• Smaller, distilled cross-encoders (like the MiniLM variant used above) trade a small amount of accuracy for meaningfully lower per-pair latency compared to full-size cross-encoder models — usually the first lever to pull before abandoning reranking altogether.
• ColBERT-style late-interaction models compute token-level embeddings that can be partially precomputed for documents, closing some of the latency gap between a bi-encoder and a cross-encoder while keeping more of the accuracy benefit than a pure bi-encoder gets.
• LLM-as-reranker with a cheap, fast model: prompting a small, fast LLM to score or reorder a short candidate list can work, but adds its own latency and cost profile that needs the same measurement treatment — don’t assume it’s cheaper than a purpose-built cross-encoder without checking.
• Async reranking outside the critical path: for use cases where the first answer doesn’t need to be the final answer (background enrichment, pre-computed FAQ-style answers refreshed periodically), rerank asynchronously and cache the result rather than paying the latency on every live request.

Deciding If Reranking Belongs in Your Pipeline

A short decision sequence, in order:

1. Check whether first-stage recall is actually the bottleneck. If the correct chunk usually isn’t in your top-30 candidates at all, reranking has nothing to promote — fix retrieval or chunking first. See why RAG retrieves the wrong chunks for the diagnostic steps before assuming reranking is the fix.
2. Measure recall@k and precision@k with and without reranking on your own eval set, using the same methodology as the worked example above — not a number borrowed from someone else’s corpus.
3. Measure the actual added latency on your own hardware and model choice, not a vendor’s benchmark page — CPU vs. GPU, batch size, and candidate pool size all move the number meaningfully.
4. Weigh the measured gain against your product’s actual latency budget. A 10-point recall improvement is a very different decision at 200ms added latency in a synchronous chat UI than at 200ms added latency in an overnight batch pipeline.
5. If it’s marginal, try a smaller pool or a distilled model before ruling reranking out entirely — the tradeoff curve above shows there’s real room to tune between “no reranking” and “rerank everything.”

Common Pitfalls

Reranking a candidate pool that’s already too small. If first-stage retrieval only returns the top 5 and you rerank those same 5, the reranker can only reorder what’s already there — it can’t rescue a correct chunk that never made the cut. Widen the pool before reranking, then narrow it after, as two distinct steps with two distinct sizes.

Comparing a reranker’s own confidence score across queries. Cross-encoder scores are not calibrated probabilities and aren’t directly comparable between different queries — a top score of 0.9 on one query and 0.6 on another doesn’t mean the first result is more reliable, only that the score distributions differ per query. Use scores for within-query ordering only, never as a cross-query confidence threshold without separately calibrating one.

Assuming latency numbers from a vendor page transfer to your setup. Batch size, sequence length, hardware, and candidate count all move the real number substantially — the only latency figure worth trusting for a go/no-go decision is one measured on your own request path, under your own load.

Skipping the retrieval-quality check before adding a reranker. It’s tempting to reach for reranking as soon as answers look wrong, but if the underlying issue is chunk boundaries or embedding mismatch, reranking adds latency without addressing the actual failure — always confirm the correct chunk is reachable in the first-stage candidate pool before spending a latency budget on reordering it.

Wrapping Up

Reranking earns its latency when first-stage retrieval is finding the right chunk but not ranking it highly enough, and it earns nothing when the right chunk was never in the candidate set to begin with. The mechanism — a cross-encoder scoring the query and document jointly instead of comparing precomputed vectors — genuinely does produce a measurable accuracy gain, but “measurable” is the operative word: it should be a number from your own eval set and your own latency budget, not an assumption inherited from a blog post’s default configuration, this one included.