Why Your RAG Retrieves the Wrong Chunks

“The model is hallucinating” is the default diagnosis when a RAG system gives a wrong answer, and it’s usually the wrong one. Hallucination — the model confidently making something up — is a generation failure. But the far more common failure in a production RAG pipeline happens a step earlier: retrieval hands the model the wrong context, or incomplete context, and the model is doing exactly what it’s supposed to do with what it was given. It’s not lying; it’s answering faithfully from bad source material.

This distinction matters because the two failures have completely different fixes, and teams routinely spend weeks tuning prompts or swapping generation models to fix a problem that was never in generation at all. This post is a diagnostic walkthrough through the four most common reasons retrieval surfaces the wrong chunks — embedding mismatch, chunk boundaries splitting the answer apart, query-document asymmetry, and missing metadata — with concrete ways to identify which one you’re actually looking at, and what to do about each, plus a worked case study tracing one bad answer through the full process end to end.

You’ll learn:

• How to tell whether a wrong answer is a retrieval failure or a generation failure before touching the prompt
• Why the embedding model itself can be systematically wrong for your domain, not just “not great”
• How chunk boundaries silently split the exact sentence a query needs from the context it depends on
• What query-document asymmetry is, and why questions and answers don’t naturally embed close together
• Why missing or incorrect metadata causes retrieval to return content from the wrong document, version, or scope entirely
• A structured checklist to work through when retrieval quality regresses

Table of Contents

1. First: Is This Actually a Retrieval Failure?
2. Embedding Mismatch
3. Chunk Boundaries Splitting the Answer
4. Query-Document Asymmetry
5. Missing or Incorrect Metadata
6. A Worked Case Study: Tracing One Bad Answer
7. When Causes Compound
8. A Diagnostic Checklist
9. Wrapping Up

First: Is This Actually a Retrieval Failure?

Before diagnosing any of the four causes below, confirm the failure is actually upstream of generation. The fastest way: take the query that produced a wrong answer, run retrieval in isolation, and manually check whether the chunk containing the correct answer appears anywhere in what was retrieved — not necessarily at rank 1, just present in the candidate set that got passed to the LLM.

• If the correct chunk is absent entirely, this is a retrieval failure — one of the four causes below. No prompt change fixes an answer that was never in the context window.
• If the correct chunk is present but the answer is still wrong, this is a generation failure — the model had the right material and didn’t use it correctly. That’s a faithfulness problem, covered with its own metrics in evaluating a RAG pipeline.

Running this check on even ten or fifteen recent bad answers, by hand, before assuming anything about the root cause, saves a lot of wasted effort — it’s common to find that most “the model hallucinated” reports are actually “the model never saw the answer.”

Embedding Mismatch

The symptom: retrieval returns chunks that are topically related but never quite contain the specific answer, consistently, across many different queries — not an occasional miss, a pattern.

The mechanism: every embedding model has an implicit domain it was trained to represent well, and text outside that domain embeds into a less meaningful region of the vector space. A general-purpose embedding model trained mostly on web text and Wikipedia-style prose will represent a legal contract’s dense clause structure, a codebase’s API surface, or a medical record’s abbreviation-heavy notes noticeably worse than it represents ordinary prose — not because the model is broken, but because it never learned reliable structure for that kind of text.

How to confirm it: pick five or six queries that are failing, and directly inspect the cosine similarity between the query embedding and the gold chunk’s embedding versus the query embedding and a handful of unrelated chunks. If the gold chunk’s similarity score isn’t meaningfully higher than the unrelated chunks’, the embedding space itself isn’t separating relevant from irrelevant content well for this kind of text — that’s the signature of a mismatch, not a chunking or metadata problem.

def diagnose_embedding_gap(query_emb, gold_chunk_emb, sample_chunk_embs) -> dict:
    gold_sim = cosine_sim(query_emb, gold_chunk_emb)
    sample_sims = [cosine_sim(query_emb, e) for e in sample_chunk_embs]
    return {
        "gold_similarity": gold_sim,
        "sample_mean": sum(sample_sims) / len(sample_sims),
        "gap": gold_sim - (sum(sample_sims) / len(sample_sims)),
    }

A small or negative gap across many queries, not just one, is the signal to act on.

The fix: for a genuinely domain-specific corpus (legal, medical, code, deeply technical internal documentation), a domain-tuned or fine-tuned embedding model usually closes this gap more reliably than switching between general-purpose models. Short of fine-tuning, combining vector search with BM25 gives exact-term matching a path around a weak embedding space entirely, which is often the faster fix to ship while a longer-term embedding solution is evaluated.

Chunk Boundaries Splitting the Answer

The symptom: the correct document is being retrieved, sometimes even the correct general area of it, but the specific fact needed to answer the question is missing from the actual chunk that made it into the top results.

The mechanism: this is a direct consequence of how the corpus was chunked, not how it was searched. A fixed-size or naively-split chunker doesn’t know that a table’s meaning depends on the paragraph introducing it, or that a definition and the sentence using it belong together — it splits on token count, and if the split happens to land between the setup and the payoff, the embedding for the chunk containing the payoff-only half doesn’t fully represent the concept the query is actually asking about.

How to confirm it: open the source document directly and look at exactly where the chunk boundaries fall relative to the sentence or table that contains the gold answer. If the answer is split across two adjacent chunks, or if it’s preceded by essential context (a variable definition, a qualifying condition) that landed in the previous chunk, this is a boundary problem, not an embedding problem.

The fix: this is the entire subject of the pillar post in this series — see chunking strategies for RAG, benchmarked on real documents for a full comparison of fixed, recursive, semantic, and hierarchical approaches, and for the specific mechanics of overlap and hierarchical parent-child chunking, both of which exist specifically to reduce this failure mode.

Query-Document Asymmetry

The symptom: questions phrased as questions (“how do I reset a forgotten password?”) retrieve worse than the same underlying need phrased as a statement (“password reset process”), even though a human would consider both equally valid ways to ask the same thing.

The mechanism: a question and its answer are not naturally close together in embedding space, even though they’re semantically linked — “How long until my refund processes?” and “Refunds are issued within 5-7 business days” share almost no surface vocabulary, and a bi-encoder trained mostly on symmetric similarity (two similar statements, rather than a question paired with a declarative answer) doesn’t automatically learn that a question and its answer should embed close together. This is a structural property of the embedding task, not a bug in any specific model.

How to confirm it: rewrite a handful of failing queries as declarative statements resembling how the answer is likely phrased in the source document, and re-run retrieval. If the declarative version retrieves the gold chunk reliably and the question form doesn’t, this is query-document asymmetry, not embedding mismatch — the embedding space is fine for prose-to-prose matching, it’s specifically the question-to-answer direction that’s weak.

The fix: three approaches, roughly in order of effort:

• Embedding models trained with instruction- or query-specific prefixes (many modern embedding models support separate query: and passage: encoding modes specifically to address this asymmetry) — check whether your current model supports this before assuming it doesn’t.
• Query rewriting or expansion before retrieval: have an LLM rephrase a question into a plausible declarative form, or generate a hypothetical answer to the question (the HyDE technique) and embed that hypothetical answer instead of the literal question, since a plausible answer embeds much closer to the real answer than the question does.

HYDE_PROMPT = "Write a short, plausible passage that would answer this question: {query}"

def hyde_retrieve(query: str, llm_fn, embed_fn, vector_search_fn, k: int = 5):
    hypothetical_answer = llm_fn(HYDE_PROMPT.format(query=query))
    hyde_embedding = embed_fn(hypothetical_answer)
    return vector_search_fn(hyde_embedding, k=k)

This costs one extra LLM call per query, so it’s worth measuring against your eval set rather than assuming it always helps — for corpora where questions and answers already share vocabulary, the gain can be small relative to the added latency.

• Hybrid search again helps here in a different way than for embedding mismatch — if the question and the document share even a few exact terms, BM25 can surface the right chunk even when the dense retrieval path is thrown off by phrasing asymmetry.

Missing or Incorrect Metadata

The symptom: retrieval returns content that’s topically correct but belongs to the wrong document version, the wrong product tier, the wrong region, or — in a multi-tenant system — the wrong customer’s data entirely.

The mechanism: vector similarity has no inherent concept of “current,” “applies to this customer,” or “correct version.” If your corpus contains both the current pricing page and last year’s archived version, and both were indexed without a metadata field distinguishing them, a query about pricing can retrieve either one with no signal to prefer the current one — the embeddings for both are legitimately similar, because the content genuinely is similar. The failure isn’t in the similarity search; it’s that a piece of information the query actually depends on (recency, scope, tenant) was never attached to the chunk as filterable metadata in the first place.

How to confirm it: when a wrong-but-plausible chunk is retrieved, check whether it differs from the correct one along a dimension that should have been a filter — a different document_version, a different product_tier, a different tenant_id — rather than being genuinely irrelevant content. If the retrieved chunk is “right except for scope,” this is a metadata problem, not a semantic one.

The fix: attach the relevant scoping fields (version, tenant, region, access tier) to every chunk at ingestion time, and apply them as a hard pre-filter before similarity search runs, not as a post-hoc check on the results:

SELECT id, body, embedding <=> :query_embedding AS distance
FROM documents
WHERE tenant_id = :tenant_id
  AND document_version = 'current'
ORDER BY embedding <=> :query_embedding
LIMIT 5;

Filtering in the WHERE clause before the similarity ordering runs means an archived or out-of-scope chunk is never a candidate in the first place, rather than relying on the similarity ranking to push it down far enough to not matter — which it often won’t, since a stale document can be genuinely, legitimately similar to the current one. This matters even more in systems serving multiple customers or tenants from a shared index — see multi-tenant RAG: metadata filtering without leaking data for the specific mechanics of pre-filtering versus post-filtering and why the choice between them has real correctness and performance consequences, not just a stylistic one.

A Worked Case Study: Tracing One Bad Answer

To make the checklist concrete, here’s a hypothetical but representative trace of one bad answer through the diagnostic steps.

The report: a user asks “Does the annual plan support seat downgrades mid-cycle?” and the system answers “No, seat count cannot be changed after the annual plan is purchased” — which is wrong; downgrades are allowed, with proration, at the next billing cycle.

Step 1 — is it retrieval or generation? Pulling the raw candidate set for this query shows the top 5 chunks are all about upgrading seats, from a “Scaling Your Plan” section. The chunk that actually covers downgrades — in a separate “Billing Cycle Adjustments” section — never appears in the top 10. This rules out generation: the model never saw the right content, so it answered from the closest thing it did see, which happened to be about the opposite operation.

Step 2 — check the similarity gap. Embedding the query against the missing gold chunk versus the retrieved upgrade-related chunks shows the gold chunk’s similarity score is only marginally lower than the top false positives — not a large gap, which argues against a severe embedding mismatch.

Step 3 — inspect chunk boundaries. The gold chunk itself is intact — the downgrade policy is fully contained within one chunk, not split. This rules out a boundary problem.

Step 4 — rewrite as a declarative statement. Re-running retrieval with “mid-cycle seat downgrade policy for annual plans” instead of the original question surfaces the correct chunk at rank 2. That’s the signal: this was query-document asymmetry — “does X support Y” as a question phrasing embeds closer to other question-shaped and upgrade-related content than to the declarative policy text that actually answers it.

The fix applied: adding HyDE-style query rewriting ahead of retrieval for this corpus (support and billing documentation, which skews heavily toward declarative policy statements rather than Q&A-formatted content) moved this specific query’s gold chunk from unranked to rank 1, and lifted recall@5 on a sample of similarly-phrased questions in the eval set by a meaningful margin — the kind of targeted, measured fix the diagnostic process above is meant to produce, rather than a blanket prompt change applied without knowing which of the four causes was actually at fault.

When Causes Compound

These four causes aren’t mutually exclusive, and on a messy real-world corpus it’s common for two to overlap on the same query. A question phrased asymmetrically to its answer and split across a chunk boundary produces a similarity gap that looks like straightforward embedding mismatch until you separately check boundaries and phrasing — the checklist below is ordered specifically so that ruling out each cause narrows what’s left, rather than trying to diagnose all four at once from a single symptom.

A practical habit: when a fix for one cause doesn’t fully resolve a failing query, don’t conclude the diagnosis was wrong — re-run the checklist against the residual failure. It’s common for a metadata-scoping fix to reveal an underlying asymmetry problem that was previously masked by irrelevant-tenant results crowding out the top ranks entirely.

A Diagnostic Checklist

When retrieval quality regresses or a specific query keeps failing, work through these in order — each rules out or confirms one of the causes above before moving to the next:

1. Confirm it’s retrieval, not generation. Check whether the gold chunk was in the candidate set at all.
2. Check the similarity gap. Is the gold chunk’s similarity score meaningfully higher than unrelated chunks’, or is the embedding space not separating them well?
3. Inspect chunk boundaries around the gold answer. Is the needed fact split across chunks, or missing essential preceding context?
4. Rewrite the query as a declarative statement and re-test. Does that alone fix it? If so, it’s asymmetry, not embedding quality.
5. Check whether the wrong result differs by a scoping dimension (version, tenant, region) rather than by topic. If so, it’s a missing metadata filter.
6. Only after ruling out all four, look at generation. If the correct chunk was present and ranked reasonably and the answer is still wrong, that’s a faithfulness issue — measure it explicitly rather than tuning the prompt by feel.

Wrapping Up

“The model hallucinated” is rarely the full story, and jumping straight to prompt tuning skips the diagnostic step that would have shown the real cause in minutes. Embedding mismatch, split chunk boundaries, query-document asymmetry, and missing metadata are four structurally different problems that happen to produce the same symptom — a wrong answer — and conflating them wastes the exact engineering time that a fifteen-minute manual check on the retrieved candidates would have saved. Work through the checklist before touching the prompt; it’s almost always faster to find the real cause than to guess at it.