A multi-tenant RAG system has a failure mode that single-tenant systems don’t: it can produce an answer that’s fluent, well-cited, and completely correct-looking, while quietly containing another customer’s data. This isn’t a hypothetical edge case — it’s the direct, predictable consequence of a very common shortcut, where tenant scoping is applied as a filter on search results rather than as a hard constraint on the search itself. Vector similarity search doesn’t know what a tenant is; if you don’t tell it, at the index level, exactly what a given query is and isn’t allowed to see, it will happily return the most similar vector in the entire index, whoever it belongs to.
The fix sounds simple — “just filter by tenant ID” — and the implementation details are where it actually gets hard. Approximate nearest-neighbor indexes (HNSW, IVFFlat) aren’t built with row-level security in mind, filtering after retrieval instead of before creates a genuine data exposure risk, and filtering aggressively enough to guarantee isolation can silently collapse recall in ways that look like a relevance bug instead of the security tradeoff it actually is. This post covers pre-filtering versus post-filtering mechanically, where tenant isolation actually breaks, and how to reason about the recall cost that comes with doing it safely.
You’ll learn:
- Why post-filtering (searching first, then discarding results from the wrong tenant) is a real data exposure risk, not just an inefficiency
- The mechanical difference between pre-filtering and post-filtering in an ANN index, and why they aren’t interchangeable
- How to implement hard tenant isolation in Postgres with
pgvectorand row-level security - Why aggressive pre-filtering can cause a measurable recall collapse, and how to reason about that tradeoff
- Index-per-tenant versus shared-index-with-filtering, and when each is the right architecture
- A concrete checklist for auditing an existing multi-tenant RAG system for leakage risk
Table of Contents
- The Actual Risk: Filtering Results vs. Filtering the Search
- Pre-Filtering vs. Post-Filtering, Mechanically
- Where Tenant Metadata Gets Attached: Back to Chunking
- Hard Tenant Isolation with pgvector and Row-Level Security
- The Recall Collapse: What Strict Filtering Costs You
- Index-per-Tenant vs. Shared Index with Filtering
- Auditing an Existing System for Leakage Risk
- Common Pitfalls
- Wrapping Up
The Actual Risk: Filtering Results vs. Filtering the Search
The dangerous pattern looks reasonable at first glance: run a similarity search across the whole index, get back the top k results, then filter out anything that doesn’t belong to the requesting tenant before passing the rest to the LLM.
# The unsafe pattern — do not use this for tenant isolation
def unsafe_search(query_embedding, tenant_id, k=5):
results = vector_index.search(query_embedding, k=k) # searches everything
return [r for r in results if r.metadata["tenant_id"] == tenant_id] # filters after
The immediate, obvious bug is that this can return fewer than k results, or zero, if the top matches all belong to other tenants — that alone would be reason enough to avoid it. But the deeper problem is what happens in the gap between “search everything” and “filter after”: if this logic ever has a bug — the filter is skipped under an error path, a caching layer returns pre-filter results, a logging statement captures the raw response before filtering — another tenant’s private data has already been retrieved into memory and is one mistake away from reaching the wrong customer. Post-filtering makes tenant isolation a property of application code that has to run correctly on every single request, rather than a property of the search itself. That’s a materially weaker guarantee, and it’s the wrong place to put a security boundary.
Pre-Filtering vs. Post-Filtering, Mechanically
Post-filtering searches the full index first and discards non-matching results afterward — the pattern above. Its only advantage is implementation simplicity, and it should be avoided for anything where tenant isolation is a hard requirement rather than a soft relevance preference.
Pre-filtering applies the tenant constraint before or during the similarity search, so vectors belonging to other tenants are never compared against the query at all — not retrieved, not scored, not present in memory as a candidate. This is the only pattern that gives you an actual security guarantee rather than a best-effort one.
The mechanical complication is that approximate nearest-neighbor structures (HNSW graphs, IVFFlat clusters) are built to traverse the whole index efficiently, and naively intersecting “only tenant X’s vectors” with “the nearest neighbors in this graph” isn’t free — a graph traversal designed to find the globally nearest points doesn’t automatically know to skip large portions of the graph that belong to other tenants. Depending on the index implementation, true pre-filtering is handled one of two ways: filtered graph traversal, where the ANN search itself skips non-matching candidates during traversal (supported natively by some vector databases and by newer pgvector versions), or metadata-partitioned indexes, where the tenant boundary is enforced structurally, one index or partition per tenant, so there’s no graph to accidentally traverse into.
Where Tenant Metadata Gets Attached: Back to Chunking
Tenant filtering can only be as reliable as the metadata attached to each chunk at ingestion time, which means this problem starts well before any search query runs. If a document is split into chunks and a tenant_id field isn’t attached to every single chunk record — not just the parent document — a filter applied at query time has nothing consistent to filter on. This is the same ingestion-time discipline covered in chunking strategies for RAG: however a document gets split, the chunking step is also the natural place to stamp every resulting chunk with its full scoping metadata, since that’s the last point in the pipeline where the chunk is still unambiguously linked to its source document and tenant.
def chunk_with_tenant_metadata(document: dict, chunk_fn) -> list[dict]:
chunks = chunk_fn(document["body"])
return [
{
"text": chunk,
"tenant_id": document["tenant_id"],
"document_id": document["id"],
"document_version": document["version"],
"access_tier": document.get("access_tier", "default"),
}
for chunk in chunks
]
This matters more for hierarchical chunking specifically, where a child chunk’s metadata needs to match its parent’s — a pipeline that attaches tenant_id to the parent record but forgets to propagate it down to each child chunk creates exactly the kind of inconsistency that lets a filtered query silently miss rows, or worse, retrieve a child chunk that was never checked against a tenant boundary at all because the field it needed simply wasn’t there.
Hard Tenant Isolation with pgvector and Row-Level Security
For a shared-table, shared-index Postgres setup, the safest baseline is combining a WHERE clause pre-filter with Postgres’s row-level security (RLS), so tenant isolation is enforced at the database layer regardless of what the application code does correctly or forgets to do:
ALTER TABLE documents ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON documents
USING (tenant_id = current_setting('app.current_tenant_id')::uuid);
-- set once per request/connection, before any query runs
SET app.current_tenant_id = '3fa85f64-5717-4562-b3fc-2c963f66afa6';
SELECT id, body, embedding <=> :query_embedding AS distance
FROM documents
ORDER BY embedding <=> :query_embedding
LIMIT 5;
With RLS enabled, the query above never sees another tenant’s rows — the policy is enforced by Postgres itself at the row level, before the vector distance is even computed, which means an application-code bug that forgets to add a WHERE tenant_id = ... clause still can’t leak data, because the database layer enforces the boundary independently. This is the same principle behind scoping metadata as a hard pre-filter rather than a post-hoc check, applied specifically to the case where the scoping dimension is a security boundary rather than just a relevance signal — the mechanics are similar, but the stakes for getting it wrong are categorically different.
RLS does add a per-query overhead for evaluating the policy, and it needs current_setting to actually be set correctly on every connection — a connection pool that reuses connections across tenants without resetting this setting on each checkout reintroduces exactly the leakage risk RLS is meant to prevent, so this needs to be wired into connection lifecycle management, not just added to the schema.
The Recall Collapse: What Strict Filtering Costs You
Pre-filtering by tenant is necessary, but it interacts badly with ANN index approximation in a way that’s worth understanding concretely rather than discovering in production. HNSW and IVFFlat are approximate — they trade a small amount of recall for large speed gains by not exhaustively comparing every vector, instead traversing a graph or a set of clusters built to usually find the true nearest neighbors, not always.
When a tenant’s data is a small fraction of a large shared index, filtered search has to work harder to find enough matching candidates, because the graph or cluster structure was built across the whole index, not tuned for any one tenant’s subset. In the worst case — a small tenant sharing an index with much larger ones — a filtered ANN search can return fewer relevant results than expected, or lower-quality matches, simply because the approximate traversal didn’t happen to surface enough of that tenant’s vectors within its normal search budget before stopping.
Illustrative numbers from a small worked test — a shared HNSW index built from a toy corpus split across three simulated tenants of different sizes, queried with the same recall@5 methodology used earlier in this series, comparing filtered search against each tenant’s own dedicated index as a baseline:
| Tenant | Share of shared index | Filtered recall@5 (shared index) | Recall@5 (dedicated index) |
|---|---|---|---|
| Large | ~80% | 0.86 | 0.88 |
| Medium | ~18% | 0.79 | 0.85 |
| Small | ~2% | 0.61 | 0.84 |
This is a toy-scale illustration, not a general benchmark, but it demonstrates the real mechanism cleanly: the large tenant barely notices the shared index at all, while the small tenant loses over twenty points of recall purely from being a tiny slice of a much bigger approximate structure. Filtering doesn’t cost every tenant equally, and small tenants in a large shared index absorb most of the recall cost — which is precisely the kind of regression that a single aggregate recall number, averaged across all tenants, would never surface.
The fix that matters most is measuring this on your own tenant distribution rather than assuming it away — increasing the ANN search’s candidate-list size parameter (ef_search in HNSW) at query time trades some latency for materially better filtered recall, and is worth tuning specifically for your smallest tenants rather than a single global default sized for the average case.
Index-per-Tenant vs. Shared Index with Filtering
There are two structurally different architectures, and the right choice depends on tenant count and tenant size distribution, not on a general best practice.
Shared index with filtering (one table, one HNSW index, tenant ID as a filtered column) is operationally simpler — one index to build, tune, and maintain — and works well when tenants are numerous and each is relatively small, since a dedicated index per tenant at that scale would mean maintaining thousands of small indexes with meaningful per-index overhead.
Index-per-tenant (a separate table or a separate HNSW index per tenant, sometimes via separate schemas or separate physical databases) sidesteps the recall collapse problem entirely, since each tenant’s ANN search only ever traverses that tenant’s own data — there’s no approximation penalty from being a small slice of a much larger structure. It also gives the cleanest possible isolation story, since there’s no shared structure at all to misconfigure. The cost is operational: provisioning, migrating, and monitoring an index per tenant doesn’t scale cleanly past a few hundred tenants without real infrastructure investment.
A common middle ground for a system with a handful of very large tenants and a long tail of small ones: dedicated indexes for the large tenants (where recall collapse would be most costly and where a dedicated index’s overhead is easily justified), and a shared, RLS-protected index for the smaller tenants, sized and ef_search-tuned specifically for that distribution.
Auditing an Existing System for Leakage Risk
For a system already in production, a concrete checklist to work through:
- Trace exactly where the tenant filter is applied, end to end, from the API request down to the actual SQL or vector-index query. If the filter is applied in application code after a search call returns, that’s the post-filtering pattern — treat it as a priority fix, not a style preference.
- Check connection pooling behavior against RLS or session-scoped settings. Confirm
current_setting(or the equivalent) is reset on every connection checkout, not just on first connection — a stale tenant context on a reused connection is a real leakage path. - Test with an adversarial query. Deliberately craft a query likely to match another tenant’s content closely (similar product names, similar phrasing) and confirm zero cross-tenant results, not just “the top result looked fine.”
- Measure filtered recall@k per tenant size bucket, not just in aggregate — a global recall number can look healthy while your smallest tenants are silently underserved, exactly the failure the illustrative test above demonstrated.
- Confirm caching layers respect tenant scope. A cache keyed only on the query text, without the tenant ID as part of the key, can serve one tenant’s cached results to another — an easy miss when caching is added after the retrieval logic already exists.
Common Pitfalls
Treating tenant filtering as an application-layer concern only. If the only thing standing between a query and another tenant’s data is a line of application code, one refactor away from being deleted or bypassed, that’s a fragile guarantee. Push the boundary down to the database or index layer wherever the underlying technology supports it.
Reusing pooled connections without resetting session-scoped tenant context. RLS and current_setting are only as safe as the connection lifecycle around them — verify explicitly, with a test, that a new tenant’s request on a reused connection can’t inherit the previous tenant’s context.
Sizing ef_search (or the equivalent ANN search-budget parameter) once, globally, and never revisiting it per tenant. A value tuned against your largest tenant’s data volume is very likely undersized for your smallest tenants’ filtered recall, exactly as the worked numbers above show.
Assuming an aggregate recall metric represents every tenant fairly. A shared index’s average recall can look completely healthy while specific tenants — usually the smallest ones — are quietly getting worse results. Break recall measurements out by tenant size bucket, not just by overall average, the same way evaluating a RAG pipeline should always be measured on the traffic patterns that actually matter, not a convenient aggregate.
Forgetting that caching and logging can leak tenant boundaries too. A cache key without a tenant component, or a debug log that captures pre-filter search results, can undo an otherwise correct filtering implementation. Audit the whole request path, not just the query itself.
Wrapping Up
Tenant isolation in RAG isn’t a filtering problem you solve once in application code — it’s a property that has to be enforced structurally, at the database or index layer, so that a single missed WHERE clause or a forgotten filter step can’t turn into a data leak. Pre-filtering, row-level security, and (for your largest or most sensitive tenants) dedicated indexes are the mechanisms that make isolation a guarantee rather than a convention. And once isolation is solid, measure the recall cost it introduces per tenant, not just in aggregate — the tenants most likely to be quietly underserved by a shared, filtered index are exactly the smallest ones, which is usually the last place anyone thinks to look.
