5 Reasons Your RAG System Fails in Production

You built a RAG pipeline. It worked beautifully in your notebook. Then you shipped it, and users started complaining about irrelevant answers and missing information.

Here are the five most common reasons RAG systems break in the real world, and what you can do about each one.

1. Bad Chunking

This is where most RAG pipelines quietly die.

Most tutorials reach for LangChain's RecursiveCharacterTextSplitter because it's the default.
It splits on paragraphs, then sentences, then words, recursively, until chunks fit your size limit. It's convenient but also blunt.

A simpler splitter that cuts by character count or paragraph breaks is even worse.
Here’s why:

• Neither approach understands structure.
• A table gets sliced mid-row.
• A code block gets split between a function signature and its body.
• An image reference gets orphaned from its caption.
• A LaTeX equation gets cut in half, leaving you with \frac{x in one chunk and }{2} in another.

The next thing is overlap. Chunk overlap matters more than people expect. Without chunk overlap, you lose the connection between sections. A sentence that references "the above approach" now lives in a chunk with zero context.

A typical overlap of 10-20% of chunk size gives the retrieval model enough surrounding text to preserve meaning.

Better strategies to consider:

- Semantic chunking: uses embedding similarity to detect topic shifts and split where meaning actually changes, rather than at arbitrary character counts. It’s slower and expensive at indexing time, but chunks stay coherent.
- Late chunking: (introduced by Jina AI) embeds the full document first through a long-context embedding model, then splits, so each chunk's vector carries global context.
- Structure-aware splitting parses markdown headers, HTML tags, code fences, and table delimiters before cutting, keeping logical units intact.
- Recursive splitting remains useful as a fallback layer, but it shouldn't be your only tool.

The right choice depends on your corpus:

• Legal contracts need section-aware splitting.
• Codebases need file-and-function-aware splitting.
• Research papers need equation and figure handling.

2. Embedding mismatch

An embedding model maps text into a vector space that only it understands. Vectors from model A mean nothing in the space of model B.

So the first rule is boring but critical: use the same embedding model for indexing and for querying. If you swap models later, you must re-embed the entire index.

The second rule is that different content wants different models. A general-purpose model does okay everywhere and great nowhere. For legal contracts, a model trained on legal text (like Voyage AI's law models) ranks the right clauses higher. For code search, code-specific models understand identifiers and syntax in ways general models don't. If your corpus is multilingual, pick a model trained across those languages.

The MTEB leaderboard is a good starting point, but always test on your own data, because leaderboard wins don't always transfer.

Then there's the dimension question. Higher dimensions capture more nuance and usually retrieve better, but they cost more in storage, memory, and latency, and the gains shrink past a point. Lower dimensions are cheaper and faster but can blur subtle distinctions.

Many newer models support Matryoshka representations, which let you truncate a vector (say from 1024 to 256 dimensions) and keep most of the quality. That gives you a dial to tune between cost and accuracy.

3. Retrieval noise

Retrieval noise is what happens when the context window fills with chunks that look related but aren't useful. The usual causes:

• Top-k set too high. Retrieving 20 chunks "to be safe" means 15 of them are marginal or irrelevant, and they dilute the signal of the good ones. Models also pay less attention to the middle of a long context (the lost in the middle effect), so the right answer can sit in the prompt and still get ignored.
• Missing metadata. A chunk without a source, date, or document type can't be filtered. You end up serving a 2019 policy document for a question about the 2024 policy, with no way to exclude it.

Here’s how to fix’em:

• Metadata filtering narrows retrieval before similarity even enters the picture: filter by source, date range, category, or user permissions.
• Hybrid search combines vector similarity with keyword matching like BM25. Vectors are great at meaning ("why does my bill look wrong") and terrible at exact tokens (order ID "AX-2291"), while keyword search is the opposite. Together they cover both. Add a reranker on top and you can retrieve a bit wider, then cut down to the few chunks that actually help.

4. Context overflow

Overflow is the quantity version of the noise problem. Every model has a context limit, and even models with huge windows get slower, more expensive, and less focused as the prompt grows. Stuffing every retrieved chunk into the prompt is the most common mistake here, usually done out of fear of missing something.

The fix is retrieval discipline:

- Keep chunks small and self-contained, so fewer tokens carry more meaning.
- Retrieve a reasonable candidate set, then rerank and keep only the top few.
- Set a hard token budget for context and enforce it in code, not in hope.
- Compress when possible: extract only the relevant sentences, or summarize passages before prompting.

If noise is "the wrong chunks," overflow is "too many chunks." Both are solved by being selective about what earns a place in the prompt.

5. Hallucination and missing guardrails

Modern models hallucinate far less than earlier ones, and with good retrieval the rate drops further. But it still happens, usually when the context is thin, contradictory, or off-topic. The model would rather sound confident than admit a gap, it’s especially more prominent in Anthropic models.

Some simple fixes like:

• Instructing the model to answer only from the provided context and to say "I don't know".
• Asking it to cite which chunk supports each claim.
• Keeping temperature low for factual tasks.

The bigger production concern is what comes in and what goes out. Retrieved documents can contain injected instructions ("ignore previous instructions and..."), and user prompts certainly will. And your corpus almost certainly holds sensitive data that should never reach the model or the user.

So put guardrails in place:

• an injection filter on incoming prompts and retrieved content.
• PII redaction before the prompt is built, and a grounded output policy.

Tools like NeMo Guardrails make this practical, and the OWASP Top 10 for LLM apps is a solid checklist.

TLDR;

RAG fails in the boring parts: splitting, embedding, filtering, budgeting, guarding. Get those five right and the model handles the easy part.

If you liked reading this article, you can read more of my writings here.

You can follow me on X and LinkedIn to see more of my content.

Happy Building : )