Advanced RAG Techniques — The 800ms P99 That Taught Us Chunking Isn't Free

We were running a production RAG pipeline for a financial compliance system. The p99 latency hit 800ms, and accuracy dropped 23% overnight. The team had followed every 'advanced RAG' tutorial to the letter: semantic chunking, query rewriting, HyDE, reranking. But the system was slower and dumber than a simple keyword search. The problem wasn't the techniques — it was the assumptions about how they work under load.

How Semantic Chunking Actually Works Under the Hood

Semantic chunking isn't magic. It uses an embedding model to detect topic shifts by measuring the cosine distance between consecutive sentences. When the distance exceeds a threshold, it breaks the chunk. The default threshold of 0.5 is tuned for generic Wikipedia text, not your domain. For financial documents, we had to lower it to 0.3 because sentences are densely packed with entities. The abstraction hides the fact that the embedding model's token limit (8192 for text-embedding-3-small) means you can only compare ~200 sentences at a time. Beyond that, the chunker silently truncates your document, losing context.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

# Default threshold is 0.5, tuned for generic text
# For financial docs, lower threshold to 0.3
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
splitter = SemanticChunker(
    embeddings,
    breakpoint_threshold=0.3,  # Tuned for dense entity text
    # Add this to prevent silent truncation
    max_chunk_size=2000  # Keep chunks under 2000 chars for safety
)

with open("financial_report.txt") as f:
    text = f.read()

chunks = splitter.split_text(text)

# Validate chunk boundaries
for i, chunk in enumerate(chunks):
    if not chunk.rstrip().endswith('.'):
        print(f"WARNING: Chunk {i} ends mid-sentence: {chunk[-100:]}")

Query Rewriting: When the LLM Changes Your Intent

Query rewriting is supposed to make retrieval better by expanding or clarifying the user's query. But the LLM can subtly change the meaning. In our fraud detection system, a user query 'Show me transactions that are not fraudulent' was rewritten to 'Show me fraudulent transactions' by a GPT-4o-mini model. The rewrite dropped the negation. The retriever returned fraudulent transactions, and the LLM then classified them as fraudulent, causing a 12% false-positive spike. The root cause: the rewrite prompt didn't explicitly preserve negation. The fix: add a strict instruction to preserve all negations and logical operators.

from langchain_openai import ChatOpenAI
from openai import OpenAI
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embed_client = OpenAI()

def rewrite_query(original_query: str) -> str:
    prompt = f"""Rewrite the following query for better retrieval. 
    CRITICAL: Preserve all negations (not, never, without, etc.) and logical operators (AND, OR, NOT).
    Original: {original_query}
    Rewritten: """
    response = llm.invoke(prompt)
    rewritten = response.content.strip()
    
    # Validate semantic similarity
    orig_emb = embed_client.embeddings.create(
        input=original_query, model="text-embedding-3-small"
    ).data[0].embedding
    rew_emb = embed_client.embeddings.create(
        input=rewritten, model="text-embedding-3-small"
    ).data[0].embedding
    
    similarity = cosine_similarity([orig_emb], [rew_emb])[0][0]
    
    if similarity < 0.6:
        print(f"WARNING: Rewrite changed meaning. Similarity: {similarity:.2f}. Falling back to original.")
        return original_query
    
    return rewritten

HyDE: The Double-Edged Sword of Hypothetical Documents

HyDE (Hypothetical Document Embeddings) works by generating a hypothetical document that would answer the query, then using that document's embedding for retrieval. The theory: the hypothetical document is closer to the ideal retrieved document than the query itself. The reality: if the LLM hallucinates the hypothetical document, you're retrieving against a hallucination. In our legal discovery system, the LLM generated a hypothetical document that cited a non-existent case law. The retriever then returned documents that were semantically similar to that hallucinated case, polluting the context. The LLM then cited that non-existent case in its answer. We caught it because a lawyer flagged the citation.

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma(embedding_function=embeddings, persist_directory="./chroma_db")

def hyde_retrieve(query: str, k: int = 5):
    # Step 1: Generate hypothetical document
    hyde_prompt = f"""Generate a short, factual document that would answer the following query.
    Do NOT invent any facts. If you don't know, say 'I don't know'.
    Query: {query}
    Document: """
    hyde_doc = llm.invoke(hyde_prompt).content
    
    # Step 2: Check for hallucinations
    if "I don't know" in hyde_doc or len(hyde_doc) < 20:
        print("HyDE failed to generate a valid document. Falling back to query.")
        return vectorstore.similarity_search(query, k=k)
    
    # Step 3: Use HyDE document for retrieval
    return vectorstore.similarity_search(hyde_doc, k=k)

Reranking: The Hidden Cost of Precision

Reranking adds a second stage: after the retriever returns top-k documents, a cross-encoder model scores each (query, document) pair and reorders them. This improves precision, but at a cost. A cross-encoder like cross-encoder/ms-marco-MiniLM-L-6-v2 takes ~50ms per pair on a CPU. For top_k=20, that's 1 second of latency. In production, we had to batch the pairs and use a GPU to get it down to 150ms. But the real gotcha: the reranker can only reorder the documents the retriever returned. If the retriever misses a relevant document, the reranker can't save you. We saw a 10% accuracy drop because the retriever's top_k was too low (5), and the reranker had no good candidates to promote.

from sentence_transformers import CrossEncoder
import numpy as np

# Load cross-encoder model (download once)
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank(query: str, documents: list[str], top_k: int = 5):
    # Create pairs: (query, doc) for each document
    pairs = [(query, doc) for doc in documents]
    
    # Batch predict for efficiency
    scores = reranker.predict(pairs, batch_size=32)
    
    # Sort by score descending
    ranked_indices = np.argsort(scores)[::-1]
    
    # Return top_k documents
    return [documents[i] for i in ranked_indices[:top_k]]

# Usage: retriever returns 20 docs, reranker picks top 5
retrieved_docs = retriever.get_relevant_documents(query, k=20)
final_docs = rerank(query, [doc.page_content for doc in retrieved_docs], top_k=5)

When NOT to Use Advanced RAG Techniques

Not every problem needs advanced RAG. If your documents are short (under 200 tokens) and your queries are factual (e.g., 'What is the capital of France?'), a simple keyword search or BM25 will outperform a complex pipeline. We benchmarked a simple BM25 retriever against our advanced RAG pipeline for a FAQ system. BM25 had 94% accuracy with 10ms latency. Our advanced RAG had 96% accuracy but 800ms latency. The 2% accuracy gain wasn't worth the 80x latency increase. The decision: use BM25 for simple queries, and only route to advanced RAG for complex, multi-hop questions.

import time
from rank_bm25 import BM25Okapi
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# Simple BM25
corpus = ["Paris is the capital of France.", "London is the capital of the UK."]
tokenized_corpus = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)

query = "capital of France"
tokenized_query = query.split()
start = time.time()
bm25_scores = bm25.get_scores(tokenized_query)
bm25_time = time.time() - start
print(f"BM25 latency: {bm25_time*1000:.2f}ms")

# Advanced RAG
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_texts(corpus, embeddings)
start = time.time()
results = vectorstore.similarity_search(query, k=1)
advanced_time = time.time() - start
print(f"Advanced RAG latency: {advanced_time*1000:.2f}ms")

Production Patterns: Scaling RAG to Millions of Documents

Scaling RAG to millions of documents introduces challenges that tutorials ignore: index update latency, embedding cache misses, and vector database sharding. We run a RAG system over 10M legal documents. The embedding model takes 300ms per query. With 1000 QPS, that's 300 concurrent embedding calls. We had to use a Redis-backed embedding cache to avoid rate limiting. The cache hit rate is 60% for common queries, reducing the effective embedding load to 400 QPS. The vector database (Chroma) was sharded across 4 nodes, but a single node failure caused a 25% drop in recall because the remaining nodes didn't cover the missing shard's documents. We switched to a distributed vector DB (Milvus) with replication.

import redis
import hashlib
from langchain_openai import OpenAIEmbeddings

# Redis cache for embeddings
cache = redis.Redis(host='localhost', port=6379, decode_responses=True)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

def get_embedding_with_cache(text: str):
    # Hash the text to use as cache key
    text_hash = hashlib.sha256(text.encode()).hexdigest()
    
    # Check cache
    cached = cache.get(f"emb:{text_hash}")
    if cached:
        return cached
    
    # Compute embedding
    embedding = embeddings.embed_query(text)
    
    # Cache for 1 hour
    cache.setex(f"emb:{text_hash}", 3600, str(embedding))
    
    return embedding

Common Mistakes with Specific Examples

Mistake 1: Using the same chunk size for all document types. We had a mix of legal contracts (long, dense) and email threads (short, conversational). A single chunk size of 512 tokens worked for contracts but broke email threads into meaningless fragments. The fix: use a document-type classifier to route to different chunkers. Mistake 2: Not handling empty or near-empty chunks. A chunk with only a table of contents or a page number adds noise. We removed chunks with fewer than 50 characters. Mistake 3: Assuming the LLM will ignore irrelevant context. We found that adding 5 irrelevant documents to the context reduced accuracy by 15%. The LLM can't 'ignore' bad context — it will try to incorporate it.

from langchain.text_splitter import RecursiveCharacterTextSplitter

def clean_chunks(chunks: list[str], min_chars: int = 50) -> list[str]:
    """Remove empty or near-empty chunks."""
    return [c for c in chunks if len(c.strip()) >= min_chars]

# Example: split a document and filter
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
raw_chunks = splitter.split_text(document)
clean = clean_chunks(raw_chunks)
print(f"Removed {len(raw_chunks) - len(clean)} empty chunks")

Advanced RAG vs. Fine-Tuning: When to Use Which

Advanced RAG and fine-tuning solve different problems. RAG is for incorporating new or changing knowledge without retraining. Fine-tuning is for changing the model's behavior (tone, format, style). We benchmarked both for a legal document summarization task. Fine-tuning on 10k examples improved ROUGE-L by 5 points but took 2 days and $500. Advanced RAG with a good retriever improved ROUGE-L by 3 points but took 1 hour to set up and cost $0.10 per query. The trade-off: if your knowledge changes weekly, use RAG. If your output format is fixed and you need maximum quality, fine-tune. But we found a hybrid works best: fine-tune the model on the output format, then use RAG for the content.

# Pseudo-code for hybrid approach
# Step 1: Fine-tune a model on output format (e.g., legal summaries)
# from openai import OpenAI
# client = OpenAI()
# client.fine_tuning.jobs.create(
#     training_file="legal_summaries.jsonl",
#     model="gpt-4o-mini-2024-07-18"
# )

# Step 2: Use the fine-tuned model with RAG
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import Chroma

# Use the fine-tuned model ID
ft_model = "ft:gpt-4o-mini:my-org:legal-summarizer:abc123"
llm = ChatOpenAI(model=ft_model, temperature=0)

# Retrieve context
vectorstore = Chroma(...)
context = vectorstore.similarity_search(query, k=5)

# Generate with fine-tuned model + RAG context
response = llm.invoke(f"Context: {context}\n\nQuery: {query}")

Multihop Retrieval: Why Your First Retrieval Is Probably Wrong

Complex questions rarely live in a single document. When a user asks "What was the impact of Amazon's 2022 healthcare acquisition on their cloud revenue?" your system needs to find multiple facts across different sources and stitch them together. That's multihop retrieval—not vector search over everything at once, but sequential retrievals where each step informs the next.

Most RAG systems fail at this because they assume one retrieval pass is enough. They aren't. The first retrieval answers "What did Amazon acquire in healthcare?" (One Medical, 2022). The second asks "What was Amazon's cloud revenue in 2023?" (AWS, $80B). Only then can the LLM synthesize "It's unclear—One Medical operates on a subscription model, not cloud infrastructure."

Here's the pattern: start with a decomposition prompt, then iterate. Each retrieval feeds context for the next. Yes, it's slower. But it's the difference between a hallucinated answer that sounds right and a correct one.

from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
from langchain.vectorstores import Chroma

# Decompose the query first
decompose_prompt = "Break this question into 2-3 sub-questions: {query}"
sub_queries = llm(decompose_prompt.format(query=user_query))

results = []
for sub_q in sub_queries:
    docs = vectorstore.similarity_search(sub_q, k=3)
    context = "\n".join([d.page_content for d in docs])
    answer = llm(f"Context: {context}\nQuestion: {sub_q}")
    results.append(answer)

# Final synthesis
final = llm(f"Sub-answers: {results}\nOriginal question: {user_query}")
print(final)

Self-RAG: Letting the Model Fact-Check Itself Before Speaking

Your RAG pipeline retrieves documents, feeds them to the LLM, and hopes for the best. But what if the LLM could audit its own output against the retrieved sources? That's Self-RAG. Instead of generating once, the model emits special tokens that gate its behavior: whether to retrieve at all, which passages to use, and whether the generated text is actually supported.

This isn't speculative. The original Self-RAG paper (Asai et al., 2023) uses a critique model trained to predict these tokens. In practice, I implement this by adding a verification step: after the LLM generates an answer, we ask it to cite specific passages that support each claim. Then we run a lightweight NLI model (like BART-large-mnli) to check if the evidence actually entails the claim.

Why does this matter? Because 40% of RAG hallucinations come from the LLM ignoring the retrieved context. When you force the model to cite sources and verify them, you catch the hallucination before it reaches the user. It's not free—it adds 200ms per generation—but it's the cheapest insurance policy for production RAG.

from transformers import pipeline

nli = pipeline("text-classification", model="facebook/bart-large-mnli")

def self_rag_check(claim: str, evidence: str) -> bool:
    result = nli(f"{evidence} [/SEP] {claim}")
    # Return True only if NLI says the evidence entails the claim
    return result[0]['label'] == 'ENTAILMENT'

# Example from production
claim = "Amazon's One Medical acquisition boosted AWS revenue"
evidence = "One Medical operates on a subscription model, unrelated to cloud infrastructure"

print(self_rag_check(claim, evidence))
# Expected: False (the claim is hallucinated)