Retrieval Quality: When the Right Chunk Never Surfaces

The Answer Was There the Whole Time

I spent an embarrassing amount of time debugging a RAG pipeline once, convinced the embedding model was broken. My chatbot kept giving vague, off-topic answers even though I knew the right information was sitting in the document store. I'd scroll through the source docs and find exactly what the user needed — but the retrieval step just kept pulling up the wrong chunks.

Turns out the embedding model was fine. The retrieval strategy was the problem. And fixing it didn't require retraining anything or swapping out my vector database. It just required understanding three techniques that I wish someone had explained to me on day one: re-ranking, hybrid search, and metadata filters.

If your RAG system keeps missing the obvious answer, this post is for you.

Why Good Chunks Don't Always Float to the Top

Here's the thing about semantic search — it's powerful, but it's not magic. When you embed a query and search for similar vectors, you're essentially asking: "which chunks sound like this question?" That works great most of the time. But it breaks down in a few predictable ways:

Vocabulary mismatch. The user asks about "car insurance costs" but your doc talks about "automobile coverage premiums." Semantically similar, but the vectors might not be close enough.

Chunk placement. The most relevant sentence is buried in the middle of a chunk that starts with something unrelated. The chunk-level embedding averages out the meaning, diluting the good stuff.

Popularity bias. Some chunks get retrieved constantly because they're semantically broad. They crowd out the specific, accurate chunk you actually need.

The good news: each of these problems has a targeted fix.

Fix #1: Hybrid Search (Keyword + Semantic Together)

Hybrid search combines traditional keyword (BM25) search with semantic vector search. Instead of picking one, you run both and merge the results. This solves the vocabulary mismatch problem because keyword search doesn't care about meaning — it just looks for exact or near-exact word matches.

Most modern vector databases support this out of the box. Here's what it looks like conceptually in Python using a library like weaviate:

python
# Hybrid search: alpha controls the blend
# alpha=1.0 = pure semantic, alpha=0.0 = pure keyword
results = client.query.get("Document", ["content", "title"])
  .with_hybrid(
    query="car insurance costs",
    alpha=0.5  # 50/50 blend
  )
  .with_limit(10)
  .do()

The alpha parameter is your tuning knob. Start at 0.5 and adjust based on your domain. Technical documentation with precise terminology? Lean more toward keyword (lower alpha). Conversational knowledge bases? Lean semantic (higher alpha).

Quick Win

If you're using Pinecone, Weaviate, or Qdrant, hybrid search is likely already available — check their docs for the "sparse + dense" or "hybrid" query option. You might just need to flip one parameter.

Fix #2: Metadata Filters (Narrow Before You Search)

Imagine asking your RAG system a question about your 2024 product roadmap, and it returns chunks from a 2021 strategy doc. Semantically, they're similar topics. But the answer is wrong. This is where metadata filters save you.

Metadata filters let you pre-filter your vector store before semantic search runs. You attach structured data to each chunk at index time — things like date, department, document type, author — and then include filter conditions in your query.

python
# Indexing a chunk with metadata
index.upsert(vectors=[
  {
    "id": "chunk_001",
    "values": embedding_vector,
    "metadata": {
      "year": 2024,
      "doc_type": "roadmap",
      "department": "product"
    }
  }
])

# Querying with a metadata filter
results = index.query(
  vector=query_embedding,
  filter={
    "year": {"$gte": 2023},
    "doc_type": {"$eq": "roadmap"}
  },
  top_k=5
)

The trick is thinking about metadata before you index. Ask yourself: "What dimensions would someone use to narrow down this question?" Date, source, category, author, status — these all make great filter fields. I once added a simple doc_status field (draft vs. published) and immediately stopped getting answers sourced from half-finished internal notes.

Design Tip

Think of metadata filters like SQL WHERE clauses — they don't find the best chunk, they eliminate the wrong ones. Use them to reduce the search space, then let semantic search find the winner.

Fix #3: Re-ranking (A Second Opinion on Your Results)

This is the technique that changed everything for me. Re-ranking is a two-step process: first, you retrieve a broader set of candidates (say, top 20 chunks) using your normal vector search. Then you pass those candidates plus the original query to a cross-encoder model that scores each one for actual relevance — not just vector similarity.

The difference matters because vector search asks "does this chunk look like the query?" while a cross-encoder asks "does this chunk actually answer the query?" Those are different questions.

The most popular re-ranker right now is Cohere's rerank API, but you can also use open-source models from Hugging Face. Here's the pattern:

python
# Step 1: Broad retrieval (get more than you need)
candidates = vector_store.search(query, top_k=20)

# Step 2: Re-rank with Cohere
import cohere
co = cohere.Client("YOUR_API_KEY")

reranked = co.rerank(
  query="What is our refund policy for digital products?",
  documents=[c["content"] for c in candidates],
  top_n=5,  # Only keep the best 5
  model="rerank-english-v3.0"
)

# Now use reranked.results[0...4] as your context

The latency cost is real — you're making an extra API call. But for most use cases, it's worth it. You can also use lightweight local re-rankers if latency is a dealbreaker. The cross-encoder/ms-marco-MiniLM-L-6-v2 model from Sentence Transformers runs fast and free on your own machine.

Putting It All Together: The Retrieval Stack

You don't have to use all three techniques at once, but when you combine them you get something genuinely powerful. Think of it as a funnel:

Step 1 — Filter. Apply metadata filters to eliminate irrelevant documents before search. Cut your search space from 100k chunks down to 10k.

Step 2 — Hybrid search. Run semantic + keyword search against the filtered set. Retrieve the top 20-50 candidates.

Step 3 — Re-rank. Pass the candidates to a cross-encoder. Keep the top 3-5. Send those to the LLM as context.

This three-stage funnel is what production RAG systems actually look like at companies that ship reliable AI features. The vector search step is fast but imprecise. The re-ranker is slower but precise. You use speed where it's acceptable and precision where it counts.

Start Small

If you're just getting started, add re-ranking first. It's the single highest-impact change you can make to an existing RAG pipeline, and you don't need to change how you index anything.

How to Know If It's Actually Working

Don't just eyeball it. When I started actually measuring retrieval quality, I found improvements I would've missed — and problems I would've attributed to the wrong thing.

A simple way to test: build a small "golden set" of 10-20 question-answer pairs where you know the right source chunk. Then check what rank that chunk appears at in your retrieval results. The metric you want is called MRR (Mean Reciprocal Rank) — basically, on average, how high does the right answer appear?

python
# Simple MRR calculation
def mean_reciprocal_rank(queries_with_expected):
  scores = []
  for query, expected_chunk_id in queries_with_expected:
    results = retrieve(query)  # your retrieval fn
    ids = [r["id"] for r in results]
    if expected_chunk_id in ids:
      rank = ids.index(expected_chunk_id) + 1
      scores.append(1 / rank)
    else:
      scores.append(0)
  return sum(scores) / len(scores)

# MRR of 1.0 = always ranked #1
# MRR of 0.5 = right chunk usually at position 2

Run this before and after each change. You'll know immediately whether hybrid search or re-ranking actually helped for your specific data.

The Real Lesson Here

When a RAG system gives bad answers, most people assume the LLM is the problem. But in my experience, the retrieval layer is wrong at least as often. The model can only work with what you hand it — garbage in, garbage out.

Re-ranking, hybrid search, and metadata filters are your three tools for making sure the right chunk actually surfaces. Start with re-ranking (easiest to add, biggest single improvement), layer in hybrid search if vocabulary mismatch is a problem, and add metadata filters once you know what dimensions matter for your use case.

Your answers will get noticeably better. And you'll stop blaming the model for something the retrieval pipeline was doing all along.

Keep going

More tutorials in this category, or explore the full field guide.

More AI Coding Tutorials Official Docs ↗