Building a RAG Pipeline End to End

Why I Finally Stopped Avoiding RAG

For the longest time, RAG (Retrieval-Augmented Generation) felt like one of those acronyms people throw around at conferences to sound smart. I knew it had something to do with making AI smarter about your own documents, but every tutorial I found either hand-waved the hard parts or assumed I already had a PhD in embeddings.

Then I had a real problem: I had about 200 pages of internal company docs and I kept asking ChatGPT questions it couldn't answer because it had never seen them. Copy-pasting chunks manually felt ridiculous. That's when I finally sat down and built a RAG pipeline from scratch — and honestly, once you see each piece click together, it's not as scary as it sounds.

This article walks you through a complete, working pipeline you can run locally today. We're going six stages deep: ingest, chunk, embed, index, retrieve, generate. Let's go.

The Big Picture: What RAG Actually Does

Before we write a single line of code, here's the mental model. A RAG pipeline answers questions by first finding relevant content, then handing that content to an LLM to generate an answer. The LLM doesn't memorize your documents — it reads the relevant snippets at query time.

Think of it like an open-book exam. The LLM is the student, your document index is the textbook, and the retriever is the student flipping to the right page before writing their answer. Six stages, one flow:

We'll use Python, LangChain, ChromaDB for the vector store, and Ollama to run a local LLM. Everything stays on your machine — no API keys required (though I'll note where you could swap in OpenAI if you prefer).

Step 0: Set Up Your Environment

First things first. Create a virtual environment and install our dependencies.

You'll also need Ollama installed and a model pulled locally. I'm using llama3 but mistral works great too.

Stage 1: Ingest — Load Your Documents

Ingestion is just loading your raw source material. We'll use a folder of PDF files, but this works with plain text, Markdown, HTML, whatever you've got.

Drop any PDFs into a ./docs folder and run this. LangChain's loaders handle the messy PDF parsing for you. If you're working with plain text files, swap in DirectoryLoader instead.

Stage 2: Chunk — Split Into Digestible Pieces

Here's where a lot of beginners (including past me) make their first mistake. You can't just shove entire documents into a vector store — you need to break them into smaller chunks that are semantically meaningful. Too big and retrieval gets noisy. Too small and you lose context.

I've found 500 tokens with a 50-token overlap works well for most docs. The overlap ensures sentences that straddle chunk boundaries don't get lost.

Stage 3: Embed — Convert Text to Vectors

Embeddings are how we make text mathematically comparable. Each chunk gets converted into a list of numbers (a vector) that captures its meaning. Similar chunks end up close together in vector space — which is exactly what lets us find relevant chunks later.

We'll use sentence-transformers locally. It downloads a small model the first time and runs fast after that.

384 dimensions means every piece of text is represented as a point in 384-dimensional space. Wild, right? But it works beautifully.

Stage 4: Index — Store Vectors in ChromaDB

Now we persist everything. ChromaDB is a local vector database — think SQLite but for embeddings. It stores your chunks and their vectors, and survives restarts so you don't re-embed every time.

Run this once. From now on, you can load the existing index without re-embedding — just use Chroma(persist_directory="./chroma_db", embedding_function=embeddings) to reload it.

Stage 5 & 6: Retrieve and Generate — The Full Pipeline

This is where it all comes together. We take a user question, find the most relevant chunks from our index, and feed both the question and those chunks to the LLM. The LLM answers based on what it actually retrieved — not what it hallucinated from training data.

The k=4 parameter means we retrieve the 4 most relevant chunks. The return_source_documents=True flag is something I always enable — it shows you exactly which chunks the LLM used to answer, which is huge for debugging bad answers.

Putting It All Together as One Script

Here's a minimal end-to-end script that runs all six stages. Drop your PDFs in ./docs, run this once to build the index, then comment out the build section and just use the query part.

Common Failures and How to Fix Them

When I first built this, my answers were garbage for about two hours until I figured out these gotchas:

Bad answers despite good documents? Check your chunk size. If chunks are too small, the LLM doesn't have enough context to form a coherent answer. Bump chunk_size to 800 and try again.

"I don't know" when the answer is clearly in the docs? Your retrieval might be off. Try increasing k to 6 or 8. Also print the retrieved chunks to see if they're actually relevant — sometimes the question phrasing doesn't match the document phrasing well.

LLM making things up? Your system prompt is your main lever here. Make it very explicit: "Answer only from the provided context. If the context doesn't contain the answer, say 'I don't have that information.'"

Where to Go From Here

This pipeline is a solid foundation, but there's a lot of room to grow. A few directions I'd explore next:

Hybrid search: Combine vector similarity with keyword search (BM25) for better retrieval on exact terms like product names or error codes.

Reranking: Use a cross-encoder model to re-score your top-k results before sending them to the LLM. Small effort, big quality improvement.

Metadata filtering: Store document metadata (date, author, category) in ChromaDB and filter by it at query time. Useful when you want to answer questions only from docs published after a certain date.

Swap in OpenAI: Replace Ollama with ChatOpenAI from LangChain and use OpenAIEmbeddings for production-quality results.

Building this pipeline was genuinely one of those "aha" moments in my AI learning journey. The first time it correctly answered a question about a document I'd ingested, from scratch, on my own machine — no cloud, no API, just local models and a vector database — I actually said "wait, that worked?" out loud. That feeling is worth the setup time. Give it a shot.