Build a RAG-Powered Knowledge Base with Smart Sampling Controls

Build a full RAG pipeline web app with a live knowledge base, MCP transport toggle, and sampling parameter controls — all prompted into existence with AI.

What You'll Build

By the end of this project you'll have a fully functional knowledge base app that ties together three of the most powerful concepts in modern AI tooling — RAG pipelines, MCP transport layers, and sampling controls. You'll be prompting the AI to build every piece of it, step by step.

  • A document ingestion panel where you paste or upload text into your knowledge base
  • A RAG retrieval engine that finds the most relevant chunks and injects them into your prompt
  • An MCP transport toggle so you can switch between stdio and HTTP connection modes
  • Live temperature and Top-P sliders that adjust AI sampling behaviour before each query
  • A results panel that shows the AI answer alongside the retrieved context chunks it used
  • A query log that records every question, its sampling settings, and the retrieved chunks

What You'll Need

Claude or ChatGPT

Claude 3.5 Sonnet or GPT-4o both work great. Use whichever you have access to — the prompts work with either.

A Text Editor

VS Code is ideal, but any editor that can save HTML, CSS, and JavaScript files will do the job perfectly.

Build It: Step by Step

Each step gives you a prompt to paste into your AI tool. You're directing the AI like a contractor — you describe what you want, it writes the code, and you put the pieces together.

1

Step 1: Build the Knowledge Base Shell

Start by prompting the AI to build the full HTML/CSS/JS shell for your app. This gives you the layout — a sidebar for document ingestion, a main query panel, and a results area — before any logic is wired up. Getting the structure right first means you won't have to fight the layout later when the complex logic arrives.

Paste this prompt into Claude or ChatGPT:

Build me a single-file HTML/CSS/JS app called "RAG Knowledge Base" with a dark theme using slate and violet colour accents.

The layout should have three sections:
1. LEFT SIDEBAR (280px wide): A "Knowledge Base" panel with a large textarea for pasting in documents, a text field for a document title, and an "Add to Knowledge Base" button. Below that, show a list of added documents with their titles.
2. MAIN PANEL: A query input field at the top with a "Query" button, and a results area below that starts empty.
3. RIGHT SIDEBAR (260px wide): A "Sampling Controls" panel with a Temperature slider (0.0 to 2.0, default 0.7), a Top-P slider (0.0 to 1.0, default 0.9), and a dropdown to select MCP Transport Mode (options: "stdio" and "HTTP"). Show the current value of each slider as a number next to it.

Add a header bar across the top with the app title and a small status badge that says "Ready".

All sections should be visible at once on a wide screen. Use CSS Grid for the three-column layout. Style the sliders to match the dark theme. Don't wire up any logic yet — just build the shell with placeholder text in the results area that says "Your answer will appear here." Make it look polished.

What to look for

You should get a complete, styled HTML file with three visible columns. The sliders should show their current values updating as you drag them. The document list in the sidebar should exist as a placeholder. If the layout collapses into a single column, ask the AI to fix the CSS Grid to enforce three columns on wide screens.

2

Step 2: Add the RAG Chunking and Retrieval Engine

Now you'll prompt the AI to wire up the core RAG logic — document chunking, simple TF-IDF-style keyword scoring, and chunk retrieval. This is where the RAG pipeline tutorial pays off: you understand that retrieval is about finding the right context, not just passing the whole document. The retrieval engine will run entirely in the browser using JavaScript.

Paste this prompt into your AI tool, following on from Step 1:

Now add a RAG retrieval engine to the app in JavaScript. Here's exactly what I need:

1. CHUNKING: When a user clicks "Add to Knowledge Base", split the document text into overlapping chunks of roughly 200 words each, with a 40-word overlap between consecutive chunks. Store each chunk as an object with: { id, documentTitle, text, wordCount }. Keep all chunks in a `knowledgeBase` array.

2. RETRIEVAL: When the user submits a query, score every chunk using a simple keyword relevance function: tokenise both the query and the chunk text into lowercase words (strip punctuation), then count how many unique query tokens appear in the chunk text. The score is: matchingTokens / totalQueryTokens. Return the top 3 chunks by score, or fewer if the knowledge base has fewer than 3 chunks.

3. DISPLAY: After retrieval, show the top chunks in the results area BEFORE the final answer placeholder. Each chunk should appear in a card with: the document title, the chunk text, and the relevance score formatted as a percentage. Use a cyan accent colour for these chunk cards.

4. Update the document list in the sidebar to show each document's title and number of chunks it produced.

Don't call any external API yet. Just show "[API response will appear here based on retrieved context]" as a placeholder after the chunk cards.

What to look for

Add a test document — paste in a few paragraphs of any text — then query something related to it. You should see cyan-bordered chunk cards appear in the results area with relevance percentages. If chunks aren't appearing, ask the AI to add a console.log inside the retrieval function so you can debug from the browser console.

3

Step 3: Wire Up the MCP Transport Layer Switcher

Here's where the MCP transport layer knowledge comes in. You won't be building an actual MCP server in this step, but you'll build a transport simulation layer that shows users exactly what changes between stdio and HTTP modes — the connection method, the message format, and the request payload structure. This teaches the concept interactively, which is the whole point.

Paste this prompt into your AI tool:

Add an MCP Transport Layer simulation to the app. When the user selects a transport mode from the dropdown (stdio or HTTP), the app should behave differently in the following ways:

STDIO MODE:
- Show a small terminal-style panel below the transport dropdown labelled "stdio stream"
- When a query is submitted, animate lines appearing in this panel as if messages are being exchanged line by line, like: "> {type: 'request', method: 'query', params: {query: '...', topChunks: 3}}"
and "< {type: 'response', status: 'ok', chunkCount: 3}"
- Use a monospace font and a green-on-dark terminal aesthetic for this panel
- Show a status badge next to the dropdown that says "stdio connected"

HTTP MODE:
- Replace the terminal panel with a "Request Preview" card that shows a formatted JSON object representing the HTTP POST body that would be sent, including the query, retrieved chunk IDs, temperature value, and top_p value
- Show a status badge that says "HTTP ready" with a blue colour
- Also show a mock response header: HTTP/1.1 200 OK, Content-Type: application/json

Both modes should update live whenever the user changes the transport dropdown. The simulation should NOT make any real network calls. Label everything clearly so users understand this is showing them what each transport mode would look like in a real MCP integration.

What to look for

Switch between stdio and HTTP in the dropdown — the right sidebar should visibly change between the terminal panel and the JSON request preview. Run a query and check that the stdio panel animates line by line. If the animation feels too fast, ask the AI to slow the line-by-line delay to 120ms per line so users can actually read it.

4

Step 4: Make the Sampling Controls Actually Do Something

Now you'll make the temperature and Top-P sliders affect the app's behaviour in a meaningful, visible way. Since we're building a self-contained browser app without a real API key, you'll prompt the AI to build a response simulator that uses the sampling values to generate responses with visibly different characteristics — so users can see and feel what high temperature versus low temperature actually means.

Paste this prompt into your AI tool:

Make the Temperature and Top-P sliders affect the generated answer in the results panel. Here's how:

1. RESPONSE SYNTHESIS: Build a `synthesiseAnswer(query, chunks, temperature, topP)` function. It should:
   - Combine the text of the top retrieved chunks into a context string
   - Generate a response by extracting and reassembling sentences from those chunks that are most relevant to the query
   - Use the temperature value to control variation: at temperature 0.0-0.3, always return the single most relevant sentence from each chunk verbatim. At 0.4-0.8, paraphrase lightly. At 0.9-2.0, add clearly labelled "[high temperature: speculative]" elaborations that extend beyond the retrieved text.
   - Use the topP value to control how many chunks contribute: topP below 0.5 means only use the single highest-scoring chunk, 0.5-0.8 uses the top 2, above 0.8 uses all 3.

2. ANSWER DISPLAY: Show the final synthesised answer in a clearly labelled "Answer" card below the chunk cards. Use an amber accent colour for this card. Include a small metadata line showing: Temperature: X | Top-P: X | Chunks used: X | Transport: stdio/HTTP

3. SAMPLING EXPLANATION: Below the answer, add a collapsible "Why this response?" section that explains in plain English what effect the current temperature and Top-P values had on the answer. For example: "Temperature 0.2 = precise and literal. Top-P 0.9 = drawing from all top chunks."

Update this explanation live whenever the sliders change, even without resubmitting a query.

What to look for

Run the same query at temperature 0.1 and then again at temperature 1.5 — the answers should feel noticeably different. The amber answer card should appear below the cyan chunk cards. The "Why this response?" collapsible should update as you move the sliders without resubmitting. If the explanation isn't updating live, ask the AI to add oninput event listeners to both slider elements.

5

Step 5: Add the Query Log and Final Polish

The final step ties everything together with a persistent query log and some quality-of-life polish. The log gives users a history of every query they ran, what sampling settings were active, and which chunks were retrieved — turning the app into a genuine learning tool for experimenting with RAG and sampling behaviour side by side.

Paste this final prompt into your AI tool:

Add these final features to complete the app:

1. QUERY LOG: Add a collapsible "Query Log" section at the bottom of the main panel. Every time a query is submitted, append a log entry containing: timestamp, the query text, temperature value, Top-P value, transport mode, number of chunks retrieved, top chunk relevance score, and the first 100 characters of the answer. Show the log as a scrollable table with these columns. Add a "Clear Log" button and an "Export as JSON" button that downloads the log entries as a JSON file.

2. KNOWLEDGE BASE STATS: In the left sidebar below the document list, add a small stats panel showing: total documents, total chunks, average chunk length in words, and a bar showing how full the "context window" is (treat 20 chunks as the max and show a progress bar).

3. STATUS BADGE: Update the header status badge dynamically: show "Ready" when idle, "Retrieving..." briefly during chunk retrieval, and "Answer ready" after synthesis completes. Use green for ready, amber for retrieving, and brand green for answer ready.

4. EMPTY STATE: If the user submits a query but the knowledge base is empty, show a friendly empty state in the results area with an icon and the message: "No documents in your knowledge base yet. Add some text in the left sidebar to get started."

5. FINAL POLISH: Add a subtle animation (CSS keyframe fade-in) when new chunk cards or the answer card appear. Make the query input submit on Enter key as well as the button click. Add a tooltip to each slider explaining what it does in one sentence.

Test the full flow works end to end and fix any bugs you spot.

What to look for

Run three or four different queries and check that the log table populates correctly with each one. Click "Export as JSON" and open the downloaded file — all your query history should be there in a clean JSON array. Try submitting a query with no documents added and confirm the empty state appears instead of an error.

Common Issues

The three-column layout collapses on my screen

This usually means the CSS Grid isn't enforcing minimum column widths. Ask the AI: "Fix the CSS Grid layout so the three columns never collapse below their set widths. Use grid-template-columns: 280px 1fr 260px and add overflow-x: auto to the wrapper if needed."

No chunks appear when I submit a query

Check the browser console for errors first. Often this means the chunking function isn't being called, or the knowledge base array is empty. Ask the AI to "add console.log(knowledgeBase) inside the query handler so I can see if documents are being stored correctly."

The stdio animation doesn't appear

The terminal panel might not be visible if you're in HTTP mode. Switch the transport dropdown to stdio first, then rerun a query. If it still doesn't animate, ask the AI: "The stdio animation isn't firing. Check that the transport mode variable is being read correctly when the query button is clicked."

Temperature doesn't seem to change the answer

The synthesiseAnswer function may be reading a cached slider value instead of the live one. Ask the AI to "ensure the temperature and topP values are read directly from the slider elements at the moment the query button is clicked, not from a variable set at page load."

JSON export downloads an empty file

This usually means the queryLog array isn't being appended to correctly. Ask the AI to "add a console.log(queryLog) call after each query so I can verify entries are being pushed to the array before the export runs."

What You Learned

RAG Pipeline Mechanics

You built chunking, keyword scoring, and context injection from scratch — the exact same pattern used in production RAG systems, just without the vector database.

MCP Transport Modes

You saw the concrete differences between stdio and HTTP transport — what the message format looks like, when each is appropriate, and how switching modes changes the connection behaviour.

Sampling Parameters in Practice

Temperature and Top-P stopped being abstract concepts the moment you dragged the sliders and watched the answer change. You now have an intuitive feel for what these knobs actually do.

Multi-Concept AI Prompting

You directed an AI to integrate three distinct technical concepts into a single coherent app — that's a real advanced prompting skill that directly maps to how professional developers use AI coding tools.

Tips for Going Further

01

Add a real API connection. Prompt the AI to add an API key input field and wire the query to the actual OpenAI or Anthropic API, passing your temperature and top_p values as real parameters in the request body. The JSON preview you built in HTTP mode shows you exactly what the body should look like.

02

Swap keyword scoring for embeddings. Ask the AI to replace the TF-IDF keyword scorer with cosine similarity on embeddings using the Transformers.js library — this is how real production RAG systems work and it dramatically improves retrieval quality.

03

Build the actual MCP server. Take the stdio simulation panel and turn it into a real connection by prompting the AI to scaffold a Node.js MCP server using the official MCP SDK. Then your browser app can talk to it over a real stdio or HTTP transport.

04

Add a sampling A/B comparison mode. Prompt the AI to add a "Compare" button that runs the same query twice simultaneously — once with your current settings and once with a preset "conservative" profile (temperature 0.2, Top-P 0.5) — and shows both answers side by side so you can directly compare the effect of sampling parameters.

05

Persist the knowledge base. Ask the AI to save your documents and chunks to localStorage so they survive a page refresh. Then add an import/export feature that lets you save your entire knowledge base as a JSON file and reload it later.

More projects

One of 36 hands-on projects.

All projects Prompting & Workflows tutorials