What You'll Build
A retrieval-augmented AI agent that reasons across your documents and calls live APIs — all through a secure, structured pipeline. Here's what the finished project does:
- An agentic reasoning loop that decides when to search documents vs. call an API
- A retrieval layer with quality checks so the right chunks actually surface
- MCP OAuth integration so the agent calls a third-party API without exposing credentials
- A confidence-scoring step that flags low-quality retrievals before the agent acts on them
- A readable trace log so you can follow every step the agent took and why
What You'll Need
Build It: Step by Step
Design the Agent's Core Reasoning Loop
Every agent is fundamentally a loop: observe, think, act, repeat. Before writing any code, you need to define exactly what your agent's loop looks like — what it observes, what decisions it makes, and what tools it can call. This step gets that skeleton in place.
I'm building an AI agent in Python that combines document retrieval (RAG) with live API calls via an MCP OAuth connection. Please design the core agent reasoning loop for me. The agent should: 1. Receive a user question 2. Decide whether to (a) retrieve from a local document store, (b) call a live API via MCP OAuth, or (c) do both 3. Evaluate the quality of what it retrieved before using it 4. Produce a final answer with a step-by-step trace log Please write this as a Python class called AgentLoop with methods for: run(question), _decide_tool(question), _retrieve(question), _call_api(endpoint, params), _evaluate_retrieval(chunks), and _format_trace(steps). Use placeholder implementations for _retrieve and _call_api — I'll fill those in later. Include docstrings explaining what each method does and why.
You should see a Python class with all six methods stubbed out, clear docstrings, and a run() method that calls the others in a logical sequence. The _decide_tool() method should include some reasoning logic (even if simple) rather than always choosing the same tool. Save this file as agent_loop.py.
Build the Retrieval Layer with Quality Checks
Retrieval quality is the hidden bottleneck in most RAG systems — you can embed perfectly and still surface the wrong chunk. This step implements the _retrieve() method with a quality-scoring pass so low-relevance chunks get flagged before the agent acts on them.
I'm implementing the _retrieve() and _evaluate_retrieval() methods for my Python AgentLoop class. The agent has access to a simple in-memory document store (a list of dicts with 'id', 'text', and 'embedding' keys). Please implement: 1. A retriever that uses cosine similarity to rank chunks against the query embedding. Return the top 5 chunks. 2. A quality evaluator (_evaluate_retrieval) that scores each returned chunk on three signals: - Similarity score (from the cosine distance) - Keyword overlap between the chunk and the original question - Chunk length (penalise very short chunks under 50 words) Combine these into a 0-1 confidence score per chunk. If the top chunk's confidence is below 0.4, add a 'LOW_CONFIDENCE' flag to the trace. 3. A small sample document store (3-4 documents about AI agents) for testing. Use numpy for the cosine similarity. Add a brief comment explaining why each quality signal matters.
Claude should give you a retriever function, a scoring function with three clearly weighted signals, and a working sample document store. The LOW_CONFIDENCE flag should appear in the trace output, not just be silently ignored. Save this as retriever.py and wire it into your agent_loop.py.
Add MCP OAuth to Connect a Live API Securely
When your local documents aren't enough, the agent should be able to call a real API — but not by hardcoding credentials into your code. This step implements the _call_api() method using an MCP OAuth flow so the agent fetches a token at runtime and never stores secrets in the source.
I need to implement the _call_api() method in my Python AgentLoop class using an MCP OAuth pattern. The goal is for the agent to call a third-party REST API (I'll use the Open-Meteo weather API as a safe public example, but structure it so any OAuth-protected API could slot in). Please write: 1. An MCPOAuthClient class in a file called mcp_client.py that: - Reads client_id and client_secret from environment variables (never hardcoded) - Implements a get_token() method using the OAuth 2.0 client credentials flow - Caches the token in memory and refreshes it when expired - Raises a clear error if credentials are missing 2. The _call_api(endpoint, params) method in AgentLoop that uses MCPOAuthClient to get a token, attaches it as a Bearer header, and returns the parsed JSON response. 3. A brief comment block explaining the security benefit of this approach vs. hardcoding API keys. For the demo, wire up a call to retrieve current weather for a city so I can test the full flow end to end.
You want to see credentials read from os.environ, a token cache with expiry logic, and the _call_api method attaching the token as a header — not as a URL parameter. The comment block should clearly explain why environment variables plus runtime token fetching is safer than a hardcoded key. Save as mcp_client.py.
Wire Up the Full Agent Loop and Run It
Now the pieces exist — the loop, the retriever, and the API client. This step connects them together in a single runnable entry point and adds the trace logger so you can see every decision the agent made. Running this end-to-end is the moment the agent becomes real.
I have three Python files: agent_loop.py (AgentLoop class), retriever.py (retriever and quality scorer), and mcp_client.py (MCPOAuthClient). Please help me wire them together. Write a main.py file that: 1. Instantiates AgentLoop with the retriever and MCPOAuthClient injected as dependencies (constructor injection, not global imports inside the class) 2. Runs three test questions through the agent: - "What is the observe-act loop in AI agents?" (should hit the document store) - "What is the weather in London right now?" (should hit the API) - "Explain how an AI agent decides what tool to use next" (may use both) 3. Prints a human-readable trace for each run that shows: the question, which tool was chosen, the confidence score for any retrieved chunks, any LOW_CONFIDENCE flags, and the final answer. Also add a TraceLogger class that writes each run's full trace to a local file called agent_trace.log with timestamps.
The final main.py should run without import errors and print three clearly labelled trace blocks to the terminal. You should see the agent making different routing decisions for each question. The agent_trace.log file should appear on disk after the first run. If the tool decision always comes back the same way for all three questions, prompt Claude to make the routing logic more sensitive to question type.
Stress-Test the Retrieval Quality and Tune the Thresholds
A system is only as good as its failure modes. This final step intentionally breaks the retrieval — ask questions the document store can't answer well — and watch the confidence scoring catch it. Then tune the thresholds and observe how the agent changes its behaviour.
I want to stress-test my retrieval quality system. My agent uses a confidence threshold of 0.4 to flag LOW_CONFIDENCE retrievals. Please help me with two things: 1. Write a stress_test.py file that runs 5 questions through the retriever — 2 that are well-covered by my sample documents (expect high confidence), 2 that are completely off-topic (expect LOW_CONFIDENCE), and 1 that is borderline. For each, print the top chunk retrieved, the confidence score, and whether the flag was triggered. 2. Then show me how to modify the _evaluate_retrieval() method to implement adaptive thresholding: instead of a fixed 0.4 cutoff, the threshold should raise to 0.55 if the question contains words like "exactly", "precise", or "specific" (indicating the user wants high-accuracy answers), and lower to 0.3 for exploratory questions with words like "roughly", "general", or "overview". Explain why adaptive thresholds improve agent reliability in production.
The stress test output should clearly show different confidence scores across the five questions — if every score is similar regardless of the question, the scoring signals aren't working well. The adaptive threshold code should modify the _evaluate_retrieval() method cleanly (not add a completely separate function), and Claude's explanation of why adaptive thresholds matter should mention the trade-off between precision and recall in agent contexts.
Common Issues
Your _decide_tool() routing logic is probably too simple. Ask Claude to add keyword-based routing (questions with "weather", "current", "live", "today" go to the API; everything else starts with retrieval) and to fall back to the API when retrieval returns a LOW_CONFIDENCE flag.
This usually means all your document embeddings are identical (because you used placeholder random vectors). Ask Claude to generate distinct mock embeddings for each sample document, or swap in a real embedding call using sentence-transformers for meaningful scores.
You haven't set the environment variables. Run export CLIENT_ID=demo export CLIENT_SECRET=demo in your terminal before running main.py, or create a .env file and use the python-dotenv library — ask Claude to add that loading step for you.
Check that your TraceLogger opens the file in append mode ('a') and that the path is writable. Ask Claude to add a confirmation print statement after each write so you know the logger is actually reaching that line.
Double-check that the keyword check happens before the threshold comparison, not after. Ask Claude to add a debug print that outputs the computed threshold value so you can confirm it's being set correctly for each question type.
What You Learned
You built a real observe-think-act loop where the agent makes a routing decision every cycle — not just a single-shot prompt. That's the core architecture that separates agents from chatbots.
You learned that returning a chunk isn't the same as returning the right chunk. Multi-signal confidence scoring — similarity, keyword overlap, length — catches the gaps that embeddings alone miss.
You implemented a runtime token fetch pattern where credentials live in environment variables and tokens are cached with expiry — the same pattern production agents use so secrets never touch your source code.
You saw why fixed thresholds break in the real world and how to tune them to the user's intent. And the trace log means you can always explain every decision the agent made — which is non-negotiable in any serious deployment.
Tips for Going Further
Right now the agent answers one question and stops. Extend it to maintain a conversation history across turns — pass previous Q&A pairs into the context so the agent can reference earlier answers when making new decisions.
Replace the mock embeddings with sentence-transformers or the OpenAI embeddings API. The confidence scores will become meaningful and you'll immediately see retrieval quality improve — or reveal new failure modes worth fixing.
Add a second MCP client — say, a GitHub API connection — and extend the routing logic to pick between weather, GitHub, and document retrieval. This forces you to think about how agents scale to many tools gracefully.
Parse the agent_trace.log file and render a simple HTML table that shows each run's tool choice, confidence scores, and flags. This gives you an audit dashboard for every decision the agent made — extremely useful when something goes wrong.