What You'll Build
A Python research agent that combines three advanced techniques into one trustworthy pipeline — grounded citations, multi-step agent orchestration, and automatic hallucination flagging.
- A multi-step research agent built with the Claude Agent SDK that breaks questions into sub-tasks
- Automatic citation attachment — every claim is tagged with the source document or URL it came from
- A hallucination detection pass using ChatGPT that scores each claim against its cited source
- A clean JSON output format that separates verified claims, uncertain claims, and flagged hallucinations
- A simple CLI interface so you can run research queries from your terminal in seconds
What You'll Need
AI Tools
Claude (claude.ai or API) for agent orchestration and grounded research, plus ChatGPT (chat.openai.com or API) for the hallucination detection pass.
Text Editor & Python
VS Code or any editor you're comfortable with, plus Python 3.10+ installed. You'll need pip to install the Anthropic SDK and OpenAI SDK packages.
Step 1: Design the Agent's Architecture
Before writing a single line of code, you need a clear blueprint. This step uses Claude to design the full agent architecture — what modules exist, how data flows between them, and where each of the three techniques (grounding, agent framework, hallucination detection) plugs in. Getting this design right upfront saves you from messy rewrites later.
I want to build a Python research agent that combines three capabilities into one pipeline: 1. Grounded research with citations — the agent finds information and tags every claim with the source it came from (document name, URL, or passage). 2. Multi-step agent orchestration — using the Anthropic Claude Agent SDK (the `anthropic` Python package with tool use), the agent breaks a user question into sub-tasks: a planning step, a retrieval step, and a synthesis step. 3. Automatic hallucination detection — after the agent produces its grounded answer, a second pass using the OpenAI API (ChatGPT) checks each claim against its cited source and scores it as VERIFIED, UNCERTAIN, or HALLUCINATED. Please design the full Python module architecture for this project. For each module, tell me: - The filename - Its single responsibility - Its main functions with input/output types - How it connects to the other modules Also draw a simple ASCII data-flow diagram showing how a user query travels through the system from input to final verified JSON output. Do not write any code yet — just the architecture design.
What to look for: Claude should give you a clean list of 4–6 Python files (something like agent.py, retriever.py, grounder.py, hallucination_checker.py, cli.py), each with a clear responsibility. The ASCII diagram should show arrows flowing from user query → planner → retriever → grounder → synthesiser → hallucination checker → JSON output. If the design looks muddled or modules are doing too many things, ask Claude to simplify until each file has exactly one job.
Step 2: Build the Claude Agent with Tool Use
Now you'll generate the core agent code. This is where the agent framework tutorial comes to life — you're using Claude's native tool-use system (the Agent SDK pattern) to create a planner that calls sub-tools to retrieve and synthesise information. This step produces the main agent.py file.
Based on the architecture we just designed, write the complete Python code for `agent.py`. This file must: 1. Use the `anthropic` Python SDK with tool use (function calling) to implement a multi-step research agent. 2. Define three tools that Claude can call: - `plan_research(question: str) -> list[str]`: breaks a question into 2–4 sub-questions - `retrieve_passages(sub_question: str, documents: list[dict]) -> list[dict]`: searches a list of document dicts (each with keys `title`, `url`, `content`) and returns the most relevant passages as dicts with keys `passage`, `source_title`, `source_url` - `synthesise_answer(sub_answers: list[dict]) -> str`: combines retrieved passages into a coherent answer 3. Run a loop that calls Claude with these tools until it produces a final answer — exactly the agentic loop pattern from the Claude Agent SDK. 4. Return a structured Python dict with keys: `answer` (string), `claims` (list of dicts, each with `text` and `citation` keys where citation has `title` and `url`). 5. Accept a `question: str` and `documents: list[dict]` as inputs to a `run_agent(question, documents)` function. Use `claude-3-5-sonnet-20241022` as the model. Include clear comments explaining each part of the agentic loop. Do not add hallucination detection yet — that comes in the next module.
What to look for: You should see a proper agentic loop — a while True or equivalent that sends tool results back to Claude and continues until a stop_reason == "end_turn". Each tool should be defined in the Anthropic tools list format with a name, description, and input_schema. The final return value should be the structured dict with answer and claims. If Claude skips any tool definition or the loop looks incomplete, ask it to fill in the missing piece specifically.
Step 3: Add Grounding and Citation Tracking
Grounding means every claim must be tied to a real source — not just asserted by the model. In this step you'll build the grounder.py module, which enforces that every claim the agent makes includes a citation object. This is the practical application of the grounding tutorial: traceability isn't optional, it's baked into the data structure.
Now write the complete Python code for `grounder.py`. This module is responsible for enforcing citation traceability on every claim produced by the agent. It must: 1. Define a `Claim` dataclass with fields: `text: str`, `citation_title: str`, `citation_url: str`, `citation_passage: str` (the exact passage from the source that supports this claim). 2. Write a function `extract_claims(agent_output: dict, documents: list[dict]) -> list[Claim]` that: - Takes the `claims` list from the agent's output dict - For each claim, finds the best matching passage in the source documents using simple keyword overlap (no ML needed) - Populates the `citation_passage` field with the actual text from the document - Raises a `CitationMissingError` (custom exception) if a claim's cited URL doesn't match any document in the input list 3. Write a function `format_grounded_report(claims: list[Claim]) -> dict` that returns a dict with: - `total_claims`: integer count - `grounded_claims`: the list of claim dicts with all citation fields populated - `ungrounded_count`: how many claims had no matching passage Include docstrings. Keep it self-contained with no external dependencies beyond Python's standard library.
What to look for: You should see a clean Claim dataclass, the custom CitationMissingError, and both functions fully implemented. The keyword overlap logic doesn't need to be fancy — splitting on whitespace and checking intersection is fine. What matters is that no claim can pass through this module without a citation_passage field. If Claude uses a third-party library like numpy or sklearn, ask it to rewrite using only the standard library.
Step 4: Build the Hallucination Detection Pass
This is where ChatGPT enters the pipeline as a dedicated verification model. For each grounded claim, you'll send ChatGPT the claim text and the supporting passage and ask it to judge whether the claim is actually supported. This is the hallucination detection tutorial in action — using a second model as an independent auditor of the first model's output.
Write the complete Python code for `hallucination_checker.py`.
This module takes grounded claims (each with a `text` and a `citation_passage`) and uses the OpenAI API (GPT-4o-mini model) to automatically detect hallucinations.
The module must:
1. Define a function `check_claim(claim_text: str, citation_passage: str, openai_client) -> dict` that:
- Sends a structured prompt to GPT-4o-mini asking it to judge whether `claim_text` is supported by `citation_passage`
- The system prompt should instruct GPT to act as a strict fact-checker and respond ONLY with a JSON object
- The JSON response must have exactly these keys: `verdict` (one of "VERIFIED", "UNCERTAIN", "HALLUCINATED"), `confidence` (float 0.0–1.0), `reason` (one sentence explaining the verdict)
- Use `response_format={"type": "json_object"}` in the API call to enforce JSON output
- Return the parsed JSON dict
2. Define a function `check_all_claims(grounded_report: dict, openai_client) -> dict` that:
- Iterates over every claim in `grounded_report["grounded_claims"]`
- Calls `check_claim` for each one
- Attaches the verdict dict to each claim under a key called `hallucination_check`
- Returns a final report dict with keys: `verified_count`, `uncertain_count`, `hallucinated_count`, `claims` (the full enriched list)
3. Handle OpenAI API errors gracefully — if a call fails, set verdict to "UNCERTAIN" and reason to the error message.
Add clear comments. Use the `openai` Python package (v1+).
What to look for: The system prompt inside check_claim should be strict — something like "You are a fact-checking auditor. Respond only in JSON. Do not explain outside the JSON object." The response_format parameter should be present in the API call. The counts in check_all_claims should be derived from the verdicts, not hardcoded. If ChatGPT generates the prompt without response_format, add it manually or ask for a revision.
Step 5: Wire It Together with a CLI
Now you'll connect all three modules into a single runnable script with a clean command-line interface. This step is about integration — making sure data flows correctly from the agent output, through the grounder, and into the hallucination checker, then printing a readable final report. You'll also add a sample document set so you can test it immediately.
Write the complete Python code for `cli.py` that wires together `agent.py`, `grounder.py`, and `hallucination_checker.py` into a single runnable pipeline. The CLI must: 1. Accept a `--question` argument (the research question) and an optional `--docs` argument pointing to a JSON file of documents. 2. If no `--docs` file is provided, use a hardcoded list of 3 sample documents about climate change (each with `title`, `url`, and a multi-sentence `content` field you write inline) so the user can test immediately without any setup. 3. Run this pipeline in order: a. Call `run_agent(question, documents)` from `agent.py` b. Call `extract_claims(agent_output, documents)` and `format_grounded_report(claims)` from `grounder.py` c. Initialise both the `anthropic.Anthropic()` and `openai.OpenAI()` clients using `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` environment variables d. Call `check_all_claims(grounded_report, openai_client)` from `hallucination_checker.py` 4. Print a formatted terminal report that shows: - The final answer - Each claim with its citation and its hallucination verdict (colour-coded: green for VERIFIED, yellow for UNCERTAIN, red for HALLUCINATED using ANSI codes) - A summary line: "X verified, Y uncertain, Z hallucinated out of N total claims" 5. Also save the full final report as `output.json` in the current directory. Include a `if __name__ == "__main__":` guard. Add a comment at the top listing the pip install commands needed: `pip install anthropic openai`.
What to look for: The pipeline should flow in the exact order described — agent first, then grounder, then hallucination checker. The ANSI colour codes should wrap each verdict word: \033[92m for green, \033[93m for yellow, \033[91m for red. The sample documents should be substantive enough to actually ground a few claims. Run it with python cli.py --question "What causes sea level rise?" and you should see a full coloured report in your terminal within 20–30 seconds.
Common Issues
The agent loop runs forever and never stops
This usually means the loop isn't checking stop_reason correctly. Ask Claude: "My agentic loop in agent.py never exits. Show me the exact condition that should break the loop when Claude returns stop_reason == 'end_turn' and there are no more tool calls."
CitationMissingError fires on every claim
The URLs in the agent's output might not exactly match the URLs in your documents list. Ask Claude to make the URL matching in extract_claims case-insensitive and strip trailing slashes before comparing.
ChatGPT returns malformed JSON despite response_format
Wrap the json.loads() call in a try/except and fall back to {"verdict": "UNCERTAIN", "confidence": 0.0, "reason": "parse error"}. Ask ChatGPT: "Show me the try/except block that gracefully handles a JSON parse failure in check_claim."
All claims come back as HALLUCINATED
The fact-checker system prompt may be too strict, or the citation passages being passed in are too short to support the claims. Ask Claude to increase the passage extraction window in extract_claims — grab 2–3 sentences around the keyword match instead of just the matching sentence.
API rate limit errors during testing
If you're hitting rate limits, ask Claude to add a time.sleep(1) between each check_claim call in check_all_claims. This slows the pipeline slightly but keeps you under free-tier limits.
What You Learned
Agent Framework Orchestration
You built a real agentic loop using Claude's tool-use system — the same pattern that powers production AI agents — and saw how it breaks a question into sub-tasks automatically.
Grounding and Traceability
You enforced citation traceability at the data-structure level — every claim carries its source passage, making it impossible for an unsupported claim to silently pass through the pipeline.
Automatic Hallucination Detection
You used ChatGPT as an independent auditor to score every claim — a practical implementation of the "second-model verification" technique that dramatically reduces trust-but-verify fatigue.
Multi-Model Pipeline Design
You learned how to use two different AI models for two different jobs in the same pipeline — Claude for generation and tool use, ChatGPT for verification — choosing each model for what it's best at.
Tips for Going Further
-
→
Swap in a LangChain or LlamaIndex agent: The architecture is modular — ask Claude to rewrite
agent.pyusing LangChain'sAgentExecutoror LlamaIndex's query engine and compare the output quality and verbosity. - → Add real web retrieval: Hook up the Brave Search API or Tavily to replace the sample documents with live web results, turning this into a true real-time research assistant.
- → Build a confidence threshold filter: Ask ChatGPT to add a flag that automatically rejects any claim with a hallucination confidence score above 0.7 and re-queries the agent for a better answer.
-
→
Wrap it in a simple web UI: Ask Claude to generate a minimal Flask or FastAPI endpoint around
cli.py's pipeline so you can query the agent from a browser form instead of the terminal. -
→
Log hallucination patterns over time: Append every
output.jsonto a SQLite database and ask Claude to write a script that shows which claim types are most frequently hallucinated — a useful dataset for prompt improvement.