What You'll Build
A single-page monitoring dashboard that combines three critical AI concepts: context failure detection, agent memory state tracking, and embedding model quality comparison. This is the kind of tool senior AI engineers wish they had when debugging long-running AI workflows.
- ✓ A Context Drift Detector that simulates a long conversation and flags the moment topics silently diverge from the original intent
- ✓ An Agent Memory State Panel showing what variables, facts, and instructions persist between agent steps — and which get silently dropped
- ✓ An Embedding Model Comparator that shows how three different model configurations (different dimensions and quality tiers) would score the same document retrieval query
- ✓ A Memory Health Score that rolls up all three signals into a single A–F grade with a plain-English explanation
- ✓ A Session Timeline that replays how context and memory evolved step by step, so you can see exactly where things went wrong
- ✓ Export and Reset controls to save your session data as JSON and start fresh without losing your history
What You'll Need
Claude or ChatGPT
Claude (claude.ai) or ChatGPT (chat.openai.com) — either works. Claude tends to produce cleaner structured HTML in one shot; ChatGPT with GPT-4o is great for iterating on logic.
A Text Editor + Browser
VS Code is ideal, but Notepad works fine. You'll save the AI's output as an .html file and open it in Chrome or Firefox. No installs, no servers required.
Why This Project Matters
Each of this week's tutorials covers a failure mode that's invisible until it bites you. Context drift means your AI quietly stops following your original instructions. Agent memory leaks mean your multi-step workflows forget things they should remember. And choosing the wrong embedding model means your semantic search returns plausible-sounding garbage. This project builds a single tool that makes all three failure modes visible.
Everything is simulated in the browser — no API keys, no backend. That means you can focus entirely on understanding the concepts and the code the AI writes for you, not on debugging network calls.
Step 1: Build the Dashboard Shell and Layout
Before any logic, you need a solid visual container. This step gives the AI enough structure to produce a dashboard with three clearly labelled panels — one for each concept — plus a top-level health score bar. Be explicit about the dark theme and the panel names so the AI doesn't invent its own structure.
You're also asking the AI to use semantic HTML so later steps can wire in real JavaScript without fighting the markup. Think of this as pouring the foundation before building the walls.
Build me a single-file HTML dashboard called "AI Memory Monitor". Use a dark theme with a near-black background (#0f0f13) and a dark card surface (#18181f). The layout should have: 1. A sticky top header with the title "AI Memory Monitor" in white, a subtitle "Context · Agent State · Embeddings" in muted purple, and a single "Memory Health Score" badge on the right that defaults to showing "— / 100" in amber. 2. Three equal-width panels in a CSS grid below the header, each with a rounded dark card: - Panel A: title "Context Drift Detector" with a violet left border accent - Panel B: title "Agent Memory State" with a cyan left border accent - Panel C: title "Embedding Comparator" with an amber left border accent 3. A full-width "Session Timeline" strip below the three panels, with a dark card, title "Session Timeline", and an empty scrollable horizontal row of timeline nodes. 4. A footer row with two buttons: "Export JSON" (outlined, zinc) and "Reset Session" (outlined, rose). Use only vanilla HTML, CSS, and JavaScript — no external libraries. Include id attributes on every panel, button, and dynamic display area. Add a CSS variable block at the top for the main colors so I can theme it later. Leave the panel bodies empty for now — I will fill them in the next steps.
What to look for: The AI should give you a complete HTML file with a sticky header, a 3-column grid of cards, a timeline strip, and footer buttons — all with ids. Open it in your browser and confirm the three panels are visible and labelled correctly. If anything overlaps or the grid breaks, paste the broken screenshot description back and ask it to fix the CSS grid.
Step 2: Wire Up the Context Drift Detector
This panel is the heart of the tutorial from "Context Failure Modes." You'll ask the AI to build a simulation where the user can type a starting goal, then add conversation turns. After each turn, a drift score is calculated by comparing how far the new message has semantically wandered from the original — using a simple keyword-overlap heuristic (since we're staying browser-only).
The key learning here is making the failure mode visible: you want a colour-coded bar that goes from green to red as drift accumulates, plus a plain-English warning when a threshold is crossed. This is exactly what real context management systems do under the hood.
Now fill in Panel A (id="panel-context") with a working Context Drift Detector. Here's exactly what it needs:
1. A labelled input field: "Original Goal" (placeholder: "e.g. Write a Python web scraper") — stored as the baseline.
2. A second input field: "Add Conversation Turn" with an "Add Turn" button. Each submitted turn gets appended to a visible scrollable list of turns inside the panel.
3. After each turn is added, calculate a "drift score" from 0–100 using this method:
- Tokenise both the original goal and the new turn into lowercase words, stripping punctuation.
- Drift score = 100 minus the Jaccard similarity percentage between the two word sets.
- Round to nearest integer.
4. Show the latest drift score as a coloured horizontal progress bar below the turn list:
- 0–30: green ("On track")
- 31–60: amber ("Drifting — consider a context reset")
- 61–100: rose ("Context failure likely — AI has probably forgotten your original goal")
5. Also display the numeric score and the label text beneath the bar.
6. Every time a turn is added, push an object { type: 'context', turn: N, score: X, label: '...' } into a global sessionLog array.
Do not touch the other panels or the header. Only modify the contents inside the panel-context div.
What to look for: Type a goal like "Build a Python web scraper" and then add turns like "Actually let's use JavaScript" and then "Can you recommend a good restaurant?" — you should see the drift bar swing from green to amber to red. If the Jaccard calculation seems off, ask the AI to console.log the two word sets so you can verify.
Step 3: Build the Agent Memory State Panel
This panel brings the "Agent Memory and State" tutorial to life. Agents pass state between steps — but not all of it survives. You'll build a mini state manager that lets you add key-value pairs to a simulated agent context, mark them as "persisted" or "ephemeral," and then run a "Next Step" simulation that randomly drops ephemeral keys to mimic what real agent runtimes do.
The goal is to make the user feel the difference between persistent and ephemeral memory hands-on. When they click "Next Step" and watch their ephemeral keys disappear, the concept clicks in a way that reading about it never does.
Now fill in Panel B (id="panel-memory") with a working Agent Memory State simulator. Here's what it needs:
1. Three input fields in a row: "Key" (e.g. userName), "Value" (e.g. Alice), and a dropdown with two options: "Persisted" and "Ephemeral". Plus an "Add to Memory" button.
2. A table below the inputs showing the current memory state. Columns: Key | Value | Type | Status. Persisted rows have a cyan left cell accent. Ephemeral rows have an amber left cell accent. Status starts as "Active" for all.
3. A "▶ Simulate Next Agent Step" button. When clicked:
- Each Ephemeral key has a 60% chance of being dropped (set its Status to "Dropped" and grey out the row).
- Persisted keys are never dropped.
- Show a small toast notification at the top of the panel: "Step complete — X ephemeral keys dropped" (disappears after 3 seconds).
4. A small "Memory Retention Rate" stat below the table showing: (active keys / total keys) × 100, as a percentage rounded to one decimal place.
5. Every time a step is simulated, push { type: 'memory', step: N, retained: X, dropped: Y } into the global sessionLog array.
Only modify the contents inside panel-memory. Do not change other panels.
What to look for: Add a few persisted keys (like userName, projectName) and a few ephemeral ones (like lastApiResponse, currentDraft). Click "Simulate Next Agent Step" several times and watch the ephemeral rows get dropped and greyed out while the persisted rows survive. The retention rate stat should update correctly with each step.
Step 4: Add the Embedding Model Comparator
This panel applies the "Choosing an Embedding Model" tutorial. Different embedding models have different dimension counts, cost tiers, and retrieval quality. You'll build a side-by-side comparator where the user types a query and sees how three simulated model configurations would score it — using a seeded scoring function that reflects the real trade-offs between small/fast/cheap models and large/slow/accurate ones.
The scoring isn't real embedding math (we'd need an API for that), but it's carefully designed to reflect the genuine patterns: higher-dimension models score better on nuanced queries, cheaper models degrade on long queries, and mid-tier models are often surprisingly competitive. This teaches the trade-off intuitively.
Now fill in Panel C (id="panel-embeddings") with a working Embedding Model Comparator. Here's exactly what it needs:
1. A textarea input: "Enter a retrieval query" (placeholder: "e.g. How do transformers handle long-range dependencies?") and a "Compare Models" button.
2. Three model cards displayed side by side after comparison. Hardcode these three models:
- Model A: "text-embedding-3-small" | 1536 dims | Tier: Economy | Cost: $0.02/1M tokens
- Model B: "text-embedding-3-large" | 3072 dims | Tier: Premium | Cost: $0.13/1M tokens
- Model C: "text-embedding-ada-002" | 1536 dims | Tier: Legacy | Cost: $0.10/1M tokens
3. For each model, calculate a simulated "Retrieval Quality Score" from 0–100 using this deterministic formula:
- Base score: use a simple seeded hash of the query string to generate a consistent base (0–60).
- Model A bonus: +25 if query length < 80 chars, +10 if longer (good on short queries, decent on long).
- Model B bonus: +35 always (highest quality, highest cost).
- Model C bonus: +15 (legacy model, lowest bonus).
- Cap all scores at 100.
4. Show each model card with: model name, dimensions badge, tier badge (colour-coded: Economy=green, Premium=violet, Legacy=amber), cost, quality score as a progress bar, and a one-sentence plain-English recommendation based on the score.
5. Highlight the "best value" card (highest score divided by cost) with a subtle amber glow border.
6. Push { type: 'embedding', query: '...', winner: 'model name', scores: {...} } into the global sessionLog array.
Only modify panel-embeddings.
What to look for: Type a short query and a long query and compare how the model rankings shift. Model B (large) should consistently score highest, but Model A should be flagged as best value on short queries. If the seeded hash produces the same score for every query, ask the AI to use a character-code-sum approach to generate variability from the query text.
Step 5: Connect the Health Score, Timeline, and Export
This final step ties everything together. You'll wire up the global sessionLog to populate the Session Timeline with a visual step-by-step replay, and compute the Memory Health Score from all three panels combined. You'll also activate the Export JSON and Reset Session buttons. This is where the dashboard stops being three separate toys and becomes one coherent diagnostic tool.
Ask the AI to be deliberate about the health score formula — it should weight context drift most heavily (since it's the sneakiest failure mode), then memory retention, then embedding quality. You want the score to feel meaningful, not random.
Now wire up the three integration pieces that tie the whole dashboard together: --- PART 1: Memory Health Score (header badge) --- Create a function called recalculateHealthScore() that runs every time any panel pushes to sessionLog. It should: - Pull the latest context drift score from sessionLog (type: 'context'). Convert it to a context health sub-score: (100 - drift) × 0.45. - Pull the latest memory retention rate from sessionLog (type: 'memory'). Use retention% × 0.35. - Pull the best embedding quality score from sessionLog (type: 'embedding'). Use bestScore × 0.20. - Sum the three for a total out of 100. Round to nearest integer. - Update the header badge: show the score, and colour it green (80+), amber (50–79), or rose (below 50). - Also show a one-line plain-English status: e.g. "Healthy session", "Context drifting — consider a reset", or "Multiple failure signals detected". If a panel hasn't logged anything yet, treat its sub-score as 50 (neutral). --- PART 2: Session Timeline --- Every time sessionLog is updated, re-render the horizontal timeline inside the #timeline div. Each entry should be a small node: a coloured circle (violet for context, cyan for memory, amber for embedding), a timestamp (HH:MM:SS), and a 2-word label (e.g. "Drift: 72", "Dropped: 2", "Winner: large"). Nodes should be connected by a thin horizontal line. The timeline scrolls horizontally if there are many nodes. --- PART 3: Export and Reset --- - "Export JSON" button: JSON.stringify the full sessionLog array with 2-space indentation and trigger a .json file download named memory-monitor-session.json. - "Reset Session" button: clear sessionLog to [], reset all three panels to their empty states, reset the health score badge to "— / 100" in amber, and clear the timeline. Ask the user to confirm with a simple window.confirm dialog before resetting. Do not change any panel internals — only add the health score function, timeline renderer, and button handlers.
What to look for: After completing actions in all three panels, the header score should update and the timeline should show coloured nodes in chronological order. Click Export JSON and verify the downloaded file contains all your session events. Click Reset and confirm everything clears cleanly. If the health score doesn't update, the most common cause is that the panels aren't calling recalculateHealthScore() after pushing to sessionLog — ask the AI to add that call to each panel's event handler.
Common Issues
The three panels don't update the health score
This usually means the panels' event handlers push to sessionLog but don't call recalculateHealthScore(). Ask the AI: "Check every place in the code where sessionLog.push() is called and make sure recalculateHealthScore() is called immediately after each one."
The drift score is the same for every input
The Jaccard similarity might be comparing the wrong strings. Tell the AI: "console.log the two word sets being compared in the drift calculation so I can verify they reflect the original goal and the latest turn respectively."
Ephemeral memory keys never get dropped
The random drop logic might be checking the wrong condition. Ask: "Make the drop condition clearly visible — add a console.log that prints each ephemeral key's random roll and whether it was dropped, so I can verify the 60% probability is working."
The timeline doesn't scroll horizontally
The timeline container probably needs overflow-x: auto and white-space: nowrap on the nodes. Tell the AI: "The timeline nodes wrap to a second line instead of scrolling. Fix the CSS so they stay on one row and scroll horizontally when there are more than 6 nodes."
Export downloads an empty file
sessionLog is probably scoped inside a function rather than declared globally. Ask: "Make sure sessionLog is declared at the very top of the script tag as a let or const at the global scope, not inside any function or DOMContentLoaded handler."
What You Learned
Context Failure is Measurable
You built a working drift detector using Jaccard similarity — the same conceptual approach used in real context management systems. You can now spot when a long conversation has silently abandoned its original goal.
Agent Memory Has Two Classes
By simulating persisted vs. ephemeral state, you now understand intuitively why agent workflows need explicit memory management — and why "the AI should just remember" is not a reliable design decision.
Embedding Trade-offs Are Real
The comparator made tangible what the tutorials explained abstractly: more dimensions cost more but retrieve better, legacy models punch below their cost, and "best value" depends entirely on your query length and budget.
Systems Thinking with AI
You learned to prompt AI to build a multi-panel system with a shared data bus (sessionLog), a derived computed metric (health score), and a replay mechanism (timeline) — the same architecture used in real AI observability tools.
Tips for Going Further
-
→
Connect a real embedding API. Replace the simulated scoring function with a real call to OpenAI's embeddings endpoint. Use cosine similarity between your query embedding and a set of stored document embeddings — now the comparator shows real numbers.
-
→
Add a context compaction button. When drift exceeds 60, show a "Compact Context" button that summarises all turns into a single new baseline — the same technique Claude Code's auto-compact uses to restart degraded conversations.
-
→
Persist sessions to localStorage. Ask the AI to serialise sessionLog to localStorage on every update so your session survives a page refresh. Add a "Load Previous Session" button on startup.
-
→
Make the memory model configurable. Add a settings panel where you can change the ephemeral drop probability (currently 60%). Run experiments: what health score do you get with 30% drop rate vs. 80%? This is how real agent runtimes are tuned.
-
→
Export to a shareable report. Instead of exporting raw JSON, ask the AI to generate a formatted HTML report from the session data — with charts, a summary paragraph, and a final health grade. Paste it into a Notion doc or share it with a teammate.