Case study · 2026
traced-research-agent
A RAG research agent with the part most demos skip: an eval harness that proves it works.
The problem
Anyone can wire an LLM to a vector store and get plausible-sounding answers. The hard part is knowing whether they're right, and when they're wrong, whether retrieval or synthesis is to blame.
Decisions, and what they cost
Every architecture is a set of trade-offs. These are the ones I made, the alternatives I rejected, and why.
25 golden questions + an LLM-as-judge scorer
vs. eyeballing or exact-matchExact match fails on paraphrase; eyeballing drifts. An LLM judge scores meaning, and the golden set stays small enough to audit by hand.
Per-component ablation
vs. tuning the whole pipeline at onceSwapping one stage at a time shows exactly where each failure comes from. Slower than vibes-driven tuning, but every gain has evidence behind it.
SSE streaming
vs. blocking request/responseAgent runs are slow. Streaming intermediate steps is the difference between a usable tool and a spinner.
Evidence it works
Accuracy 0.641 → 0.734 on the golden set, every gain attributed to a specific change.
Every answer is traceable through the LangGraph steps that produced it. No black box.
What I'd do differently
The lecture-slides corpus is a toy; I'd swap it for something messier. Next: extracting the harness into a standalone retrieval-eval tool with recall@k and MRR scoring, per-stage failure attribution, and a GitHub Action that fails builds on regression.