Case study · 2026
traced-research-agent
A RAG research agent with the part most demos skip: an eval harness that proves it works.
The problem
Anyone can wire an LLM to a vector store and get plausible-sounding answers. The hard part is knowing whether they're right, and when they're wrong, whether retrieval or synthesis is to blame. So I built the measurement first, then let it decide what shipped.
Decisions, and what they cost
Every architecture is a set of trade-offs. These are the ones I made, the alternatives I rejected, and why.
Hybrid dense + BM25 retrieval fused with Reciprocal Rank Fusion
vs. pure vector similarityEmbeddings miss exact terminology and BM25 misses paraphrase, so each covers the other's blind spot. RRF (k=60) merges both rankings without tuning a weight. This was the single biggest win of the whole project: 0.641 to 0.831.
An LLM self-critique node that scores each chunk 1 to 5 and drops anything under 3
vs. passing everything retrieved straight to synthesisPrecision beats recall once the synthesizer is the bottleneck, since irrelevant context actively degrades answers. Worth +0.09 on its own, at the cost of one extra LLM call per question.
8 rubric-scored questions with an LLM-as-judge scorer
vs. eyeballing or exact-matchExact match fails on paraphrase; eyeballing drifts. Each question carries explicit rubric points and a source slide, and two deterministic checks (did it cite a real chunk, did retrieval return anything) run alongside the model score so the judge is never the only signal. The set stays small enough to audit by hand.
One change per run, nine runs, every trace persisted
vs. tuning the whole pipeline at onceSwapping one stage at a time shows exactly where each gain came from. Slower than vibes-driven tuning, but it is the only reason I could tell an improvement from a regression.
SSE streaming
vs. blocking request/responseAgent runs are slow. Streaming intermediate steps is the difference between a usable tool and a spinner.
Evidence it works
Mean score 0.641 to 0.831 across nine recorded ablations, each attributable to one specific change.
The harness rejected two of my own later ideas: hybrid-rrf-idf scored 0.740 and hybrid-rrf-v2 scored 0.699, both worse than the 0.831 they were meant to beat, so I kept the simpler implementation.
Every run persists per-question traces, citations, judge reasoning, and elapsed time to a named JSON file, so any two runs can be compared after the fact.
Five LangGraph nodes stream to the UI over SSE, so the agent's reasoning is visible while it runs rather than after.
What I'd do differently
The confidence-check node is still a stub that always returns 1.0, so the conditional retry edge it was meant to trigger never fires. It is wired but inert, and I would rather say that than imply a feedback loop I have not finished. The lecture-slides corpus is also a toy and deserves something messier. Next: extracting the harness into a standalone retrieval-eval tool with recall@k and MRR, per-stage failure attribution, and a GitHub Action that fails builds on regression.