A fully-offline evaluation rig for your RAG pipeline: a synthetic golden set from your own chunks, Retrieval Hit-Rate@k, and LLM-as-judge groundedness + correctness — with a report you diff before and after every change. No LangSmith account needed.
| Question | Retrieved | Grounded | Correct |
|---|---|---|---|
| Allowed hours for recovery agents? | #1 | grounded | correct |
| Email & phone on the resume? | #1 | grounded | correct |
| Languages & tech the candidate knows? | #2 | grounded | partial |
| What to do if an agent visits? | #1 | grounded | correct |
Resume.pdf · chunk 3 · retrieved rank #2The Evaluate view — verdict vs healthy thresholds, metric cards with run-to-run deltas, the per-question table, and the failed questions that point at the fix.
It builds an evaluation set from your own indexed chunks, scores retrieval and generation with real metrics, and prints a report you diff run to run. Change exactly one variable — chunk size, hybrid weights, reranker, the grounding prompt — re-run, and read the deltas. Healthy on a clean corpus: Hit-Rate@3 ≥ 90%, groundedness ≥ 95%.
For a sample of the chunks the app already indexed, the model writes a realistic question answerable from that one chunk plus the ideal answer. The chunk is the gold target; the answer is the correctness reference. Saved to golden.json, stable across runs.
For every golden question, is the gold chunk in the top-k? Reported at @1 / @3 / @5 with MRR — the fast, token-free signal you sweep first when tuning chunking or retrieval.
An LLM judge scores whether every claim in the answer is supported by the retrieved sources — catching hallucination, the answer inventing facts the retrieval never surfaced. Graded against the context only.
A second judge scores the answer against the golden ideal answer — catching “grounded but wrong”: fluent, cited, and still not what was asked. Groundedness and correctness are independent for a reason.
Dense (the app default, reusing its saved vectors), from-scratch BM25, weighted hybrid, and an optional cross-encoder reranker — the exact knobs the harness exists to compare.
The golden set is seeded and stable, so a diff reflects your change, not sampling noise. Every metric shows a green/red delta against the previous run — Hit-Rate@3 85.7% → 100% ▲+14.3.
A console summary and a standalone HTML report in the app’s design language — headline metrics with PASS/FAIL, and every failed question at the bottom with the reason. Read them: they point at the fix.
It owns no credentials. Region, Bedrock key, model and embedding model are read straight from the app’s app_config.db; the chunks and saved vectors come from its index/. It measures the real thing.
No LangSmith account, no tracker, no telemetry — the only network call is to the same Bedrock endpoint the app uses. When you outgrow this, the same golden.json uploads to LangSmith unchanged.
Every RAG tuning decision is a trade-off you can’t eyeball. This turns “it feels better” into a measured, repeatable verdict — and shows you exactly which questions regressed.
Run once to set the bar, then re-run after any pipeline edit and diff. Regressions show up as red deltas before they reach users.
Rebuild the index at a new chunk size in the app, re-run, and watch Hit-Rate move. Too big buries the answer; too small splits it — the number tells you which.
Sweep dense vs. BM25 vs. hybrid α, and flip the cross-encoder reranker — a token-free retrieval-only pass finds the winner before you spend on judging.
Swap in a different grounding prompt and see groundedness and correctness move independently — proof a wording change actually reduced hallucination.
harness.py run exits non-zero below the healthy baseline, so a pipeline change that drops Hit-Rate or groundedness can fail the build automatically.
A small, dependency-light Python backend and a vanilla-JS frontend with no build step — reusing the sibling RAG app’s data and settings verbatim.
Each golden question flows through retrieval and generation, and comes out the other side with a Hit-Rate, a groundedness verdict and a correctness verdict.
golden question ──▶ retrieve top-K ──▶ Hit-Rate@k (is the gold chunk in?) │ ▼ grounded answer ──▶ judge groundedness (supported by sources?) └──▶ judge correctness (matches the ideal?) │ ▼ aggregate ─▶ grade vs thresholds ─▶ report + diff
The harness reads the chunks the app persisted, samples the substantive ones, and asks the model for one question answerable from each chunk plus the ideal answer. The chunk becomes the gold retrieval target; the ideal answer becomes the correctness reference. Seeded, so it stays put across runs.
Each question is retrieved with the chosen mode — dense (reusing the app’s saved vectors), BM25, or hybrid, optionally reranked. If the gold chunk lands in the top-k it’s a hit; @1/@3/@5 and MRR fall out of one search per question.
The pipeline answers each question from only its retrieved context, citing sources — the same grounding contract the app enforces. Single-shot (no multi-hop) so the metrics isolate the variables you tune.
Two independent LLM judges score each answer: groundedness against the retrieved context (hallucination), correctness against the golden ideal answer (grounded-but-wrong). Each returns a three-way verdict and a one-line reason.
Metrics roll up to Hit-Rate@k, MRR, groundedness and correctness rates, graded PASS/FAIL against the healthy-baseline thresholds. Any question that missed retrieval, or was judged ungrounded or incorrect, lands in the failures list with its reason.
Every run is saved to runs/ as JSON plus a standalone HTML report, so any two runs diff into green/red deltas. Read the failed questions printed at the bottom — a retrieval miss points at chunking; ungrounded at the grounding prompt; incorrect despite a hit at generation.
It reads the sibling app’s app_config.db for the Bedrock region, key and model — so it runs against your real pipeline out of the box.
# install pip install -r requirements.txt # CLI — the "run this script" loop python harness.py config --test # confirm it sees app_config.db + Bedrock python harness.py generate # build golden.json from your chunks python harness.py run --diff last # evaluate, report, diff the previous run # ...or the web UI python server.py # → http://127.0.0.1:8300 (this page at /overview)