Run this before and after every pipeline change. Baseline once (healthy on a clean corpus: Hit-Rate@3 ≥ 90%, groundedness ≥ 95%), change exactly one variable — chunk size, hybrid weights, reranker, the grounding prompt — re-run, and diff the report. Everything is offline; settings come from the app's app_config.db.
Resolved configuration
Read-only — the harness reads these from the app's settings DB. Change them in the app's Settings page. Embedding model must match how the index was built (or the harness re-embeds the chunks).
Golden set
Checking…
Generate a synthetic golden set from your own chunks: for a sample of indexed chunks the model writes a realistic question answerable from that one chunk, plus the ideal answer. The chunk becomes the gold retrieval target; the ideal answer is the correctness reference. The set is saved to golden.json and stays stable across runs so diffs are meaningful.
Generate
One model call per chunk. Chunks shorter than the minimum (boilerplate, fragments) are skipped.
Golden set 0
No golden set yet — generate one above.
Change exactly one variable, then re-run and diff. Failed questions print at the bottom — read them; they tell you which part of the pipeline to fix next.
The variables
Grounding prompt under test (A/B it)
To change chunk size: rebuild the index in the app with the new size, then re-run here. The harness measures whatever the app currently serves.
Every run is saved to runs/ as JSON + a standalone HTML report. Open a report, or compare two runs.
| Run | When | Retrieval | Hit@3 | Ground | Correct | Report |
|---|
No runs yet — run an evaluation first.