support-model-prompt-retrieval
Warnings
- Case-level pass-rate intervals and p-values treat each case attempt as independent; attempts on the same case are correlated, so read them as approximate.
Quality, cost, latency, and reliability
| Variant | Case pass rate | Mean score | Mean latency | Mean cost / trial | Reliability | Consistency |
|---|---|---|---|---|---|---|
| Baseline (large model, prompt v3) baselinepareto baseline · model=large, prompt=v3, retrieval=bm25 |
90.0% 74.4%–96.5% | 0.900 | 1380.4 ms | $0.0210 6/6 trials priced |
50.0% 6/6 completed |
60.0% 2 flaky |
| Small model pareto small-model · model=small, prompt=v3, retrieval=bm25 |
70.0% 52.1%–83.3% | 0.700 | 654.2 ms | $0.0045 6/6 trials priced |
16.7% 6/6 completed |
40.0% 3 flaky |
| Terse prompt terse-prompt · model=large, prompt=v4-terse, retrieval=bm25 |
70.0% 52.1%–83.3% | 0.700 | 1116.9 ms | $0.0155 6/6 trials priced |
16.7% 6/6 completed |
40.0% 3 flaky |
| Reranked retrieval pareto reranked-retrieval · model=large, prompt=v3, retrieval=bm25+rerank |
96.7% 83.3%–99.4% | 0.967 | 1706.3 ms | $0.0240 6/6 trials priced |
83.3% 6/6 completed |
80.0% 1 flaky |
pass@k and pass^k
pass@k: chance at least one of k attempts passes. pass^k: chance all k attempts pass. Averaged over cases.
| Variant | pass@1 | pass^1 | pass@3 | pass^3 | pass@6 | pass^6 |
|---|---|---|---|---|---|---|
| Baseline (large model, prompt v3) | 90.0% | 90.0% | 100.0% | 74.0% | 100.0% | 60.0% |
| Small model | 70.0% | 70.0% | 95.0% | 45.0% | 100.0% | 40.0% |
| Terse prompt | 70.0% | 70.0% | 89.0% | 51.0% | 100.0% | 40.0% |
| Reranked retrieval | 96.7% | 96.7% | 100.0% | 90.0% | 100.0% | 80.0% |
Against the baseline
| Candidate | Case pass rate Δ | Score Δ | Latency Δ | Cost Δ | Regressions |
|---|---|---|---|---|---|
| small-model model: large → small |
-20.0% not significant (p=0.053) |
-0.200 | -726.1 ms | -0.0165 | pass rate, score 2 case(s) regressed · 0 improved |
| terse-prompt prompt: v3 → v4-terse |
-20.0% not significant (p=0.053) |
-0.200 | -263.5 ms | -0.0055 | pass rate, score 2 case(s) regressed · 0 improved |
| reranked-retrieval retrieval: bm25 → bm25+rerank |
+6.7% not significant (p=0.301) |
+0.067 | +325.9 ms | +0.0030 | latency, cost 0 case(s) regressed · 2 improved |
Test-level heatmap
Each cell: passes/attempts and mean score; colour follows mean score. Dashed cells are flaky (mixed outcomes).
| refund-window | refund-damaged | escalate-legal | shipping-eta | password-reset | |
|---|---|---|---|---|---|
| Baseline (large model, prompt v3) | 6/61.00 | 5/60.83 | 6/61.00 | 4/60.67 | 6/61.00 |
| Small model | 6/61.00 | 2/60.33 | 3/60.50 | 4/60.67 | 6/61.00 |
| Terse prompt | 1/60.44 | 5/60.83 | 6/61.00 | 3/60.50 | 6/61.00 |
| Reranked retrieval | 6/61.00 | 6/61.00 | 6/61.00 | 5/60.83 | 6/61.00 |
Factor levels
Variants sharing a factor value, averaged. Only factors with at least two values are shown.
| Factor | Value | Variants | Pass rate | Score | Latency | Cost | Reliability |
|---|---|---|---|---|---|---|---|
| model | large | baseline, terse-prompt, reranked-retrieval | 85.6% | 0.856 | 1401.2 ms | $0.0202 | 50.0% |
| model | small | small-model | 70.0% | 0.700 | 654.2 ms | $0.0045 | 16.7% |
| prompt | v3 | baseline, small-model, reranked-retrieval | 85.6% | 0.856 | 1247.0 ms | $0.0165 | 50.0% |
| prompt | v4-terse | terse-prompt | 70.0% | 0.700 | 1116.9 ms | $0.0155 | 16.7% |
| retrieval | bm25 | baseline, small-model, terse-prompt | 76.7% | 0.767 | 1050.5 ms | $0.0137 | 27.8% |
| retrieval | bm25+rerank | reranked-retrieval | 96.7% | 0.967 | 1706.3 ms | $0.0240 | 83.3% |
Randomization
Execution order: terse-prompt#4, baseline#6, reranked-retrieval#1, reranked-retrieval#6, terse-prompt#2, small-model#5, terse-prompt#6, small-model#6, terse-prompt#3, terse-prompt#5, baseline#2, terse-prompt#1, reranked-retrieval#4, small-model#1, small-model#4, baseline#3, reranked-retrieval#5, baseline#5, reranked-retrieval#2, small-model#2, small-model#3, baseline#1, baseline#4, reranked-retrieval#3
Generated by AgenticLens · 2026-10-12T21:04:28.082303+00:00