benchmark
HotpotQA distractor — the worst case for pruning
300 multi-hop questions, each with 10 Wikipedia paragraphs (2 gold, 8 distractors), mean 1,333 context tokens. Multi-hop + distractor-heavy is deliberately the hardest public setting for lexical pruning — your single-hop product docs will prune better than this.
| mode | token reduction | answer retention | supporting facts kept | latency |
|---|---|---|---|---|
| conservative | 35.3% | 95.3% | 91.1% | 3.9 ms |
| balanced (default) | 55.5% | 87.5% | 83.2% | 2.3 ms |
| aggressive | 75.6% | 73.8% | 71.4% | 2.2 ms |
Metrics. Answer retention: of the 279/300 span answers present verbatim in the full context, the share still present after pruning. Supporting-fact retention: share of the 740 gold supporting sentences that survive. Tokens counted with o200k_base.
Cost translation. At GPT-4o input pricing ($2.50/M), balanced mode saves ~$1.85 per 1,000 calls on this workload — ~$185/month at 100k calls.
The honest read.Multi-hop bridge questions are the known weak spot: hop-2 sentences can share zero words with the question. Use conservative mode when a missed sentence is expensive; aggressive when a cheap fallback re-ask beats always paying for full context. Scorer changes only ship if they don't lower answer retention at comparable reduction on this subset.
Reproduce it: the harness is scripts/benchmark.mts in the repo, runnable against any HotpotQA subset. Or try it on your own chunks.