Measured results
What has been measured, on which topics, by which judge — and what the numbers explicitly do not claim.
Every default in Research Scan was gated on a measurement. This page reports those measurements with the context they were made in, because a recall figure without its topic, profile and judge is not a result.
The full record, including everything that was tried and reverted, is
docs/measurements.md
in the main repo.
What these numbers do not claim
Stated first, in the project’s own words:
What the numbers do not claim. They come from two curated golden topics. They say the defaults beat the alternatives that were tried on those two topics; they are not a benchmark against other tools, and no comparison against one was run.
Two further limits are worth carrying with every figure below:
Most runs are not independently checkable. They are local artefacts under
research/scans/, which is gitignored. The acceptance results are committed under
eval/results/; most of the comparison runs are not.
The golden topics are curated, and one carries a reservation.
defaults-savings is ratified-with-caveat — accepted with a recorded
reservation. llm-lit-search is ratified.
The two golden topics
| Topic | Field | Status |
|---|---|---|
defaults-savings | Behavioural economics | ratified-with-caveat |
llm-lit-search | LLM literature-search agents | ratified |
Both live in the main repo under eval/golden/. Scoring a run against one is an
ordinary command:
research-scan eval --topic defaults-savings --run <dir>V1 acceptance run
Two golden topics, each scanned end to end and then scored by an independent
judge model — a different and stronger model than the one that wrote
ranked.json.
| Topic | Run | recall@10 | recall@25 | precision_ge2 (raw) | precision_ge2_in_window |
|---|---|---|---|---|---|
defaults-savings (ratified-with-caveat) | 2026-08-19-s3-e2e | 0.50 (5/10) | 0.80 (8/10) | 0.80 (8/10) | 0.875 (7/8) |
llm-lit-search (ratified) | 2026-08-19-topic2b | 0.50 (3/6) | 0.67 (4/6) | 0.70 (7/10) | 0.875 (7/8) |
Both results are committed with full per-rank judge scores and reasons.
Why there are two precision numbers
emit reserves the last two slots of the top 10 for out-of-window classics
(selection_reason: foundational), placed after the current work so they read as
context rather than as the answer. The judge scored them on the same “does this
inform a decision the brief names” scale as everything else — a scale a classic
cannot win by construction.
So llm-lit-search reads 0.70 raw and fails the ≥ 0.80 target on that basis,
while its seven in-window packets score 7/8 and pass.
precision_ge2_in_window is the acceptance number. The raw figure still ships so
that the cost of reserving those slots stays visible rather than being quietly
discounted.
Two caveats on this table
The foundational scores were not re-rolled. They were produced under an earlier
judge prompt that scored every packet on decision-relevance. The judge now scores a
foundational packet on whether it is canonical background a newcomer must know.
That change applies from the next judge run onward; the committed scores predate
it.
The two runs were not at the same setting. This table was measured before profiles existed:
| Acceptance run | per_query | max_candidates | Profile it corresponds to |
|---|---|---|---|
2026-08-19-s3-e2e | 20 | 250 | quick depth and cap, holding standard’s out-of-window cap |
2026-08-19-topic2b | 40 | 675 | deep, three sources scaled |
So llm-lit-search’s figures are deep numbers and defaults-savings’s are
quick-depth numbers. Eval results are keyed by profile from v0.2.1 onward so
this cannot silently recur.
Profile comparison
Measured on both topics with every routed source reporting zero failures.
| Profile | Knobs | T1 pool | T1 recall | T1 per 100 | T2 pool | T2 recall | T2 per 100 |
|---|---|---|---|---|---|---|---|
quick | 20 / 250 / 12 · no gap round | 351 | 7/10 | 0.199 | 362 | 4/6 | 0.184 |
standard | 40 / 450 / 20 · conditional | 565 | 8/10 | 0.142 | 570 | 5/6 | 0.146 |
deep | 40 / scaled / 30 · always | 575 | 9/10 | 0.157 | 805 | 5/6 | 0.104 |
Recall is monotone in the profile on both topics, and recall per 100 candidates screened falls as the pool grows — which is what a cost dial should look like.
What separates the profiles most is the out-of-window budget (12 / 20 / 30). Topic 1’s remaining misses are all pre-window classics, and each step up admits one more.
Cost and wall clock
| Pool | Screening batches | Wall clock |
|---|---|---|
| ~675 candidates | ~27 | 37 min |
| ~250 candidates | ~10 | 22 min |
| ~120 candidates | 5 | 19 min |
The one end-to-end run with a recorded wall clock took 2,230 seconds for a 593-paper pool — roughly 3.8 seconds per candidate screened.
OpenAlex bills about $0.008 a run at every profile. Agent tokens are the real cost, and they do not fall as fast as the pool: the 120-candidate run still cost about $5.70 in frontier-model tokens and ran only three minutes quicker than the 250-candidate one, because reranking and fixed stage overhead dominate once screening is small.
A measurement rule worth borrowing
One early published figure had to be withdrawn. A topic-2 quick run had reported
6/6 recall — but in that run arXiv had failed all eight queries, returning HTTP 429
throughout. Two clean re-runs give 4/6.
The rule adopted afterwards:
A measured run is not quotable until every routed source reports zero failures.
The corrected table above is the one that stands. It is also why manifest.json
counts per-source failures at all: a degraded run that looks like a good number is
the failure mode that rule exists to catch.
Transport evidence
Two end-to-end scans driven over MCP: 23 and 36 tool calls, zero timeouts, zero retries, 47/47 and 49/49 DOIs verified.
These establish that the transport works. They are not golden-set retrieval measurements and are never quotable as such.
Related
- Design decisions — what was tried and rejected.
- Contributing — the measured-or-reverted rule for changes.
Last updated Aug 21, 2026