Measured results

Every default in Research Scan was gated on a measurement. This page reports those measurements with the context they were made in, because a recall figure without its topic, profile and judge is not a result.

The full record, including everything that was tried and reverted, is docs/measurements.md in the main repo.

What these numbers do not claim

Stated first, in the project’s own words:

What the numbers do not claim. They come from two curated golden topics. They say the defaults beat the alternatives that were tried on those two topics; they are not a benchmark against other tools, and no comparison against one was run.

Two further limits are worth carrying with every figure below:

Most runs are not independently checkable. They are local artefacts under research/scans/, which is gitignored. The acceptance results are committed under eval/results/; most of the comparison runs are not.

The golden topics are curated, and one carries a reservation. defaults-savings is ratified-with-caveat — accepted with a recorded reservation. llm-lit-search is ratified.

The two golden topics

Both live in the main repo under eval/golden/. Scoring a run against one is an ordinary command:

research-scan eval --topic defaults-savings --run <dir>

V1 acceptance run

Two golden topics, each scanned end to end and then scored by an independent judge model — a different and stronger model than the one that wrote ranked.json.

Both results are committed with full per-rank judge scores and reasons.

Why there are two precision numbers

emit reserves the last two slots of the top 10 for out-of-window classics (selection_reason: foundational), placed after the current work so they read as context rather than as the answer. The judge scored them on the same “does this inform a decision the brief names” scale as everything else — a scale a classic cannot win by construction.

So llm-lit-search reads 0.70 raw and fails the ≥ 0.80 target on that basis, while its seven in-window packets score 7/8 and pass. precision_ge2_in_window is the acceptance number. The raw figure still ships so that the cost of reserving those slots stays visible rather than being quietly discounted.

Two caveats on this table

The foundational scores were not re-rolled. They were produced under an earlier judge prompt that scored every packet on decision-relevance. The judge now scores a foundational packet on whether it is canonical background a newcomer must know. That change applies from the next judge run onward; the committed scores predate it.

The two runs were not at the same setting. This table was measured before profiles existed:

So llm-lit-search’s figures are deep numbers and defaults-savings’s are quick-depth numbers. Eval results are keyed by profile from v0.2.1 onward so this cannot silently recur.

Profile comparison

Measured on both topics with every routed source reporting zero failures.

Recall is monotone in the profile on both topics, and recall per 100 candidates screened falls as the pool grows — which is what a cost dial should look like.

What separates the profiles most is the out-of-window budget (12 / 20 / 30). Topic 1’s remaining misses are all pre-window classics, and each step up admits one more.

Cost and wall clock

Every figure in this section was measured end to end (counts.wall_clock_s, init through emit) on the supported path: a hosting agent driving the pipeline conversationally. None of it is comparable with the stateless screening replay below, which replays a single stage of a different run against recorded inputs.

The one end-to-end run with a recorded wall_clock_s is the acceptance run 2026-08-19-topic2b: 2,230 seconds for a 593-paper pool, or roughly 3.8 seconds per candidate screened.

OpenAlex bills about $0.008 a run at every profile. Agent tokens are the real cost, and they do not fall as fast as the pool: the 120-candidate run still cost about $5.70 in frontier-model tokens and ran only three minutes quicker than the 250-candidate one, because reranking and fixed stage overhead dominate once screening is small.

A measurement rule worth borrowing

One early published figure had to be withdrawn. A topic-2 quick run had reported 6/6 recall — but in that run arXiv had failed all eight queries, returning HTTP 429 throughout. Two clean re-runs give 4/6.

The rule adopted afterwards:

A measured run is not quotable until every routed source reports zero failures.

The corrected table above is the one that stands. It is also why manifest.json counts per-source failures at all: a degraded run that looks like a good number is the failure mode that rule exists to catch.

Screening in controlled replay

In controlled replay, stateless parallel screening reduced the screening stage from 979 seconds to 73 seconds and from $3.00 to $0.76. The implementation remains experimental while end-to-end golden non-inferiority is validated.

The baseline it improves on is the screening stage of one recorded conversational run, 2026-08-26-claim-grounding-sonnet — a different run from the acceptance run in Cost and wall clock above, so the two sets of figures cannot be combined into an end-to-end speedup. This is a replay of one stage against recorded inputs, not an end-to-end scan result, and nothing about it is a default: the shipped path is unchanged. Screening agreement with the baseline fell as well as cost, which is why non-inferiority is still open. The full record, including what was measured and what was only projected, is in docs/measurements.md at the released commit.

Transport evidence

Two end-to-end scans driven over MCP: 23 and 36 tool calls, zero timeouts, zero retries, 47/47 and 49/49 DOIs verified.

These establish that the transport works. They are not golden-set retrieval measurements and are never quotable as such.

Related

• Design decisions — what was tried and rejected.
• Contributing — the measured-or-reverted rule for changes.