# Measured results

What has been measured, on which topics, by which judge — and what the numbers explicitly do not claim.

Every default in Research Scan was gated on a measurement. This page reports those
measurements with the context they were made in, because a recall figure without
its topic, profile and judge is not a result.

The full record, including everything that was tried and reverted, is
[`docs/measurements.md`](https://github.com/Synectic-Research/research-scan/blob/main/docs/measurements.md)
in the main repo.

## What these numbers do not claim

Stated first, in the project's own words:

> **What the numbers do not claim.** They come from two curated golden topics.
> They say the defaults beat the alternatives that were tried on those two topics;
> they are not a benchmark against other tools, and no comparison against one was
> run.

Two further limits are worth carrying with every figure below:

**Most runs are not independently checkable.** They are local artefacts under
`research/scans/`, which is gitignored. The acceptance results are committed under
`eval/results/`; most of the comparison runs are not.

**The golden topics are curated, and one carries a reservation.**
`defaults-savings` is `ratified-with-caveat` — accepted with a recorded
reservation. `llm-lit-search` is `ratified`.

## The two golden topics

| Topic | Field | Status |
|---|---|---|
| `defaults-savings` | Behavioural economics | `ratified-with-caveat` |
| `llm-lit-search` | LLM literature-search agents | `ratified` |

Both live in the main repo under `eval/golden/`. Scoring a run against one is an
ordinary command:

```bash
research-scan eval --topic defaults-savings --run <dir>
```

## V1 acceptance run

Two golden topics, each scanned end to end and then scored by an **independent
judge model** — a different and stronger model than the one that wrote
`ranked.json`.

| Topic | Run | recall@10 | recall@25 | `precision_ge2` (raw) | `precision_ge2_in_window` |
|---|---|---|---|---|---|
| `defaults-savings` (`ratified-with-caveat`) | `2026-08-19-s3-e2e` | 0.50 (5/10) | 0.80 (8/10) | 0.80 (8/10) | **0.875 (7/8)** |
| `llm-lit-search` (`ratified`) | `2026-08-19-topic2b` | 0.50 (3/6) | 0.67 (4/6) | 0.70 (7/10) | **0.875 (7/8)** |

Both results are committed with full per-rank judge scores and reasons.

### Why there are two precision numbers

`emit` reserves the last two slots of the top 10 for out-of-window classics
(`selection_reason: foundational`), placed after the current work so they read as
context rather than as the answer. The judge scored them on the same "does this
inform a decision the brief names" scale as everything else — **a scale a classic
cannot win by construction.**

So `llm-lit-search` reads 0.70 raw and fails the ≥ 0.80 target on that basis,
while its seven in-window packets score 7/8 and pass.
`precision_ge2_in_window` is the acceptance number. The raw figure still ships so
that the cost of reserving those slots stays visible rather than being quietly
discounted.

### Two caveats on this table

**The foundational scores were not re-rolled.** They were produced under an earlier
judge prompt that scored every packet on decision-relevance. The judge now scores a
`foundational` packet on whether it is canonical background a newcomer must know.
That change applies from the next judge run onward; the committed scores predate
it.

**The two runs were not at the same setting.** This table was measured before
profiles existed:

| Acceptance run | per_query | max_candidates | Profile it corresponds to |
|---|---|---|---|
| `2026-08-19-s3-e2e` | 20 | 250 | **`quick`** depth and cap, holding `standard`'s out-of-window cap |
| `2026-08-19-topic2b` | 40 | 675 | **`deep`**, three sources scaled |

So `llm-lit-search`'s figures are `deep` numbers and `defaults-savings`'s are
`quick`-depth numbers. Eval results are keyed by profile from v0.2.1 onward so
this cannot silently recur.

## Profile comparison

Measured on both topics with every routed source reporting zero failures.

| Profile | Knobs | T1 pool | T1 recall | T1 per 100 | T2 pool | T2 recall | T2 per 100 |
|---|---|---|---|---|---|---|---|
| `quick` | 20 / 250 / 12 · no gap round | 351 | 7/10 | **0.199** | 362 | 4/6 | **0.184** |
| `standard` | 40 / 450 / 20 · conditional | 565 | 8/10 | 0.142 | 570 | 5/6 | 0.146 |
| `deep` | 40 / scaled / 30 · always | 575 | **9/10** | 0.157 | 805 | **5/6** | 0.104 |

Recall is monotone in the profile on both topics, and recall per 100 candidates
screened falls as the pool grows — which is what a cost dial should look like.

What separates the profiles most is the out-of-window budget (12 / 20 / 30).
Topic 1's remaining misses are all pre-window classics, and each step up admits
one more.

## Cost and wall clock

| Pool | Screening batches | Wall clock |
|---|---|---|
| ~675 candidates | ~27 | **37 min** |
| ~250 candidates | ~10 | **22 min** |
| ~120 candidates | 5 | **19 min** |

The one end-to-end run with a recorded wall clock took 2,230 seconds for a
593-paper pool — roughly 3.8 seconds per candidate screened.

OpenAlex bills about $0.008 a run at every profile. **Agent tokens are the real
cost**, and they do not fall as fast as the pool: the 120-candidate run still cost
about $5.70 in frontier-model tokens and ran only three minutes quicker than the
250-candidate one, because reranking and fixed stage overhead dominate once
screening is small.

## A measurement rule worth borrowing

One early published figure had to be withdrawn. A topic-2 `quick` run had reported
6/6 recall — but in that run arXiv had failed all eight queries, returning HTTP 429
throughout. Two clean re-runs give 4/6.

The rule adopted afterwards:

> A measured run is not quotable until every routed source reports zero failures.

The corrected table above is the one that stands. It is also why `manifest.json`
counts per-source failures at all: a degraded run that looks like a good number is
the failure mode that rule exists to catch.

## Transport evidence

Two end-to-end scans driven over MCP: 23 and 36 tool calls, zero timeouts, zero
retries, 47/47 and 49/49 DOIs verified.

These establish that the transport works. They are **not** golden-set retrieval
measurements and are never quotable as such.

## Related

- [Design decisions](/evaluation/design-decisions/) — what was tried and rejected.
- [Contributing](/contributing/) — the measured-or-reverted rule for changes.