# Evidence-first research

Why a verified evidence object beats a chat answer, and what the retrieve, judge, verify, refine loop buys you.

Ask a language model for the important recent papers on a topic and you get a
list. Some of it is right. Some of it is a plausible title attached to a plausible
author in a plausible year, and you cannot tell which is which by reading it.

The failure is not that the model is careless. It is that a generated list and a
retrieved list look identical on the page. Nothing in the output records where a
title came from, whether the DOI resolves, or what was searched and discarded on
the way.

Research Scan is built the other way around: an evidence-first workflow, where
retrieval and verification happen before anything is ranked, and the record of
both ships with the result.

## An evidence object, not an answer

The output of a scan is a directory of validated JSON. Each paper arrives as an
`EvidencePacket` carrying, at minimum:

- the bibliographic record, from a scholarly API rather than from memory
- a verification record showing that the DOI, title, year and first author were
  checked against the live record
- the scores it was given, per sub-criterion and overall
- an argument — what it found, why it matters here, and at least one limitation
- how it was discovered: which query, or which citation-graph hop

You can disagree with the ranking. You can re-run the rerank with a different
rubric. What you cannot do is confuse an invented paper for a real one, because
the record either resolves or is marked `[UNVERIFIED — check manually]`.

## The loop

A scan is not one pass. It is a loop that widens, then narrows, then checks.

**Retrieve.** A query plan of 6–8 queries goes out across the routed sources.
Mandatory query types make the plan argue with itself: alongside the `direct`
query there is a `terminology` query for the other vocabulary the field uses, a
`contradictory` query aimed at the brief's own premise, and a `review` query.

**Judge.** Every candidate is scored 0–3 against the brief's sub-criteria, on the
title and abstract it was given — not on citation count, and not on venue, which
the screener is deliberately never shown.

**Expand.** Everything scoring 2 or 3 becomes a seed for citation-graph expansion
over references, citations and recommendations. What comes back is screened the
same way. This is where the papers a keyword search would never surface arrive.

**Check coverage, then refine.** Coverage counts, per sub-criterion, how many kept
papers actually cover it. If the criteria came back unevenly covered, the agent
writes one or two gap queries aimed at the thinnest, and a second round adds what
they find. The pool only grows: round one's papers and their scores are never
discarded.

**Verify.** Every ranked paper is checked against the live record before it ships.
Verification is last, so it runs on the papers that will actually be delivered.

**Emit.** Selection rules apply — the guaranteed review and counter-result slots,
the per-author cap, the out-of-window budget — and the deliverable renders.

## Why verify before you rank, not after

Ranking a list you have not verified means spending the most expensive judgement
in the pipeline on records that may not exist. Verifying afterwards means
discovering the problem once the answer has already been written.

Doing it in this order has a second effect worth naming. Because verification is a
deterministic step with a recorded outcome, a paper that fails it is *flagged and
kept*, not quietly dropped and not quietly fixed. A reader can see exactly which
records disagreed with the live source and decide for themselves. Repairing a DOI
from a model's memory would erase precisely the signal you needed.

## What this does not give you

An evidence-first pipeline narrows the failure modes; it does not eliminate them.
Screening and retrieval run on titles, abstracts and metadata — no PDF is fetched,
parsed or read — so a paper whose abstract undersells it can still be scored low.
The scan finds 5–10 papers that bear on a question. It is not a systematic review,
and there is no PRISMA trail.

Those limits are in the [measured results](/evaluation/measurements/), stated as
numbers rather than as reassurance.