# Design decisions What was tried, what shipped, and what was measured and reverted — with the reasoning behind each. Several of the most useful results in this project are negative: changes that sounded right, were built, were measured, and were then reverted. They are recorded rather than quietly dropped, because a rejected idea that leaves no trace gets reinvented. Everything here is drawn from [`docs/measurements.md`](https://github.com/Synectic-Research/research-scan/blob/main/docs/measurements.md). ## Shipped ### Deeper queries, and a scaled pool cap Per-query depth moved from 20 to 40, and the pool cap became a function of how many sources are actually routed rather than a flat number. A `cs` scan with three sources gets a larger cap than a two-source `behavioral` scan, because it has more sources to divide it between. ### Ranked citation-graph expansion Semantic Scholar reference lists are ordered by a blend of citations-per-year and recency within the window, rather than taken in whatever order the API returns. Expansion has a budget; spending it on the arbitrary first N references wastes it. ### A real arXiv source Added as a retrieval source for `cs`, and for `general` when a query is a method query. Measured at the *scaled* cap. At the unscaled 450 cap it was a regression — topic 2 fell from 6/6 to 5/6 — because arXiv's volume crowded out the other sources within a fixed budget. That pairing matters: the source helped, but only in combination with the cap that made room for it. ### The gap round A second round aimed at thinly-covered criteria. It recovered a golden paper on topic 1 (8/10 to 9/10) and was flat on topic 2. Two findings came out of building it. A flat "fewer than five hits" threshold never fires on a real pool — the trigger has to be relative to the other criteria, not absolute. And round one's out-of-window budget cannot be shared with round two; the gap round needs its own. ### The contradicting reserve, keyed on `relation` A slot reserved for counter-results. The interesting part is which field it reads. The obvious choice was the `contradicts` flag. Measured across 21 committed runs, that flag is set 3–9 times more often than `relation: contradicting` — on one golden topic, 5 of 10 emitted papers carried the flag against none with the relation. Keying the guarantee on the flag let papers that *answer* the brief fill a reserve meant for papers that *argue with* it, and the guarantee had effectively never fired. It now reads `relation`. ## Measured and reverted ### Per-seed round-robin admission Expansion admitted candidates round-robin across seed papers, so no single well-connected seed could dominate. It was a substantial regression: topic 1 fell from 7/10 to 4/10, 8/10 to 5/10, and 9/10 to 6/10 across the three profiles. The lesson generalises beyond this codebase: > Fair-share admission helps when the producers are interchangeable and hurts when > agreement between producers is itself the signal. When several independent seeds all cite the same paper, that convergence *is* the evidence. Round-robin admission discards it by construction. ### Collapsing three profiles into two A candidate `standard` at `quick`'s knobs plus the gap trigger, tested against a rule set in advance: match current `standard` recall on both topics with a pool under 420. It missed on both topics and both terms — 7/10 and 4/6 against a bar of 8/10 and 5/6, at pools of 521 and 619 against a bar of 420. It was also the worst row in the table on efficiency: it paid the gap round's screening bill and bought nothing with it. Three profiles stayed. ### Full-bibliography expansion from anchors Expanding across an anchor's entire bibliography rather than a ranked subset. The finding was about a different parameter than the one under test: the out-of-window cap of 20 costs topic 1 two golden papers. Lifting it reaches 10/10 — at the price of roughly eight more screening batches, which is most of an hour of agent time. The out-of-window budget is the real dial, and it is exposed as one. ### Two rerank guards Both were implemented, and in both cases the guard changed nothing: the code-only arm produced a zero-line diff across the committed runs. The project's position on that outcome is worth quoting, because the temptation is always to ship the code anyway: > Publishing the zero-line diff is the honest form of that. ## How these decisions are made The rule in `CONTRIBUTING.md` is short: any change to retrieval, expansion, screening caps or selection must arrive with golden-set eval results. Measured, or reverted. Two supporting rules make that possible: **A measured run is not quotable until every routed source reports zero failures.** Adopted after a degraded run — arXiv returning 429 to every query — produced a number that looked good and was withdrawn. **Never build a golden set out of a run's own output.** A golden set assembled from what the tool already found measures nothing except its own consistency. ## Related - [Measured results](/evaluation/measurements/) — the numbers themselves. - [Contributing](/contributing/) — what a change has to arrive with.