Design decisions

Several of the most useful results in this project are negative: changes that sounded right, were built, were measured, and were then reverted. They are recorded rather than quietly dropped, because a rejected idea that leaves no trace gets reinvented.

Everything here is drawn from docs/measurements.md.

Shipped

Deeper queries, and a scaled pool cap

Per-query depth moved from 20 to 40, and the pool cap became a function of how many sources are actually routed rather than a flat number. A cs scan with three sources gets a larger cap than a two-source behavioral scan, because it has more sources to divide it between.

Ranked citation-graph expansion

Semantic Scholar reference lists are ordered by a blend of citations-per-year and recency within the window, rather than taken in whatever order the API returns. Expansion has a budget; spending it on the arbitrary first N references wastes it.

A real arXiv source

Added as a retrieval source for cs, and for general when a query is a method query. Measured at the scaled cap. At the unscaled 450 cap it was a regression — topic 2 fell from 6/6 to 5/6 — because arXiv’s volume crowded out the other sources within a fixed budget.

That pairing matters: the source helped, but only in combination with the cap that made room for it.

The gap round

A second round aimed at thinly-covered criteria. It recovered a golden paper on topic 1 (8/10 to 9/10) and was flat on topic 2.

Two findings came out of building it. A flat “fewer than five hits” threshold never fires on a real pool — the trigger has to be relative to the other criteria, not absolute. And round one’s out-of-window budget cannot be shared with round two; the gap round needs its own.

The contradicting reserve, keyed on relation

A slot reserved for counter-results. The interesting part is which field it reads.

The obvious choice was the contradicts flag. Measured across 21 committed runs, that flag is set 3–9 times more often than relation: contradicting — on one golden topic, 5 of 10 emitted papers carried the flag against none with the relation. Keying the guarantee on the flag let papers that answer the brief fill a reserve meant for papers that argue with it, and the guarantee had effectively never fired.

It now reads relation.

Measured and reverted

Per-seed round-robin admission

Expansion admitted candidates round-robin across seed papers, so no single well-connected seed could dominate.

It was a substantial regression: topic 1 fell from 7/10 to 4/10, 8/10 to 5/10, and 9/10 to 6/10 across the three profiles.

The lesson generalises beyond this codebase:

Fair-share admission helps when the producers are interchangeable and hurts when agreement between producers is itself the signal.

When several independent seeds all cite the same paper, that convergence is the evidence. Round-robin admission discards it by construction.

Collapsing three profiles into two

A candidate standard at quick’s knobs plus the gap trigger, tested against a rule set in advance: match current standard recall on both topics with a pool under 420.

It missed on both topics and both terms — 7/10 and 4/6 against a bar of 8/10 and 5/6, at pools of 521 and 619 against a bar of 420. It was also the worst row in the table on efficiency: it paid the gap round’s screening bill and bought nothing with it.

Three profiles stayed.

Full-bibliography expansion from anchors

Expanding across an anchor’s entire bibliography rather than a ranked subset.

The finding was about a different parameter than the one under test: the out-of-window cap of 20 costs topic 1 two golden papers. Lifting it reaches 10/10 — at the price of roughly eight more screening batches, which is most of an hour of agent time. The out-of-window budget is the real dial, and it is exposed as one.

Two rerank guards

Both were implemented, and in both cases the guard changed nothing: the code-only arm produced a zero-line diff across the committed runs.

The project’s position on that outcome is worth quoting, because the temptation is always to ship the code anyway:

Publishing the zero-line diff is the honest form of that.

How these decisions are made

The rule in CONTRIBUTING.md is short: any change to retrieval, expansion, screening caps or selection must arrive with golden-set eval results. Measured, or reverted.

Two supporting rules make that possible:

A measured run is not quotable until every routed source reports zero failures. Adopted after a degraded run — arXiv returning 429 to every query — produced a number that looked good and was withdrawn.

Never build a golden set out of a run’s own output. A golden set assembled from what the tool already found measures nothing except its own consistency.

Related

• Measured results — the numbers themselves.
• Contributing — what a change has to arrive with.