Skip to content

What counts as evidence

Status: in force · Enforced by: tests/test_routes.py, tests/test_graph.py, tests/test_catalogue_probes.py, tests/test_catalogue.py

The ranking is computed from evidence, so what qualifies as evidence is not a methodological footnote — it is the input to every routing decision the tool makes. Each rule below exists because breaking it produced a wrong number that looked right.

A verdict has three states, not two

RESOLVED, EMPTY, FAILED — and absence is a fourth thing, meaning the accession was never queried.

Collapsing empty into failure made bioproject->geo_series look broken when most of its empties are correct: a PRJEB project genuinely has no GEO series, and saying so is the right answer. Collapsing failure into empty is worse — it manufactures a true negative out of a request that never completed, which is how a stale experiment list becomes indistinguishable from a broken one.

fetch survey compare keys agreement on (outcome, results) for the same reason. An earlier version reported a FAILED and an EMPTY route as agreeing, because both returned nothing.

Unique results, never rows

The 2026-05 predecessor to this project counted rows. 69,923 of its 824,216 "runs" were within-BioProject duplicates — an 8.5% inflation that made the sra-db-be routes look closer to ENA than they are.

Every unique_results figure in graph.py is a distinct-accession count, and sra_be.column() deduplicates for this reason.

That sentence was false for two routes until 2026-10-19. The 2026-05 BioProject → GEO series routes recorded 15,186 and 15,185 "unique" series, which were rows: five series are reached from more than one project, GSE305915 from three, so the distinct counts are 15,180 and 15,179. Splitting the routes by archive recounted them from the result tables, which is how it surfaced.

The corpus name must identify the accession set exactly

The name is persisted with every verdict as provenance. If it does not pin down which accessions were queried, two runs recorded under one name are not comparable, and nothing in the data says so.

This is why:

  • --limit is never used to produce evidence. A limited run records its verdicts against the full corpus name, so two differently-limited runs look comparable and are not. sample:<n>@<corpus> exists instead: seeded, reproducible, and named for its size.
  • geo-sample-15 holding 10 accessions was a bug worth fixing, not a rounding detail. n // strata silently returned 10 for any n below 20.
  • A sampled draw records its parent. sample:3000@reprocessed-srr says both the size and where it came from; results-of:<route>@<corpus> says which survey produced the population.
  • A placeholder is not a query. The 2026-05 scripts sent the table's - as a BioProject, recorded its failure, and counted 12,756 projects queried where there are 12,755. The harness parses its corpora and never sends one; catalogue/route_splits.json names the old scripts' as unparsed.

hard-cases is a gate, not evidence

Every route must produce a verdict for all 46 curated accessions before it earns a larger run. None of those verdicts is recorded as RouteEvidence.

hard-cases is adversarial by construction: every accession in it is there because it broke something. Its empty rate is high on purpose. Feeding that into the ranking would order routes by how often they were deliberately pointed at withdrawn projects and private series.

Tier one proves a route runs. Only a representative corpus measures how well.

A probe is not evidence

The link catalogue confirms each link no route measures with one live request: GSE60450's gds record names GPL13112, so GEO series → GEO platform exists. That is all it shows.

A probe is one accession out of one. It cannot say how often a link answers, how often it fails, or whether it agrees with another, so it has none of the properties ranked() sorts on, and recording it as RouteEvidence would put a 100% yield into the ranking on the strength of a single request. The rule is the same as for hard-cases, for a related reason: both are chosen, not sampled.

So a probe:

  • never becomes RouteEvidence, and a catalogue link never enters REGISTRY without a survey behind it;
  • is drawn on the route graph as a dotted line with no counts, no rank and no ink;
  • is kept, with its date and accession, in catalogue/probes.json, and a test fails if a probe is recorded beside a measured link, or used an accession other than its link's fixed one.

A probe that found nothing leaves its link documented, not confirmed, which is not the same as absent: the mechanism exists, and the record chosen did not carry the field. Promoting an untested link means surveying it, over a named corpus, like any other route.

A probe into a file or a document confirms its link only when what it found is usable: a read file that downloads at the size the archive published, has a published md5 and size, and holds both reads a 10x run needs; a supplementary file that downloads and whose size can be checked; a metadata document that downloads and parses. A link that names a file nobody can fetch is not a route to that file. Each check is recorded with the probe, so an unconfirmed link says which check it failed.

A probe is one accession out of one; a link survey is the same request over a seeded draw of 300 of the link's source node (or all of them, where there are fewer). That is measurement. It says how often a link answers, how often what it finds is usable, and how often it fails, over a named and reproducible population. So it has a status of its own, surveyed, that outranks a probe.

It is still not RouteEvidence, for a reason of scale rather than of kind. At 300, a yield's 95% interval is up to ±5.7 points wide. The ranking separates routes on a 1% band because noise once ranked above substance at far smaller margins. A draw that cannot resolve differences ten times larger than the band would, if ranked, put a coin flip in charge of the resolver's order.

So a link survey:

  • records under link:<link id>, never under a route's id, even where the two are the same string;
  • is drawn on the route graph with counts and no rank;
  • is never read by graph.py or resolver.py, and a test checks the imports;
  • excludes the gates, hard-cases and probe-families, which are chosen, not sampled: every probe accession is there because its link answered;
  • is read from the committed exports, so link_splits.json regenerates on any checkout.

Promoting a surveyed link to a route is a per-link human decision. It needs a ranking-scale draw (sample:3000), a RouteFn, and a RouteEvidence block like any other route. The link survey is what shows which links are worth that.

A route's split is not new evidence

INSDC objects are split by archive on the graph, so one route covers several links: bioproject->run:ena_filereport reaches SRRs, ERRs and DRRs. How it did on each is read out of its own recorded verdicts (catalogue/splits.py), by the archive each accession's prefix names and the format each file's name names. That is filtered measurement: the same survey, the same corpus, the same date, cut by source and target. It adds no evidence and is never ranked; RouteRegistry.ranked() still sees one route with one RouteEvidence.

Two things follow. A split is drawn only where the survey actually queried that source: experiment->run:ena_filereport was surveyed on GEO series' experiments, which are all SRXs, so it measured nothing about an ERX and that link is probed like any untested one. And a split whose source was queried and never reached the target is a measured absence, listed and not drawn: run->file:ena_submitted found submitted files for all 246 ERRs it was asked about and for none of 2,737 SRRs.

An undated number is not evidence

RouteEvidence requires surveyed_on, and a test asserts it is not in the future. Archive contents drift — 0.77% of runs that reprocessed successfully no longer exist in either archive — so a coverage figure without a date makes a claim it cannot support.

Dates are UTC. A survey finishing at 00:12 BST is 23:12 UTC the previous day; recording local time stamped two blocks a day ahead and CI, an hour behind, rejected them as being in the future.

Every measured route has a page, and that page cites a number

Two tests enforce it: a measured route with no knowledge-base page fails the suite, and so does a page containing no digits. The rule is only worth having if it cannot be quietly skipped, and a route whose evidence lives solely in a dataclass is not reviewable by anyone who does not read the source.

Recall and precision are not symmetric, and the asymmetry is stated

The assay layer reports 90.9% recall against a corpus that is 10x by construction, and reports no precision figure at all — there is no labelled non-10x corpus at comparable scale to measure it against.

Claiming one would be exactly the invented default this project exists to avoid. What can be said instead is narrower and checkable: the trigger phrases are platform names, and scRNA-seq is deliberately not among them because it appears in Smart-seq, Drop-seq and inDrop submissions too. Adding it would buy recall by manufacturing false positives.

Chemistry was the same until the reads were read. Agreement between a protocol's kit and the reads' is now measured, by fetch inspect over 240 labelled runs on the 2026-10 benchmark: the reads named the same kit for 186, and the metadata was wrong for 26. The labels came from metadata, never from the reads. That is what keeps the measurement from grading the classifier against itself.

A census is not always worth its cost

Some routes were censused over the full corpus and some over a named seeded draw. That is a judgement about cost, and it is recorded rather than hidden:

  • Censused where the population definition was the point — arrayexpress-ena (20,693) because the ENA link is known to exist from outside the route being measured, so every miss is a countable recall failure.
  • Sampled where the answer converges long before the corpus does. The run, sample and experiment columns hold 100k–263k accessions at one request each; a 3,000-run draw answers to within a percentage point in a twentieth of the time.
  • Not re-censused at all for the two sra_be/efetch BioProject→run routes: roughly five hours each, already 4–14% behind ENA on the 2026-05 figures, and the ranking outcome cannot change. Recorded as a scoping decision rather than left looking like an oversight.

What would change this

Any of these rules is reversible on evidence that it produces a worse answer. The one that is not negotiable is the meta-rule: a number in this project carries its corpus and its date, or it is not used.