Skip to content

The file layer

A run accession is not a deliverable. The deliverable is a set of files that exist, are complete, and are what the submitter actually deposited. Those three properties fail independently, and the archives disagree about all of them.

Why every route is asked, not just the first

Identity routes answer the same question, so the resolver's union is a corroboration: two routes returning the same run are agreeing. File routes do not answer the same question, so their union is a set of competing offers that the file layer has to choose between:

Route What it holds
run->file:ena_fastq ENA's own derived fastqs, generated from whatever was deposited
run->file:ena_submitted The submitter's original deposit, in its original format
run->file:ena_sra The NCBI-format archive object, mirrored at EBI
run->file:sdl NCBI's view of all of the above, plus cloud mirrors and retrieval cost

A run can have derived fastqs and no submitted deposit, a submitted deposit and no derived fastqs, or both. All three occur in the first two runs the tier-one corpus offers:

SRR25056225   ena_fastq: 2 files   ena_submitted: none     sdl: the .sra object
ERR2861957    ena_fastq: none      ena_submitted: 1 BAM    sdl: the BAM and the .sra object

Neither route is wrong about the other's run. Each returns empty and means it.

Two kinds of judgement, kept apart

Which route to try first is an evidence question. It is answered by RouteRegistry.ranked() from measured survey results, like every other route ranking in this project, and no file route ships a hand-written preference.

Which files to use, once several routes have answered, is a requirements question — and the requirements belong to the downstream pipeline, not to any archive. In order:

  1. A complete read pair beats everything. Half a pair is not half a download, it is an unusable one.
  2. A file set the archive's own metadata proves incomplete ranks below every format that is merely inconvenient. See below.
  3. Format: fastq, then BAM, then CRAM, then .sra. STARsolo reads fastq; an .sra object must be dumped before it is anything at all. This is a statement about the tool, not about the archive.
  4. Free before paid, then checksummed before not, then more files, then route id for determinism.

Every alternative is carried alongside the recommendation. A recommendation the caller cannot argue with is one they cannot check.

The two traps that are not visible in a file listing

An index file is not a data file

x.bam.bai contains .bam. Any classifier that tests for the data suffix first calls every index in the archive by the name of the thing it indexes, and ERR2861957's .bai is 16 bytes — so counting it as data turns a one-file submission into a pair without anything looking odd.

A mate marker means nothing outside a fastq name

_1 means "mate 1" by convention in fastq naming and by coincidence everywhere else. E-MTAB-8060's runs deposit Sample_1.bam, where the _1 is the submitter's sample name. Reading it as a mate made a single BAM report as half a pair. A BAM carries both mates interleaved by construction, so a mate number on one is not unreliable — it is meaningless.

Completeness the listing does not admit to

A fastq set can be incomplete in a way only the archive's other metadata reveals, and this is not a rare corner: 39.2% of a random 3,000-run sample of the reprocessed corpus. ENA reports library_layout=PAIRED and publishes one fastq, and SRA holds a median of 1.40x the bases ENA serves — the missing ~28 bp read being the one that carries the 10x cell barcode.

A pipeline taking the convenient format downloads cleanly, checksums correctly, and produces a matrix in which every read is unassignable to a cell. The file layer ranks that set below every other format, so a complete source wins whenever any route offers one — in practice the .sra object through SDL — and says which fact drove the choice. It does not drop the set: when nothing better is offered it is still returned, labelled every offer is incomplete, because an empty answer would read as "this run has no files".

See ENA publishes an unpaired fastq for a paired library.

And the reads say so

The listing's evidence is the archive's own word: a declared layout and a file count. The reads are the other evidence. fetch files --inspect reads the first 100,000 reads of each offer, and refuses an offer whose reads prove it unusable, such as a lone cDNA read with no barcode read anywhere in it. Only positive evidence refuses an offer. See reading the reads.

1,566 of 1,566. Every link in the recommended set for a seeded 1,000-run draw from reprocessed-srr returned HTTP 200 and served exactly the byte count the archive published (2026-09-16).
Runs asked about 1,000
Runs with a recommended set 999
Links verified 1,566
HTTP 200 1,566 (100%)
Served size matches published size 1,566 (100%)

Which routes the recommendations actually came from:

Source Links Format
run->file:ena_fastq 1,082 fastq
run->file:sdl 448 .sra
run->file:ena_submitted 36 fastq

Verification is a HEAD, so this is a claim about resolvability and size, not about content. Regenerate it with:

uv run python scripts/verify_file_links.py 1000

The two claims that must not be confused

The links are reliable. That says nothing about whether the data behind them is complete — and for 39.2% of runs, ENA's fastq is not.

Nothing in a link check could catch the missing barcode read: those files resolve, their published md5 is correct, and their published size matches what is served — because the file ENA published really is that size. It is simply half the run. A verification layer that only checked links would have called ENA's fastq perfectly healthy.

What keeps those files out of the recommended set is not this verification but the completeness rule above, which reads library_layout. The export shows it working (data/surveys/file-links/sample_1000@reprocessed-srr.csv.gz, rows per run):

Recommended set Runs
ENA fastq, both mates 538
SDL .sra object, one file 442
SDL, three files 2
ENA submitted fastq, three or four files 11
ENA fastq, a single file 6

442 runs went to SDL, and only 6 of the 999 recommended sets are a lone ENA fastq (the export does not record their declared layout, so whether those 6 are single-end libraries or offers with no alternative is not established). That is why the links come to 1.57 per run: a mix of fastq pairs and single .sra objects, not a corpus of half pairs.

Using it

fetch files SRR25056225                 # the recommended set, as TSV with md5 and size
fetch files GSE236084 --explain         # every route's offer and why one won
fetch files E-MTAB-8060 --all           # emit the alternatives too
fetch files GSE109816 --limit 5         # 880 runs at four requests each; cap it