Skip to content

ENA lists a fastq its file server does not serve

Archive: EBI ENA Affected route: run->file:ena_fastq Status: detected, measured, and caught by fetch files --verify; ready to report upstream Measured: 2026-10-23, every ENA fastq link of a seeded 3,000-run draw of the reprocessed corpus

The number

filereport lists each fastq with a size and an md5. The file server was asked about every one of the 4,466 links ENA lists for the 2,844 runs of sample:3000@reprocessed-srr that list any. It was asked as fetch files --verify asks: a HEAD with redirects followed.

What the server did Links
served a file of the published size 4,381
answered 403, then 200 when asked again one at a time 80
answered 403 twice 1
timed out 1
served an empty directory at the file's path 2
served a file of 4,096 bytes where 7,197,985,953 are published, then the whole file 1

Two listed files are not there to download, 2 of 4,466 links. Each looks complete to a listing: a name, a size and an md5. They were still directories every time they were asked on 2026-10-23.

  • SRR11040464 _1.fastq.gz is listed at 583,713,371 bytes, and SRR27782870 _1.fastq.gz at 606,656,529. At each path the server answers 301 to the same path with a trailing slash, and serves an empty directory listing. The _2 file beside each is real.
  • ERR15320482 _2.fastq.gz is listed at 7,197,985,953 bytes. The census read 4,096 bytes for it, the start of a gzip stream, and at 12:54 UTC the directory index still listed it as 4.0K. By 13:10 UTC the server answered the full 7,197,985,953 bytes, with the same Last-Modified. Two backends of one server disagreed, briefly. This is not counted as a missing file, and it cannot be tested for.

Reproducer

curl -sI https://ftp.sra.ebi.ac.uk/vol1/fastq/SRR110/064/SRR11040464/SRR11040464_1.fastq.gz | head -1
# HTTP/1.1 301 Moved Permanently   -> .../SRR11040464_1.fastq.gz/  (an empty directory)

curl -sI https://ftp.sra.ebi.ac.uk/vol1/fastq/SRR277/070/SRR27782870/SRR27782870_1.fastq.gz | head -1
# HTTP/1.1 301 Moved Permanently   -> .../SRR27782870_1.fastq.gz/  (an empty directory)

Checked on 2026-10-23.

What it is not

  • Not the 403s. 81 links answered 403 under the census's concurrency. 80 answered 200 when asked again one at a time at one a second, which makes them throttling, not missing files. SurveyClient now retries a 403 from ftp.sra.ebi.ac.uk (survey.client.THROTTLES_WITH_403), and only from there. The content survey's 403s on ERR4065012 and SRR34482399 (2026-10-22) were of this kind: both answered 200 the next day.
  • Not the single-fastq defect. That one lists one real file where two reads exist. Here a listed file is not real.

Why it matters

  • A directory answers 200. A client that follows the redirect, as every client must for ENA's ftp.sra.ebi.ac.uk links, gets a 200 and an HTML index.
  • A stub served for a moment decompresses for its first 4 KB and then ends. A download that landed in that window would be caught only by its md5.
  • The fit rule is protected. In the content survey SRR11040464's set was refused on its other file alone, until fit-5 stopped any refusal while a listed file cannot be read.

Detection and workaround

fetch files --verify flags both directories: a redirect to a path ending / is served_directory, and never resolves (filesets.Verdict). A stub fails size_matches while it is being served.

The other offers of the same run are unaffected: NCBI's SRA objects of SRR11040464 read fit.

Affected accessions

uv run python scripts/ena_listing_census.py \
  --census data/surveys/sample_3000@reprocessed-srr/run-to-file_ena_fastq.csv.gz \
  --out data/benchmarks/ena-listing/<date>

The 2026-10-23 census is data/benchmarks/ena-listing/2026-10-23/links.csv.

Regression test

tests/pathologies/test_ena_serves_a_listed_fastq_wrongly.py asserts that SRR11040464's and SRR27782870's _1.fastq.gz are still served as directories, and that verify never calls either a file.