Silent Failures

No errors. Worse answers.

Silent Failures is a benchmark that breaks RAG retrieval on purpose, then tests which label-free signals notice before users do.

Index healthy3,633 docs0 errors323 / 323 queries answered
Query
  1. 1Bronchiolitis obliterans organizing pneumonia (BOOP) after thoracic radiotherapy for breas0.614relevant
  2. 2Bronchiolitis obliterans and consumer exposure to butter-flavored microwave popcorn: a cas0.530relevant
  3. 3Pyrogallol, an active compound from the medicinal plant Emblica officinalis, regulates exp0.474not relevant
  4. 4Lipoid pneumonia presenting as non resolving community acquired pneumonia: a case report0.473not relevant
  5. 5Surgical management of bisphosphonate-related osteonecrosis of the jaw in oncologic patien0.438not relevant
nDCG@10, all 323 queries (needs labels)0.292
Mean top-1 score (label-free)0.523
Re-embed check, 200 sampled docs0 mismatched

Real NFCorpus numbers from the benchmark’s F1 injector: part of the index re-embedded with all-MiniLM-L6-v2 while queries stay on multi-qa-MiniLM-L6-cos-v1.

How a check runs

Window 7sampled
Queries
100 of 323
Reference
separate 100

Sample a window

100 NFCorpus test queries, with a separate 100 as the clean reference.

Window 7damaged
Failure
F1, 30%
Swapped
1,090 of 3,633

Break the index

Inject one failure at a set severity, here F1 at 30%.

Window 7scored
Re-embed
58 / 200 off
Top-1 mean
0.505

Score from logs

Detectors see queries, top-10 results and the index. Never labels.

Window 7harmful
nDCG@10 drop
0.032
p-value
0.001

Grade with labels

A paired bootstrap on nDCG@10 decides whether the damage was real.

Same text, new vectors

Each line is one NFCorpus document, from where the index model put it to where all-MiniLM-L6-v2 puts the same text. Both models output 384 dimensions, so nothing complains when they are mixed, yet a document’s two vectors share a cosine of only 0.61 on average.

All 3,633 documents on the clean index’s first two principal components; lines show 260 of them.

Five ways to break it

Each failure runs at 10% to 90% of the corpus or query traffic. Harmless changes run alongside to measure false alarms.

  • F1Partial embedding upgrade, two models’ vectors in one index
  • F2Stale documents, outdated content next to its replacement
  • F3Chunking change, part of the corpus re-chunked
  • F4Query shift, traffic moves to topics the corpus lacks
  • F5Config mismatch, index and queries encoded differently

Harmless changes

Reordering index rows and mixing 256- and 512-token truncation are the candidates. The final set is chosen in issue #9.

Datasets

NFCorpus (3,633 medical documents) for all tuning. SciFact (5,183 abstracts) and FiQA (57,638 forum posts) are held out until the final run.

What counts as harm

A window is harmful when its nDCG@10 drop is significant under a one-sided paired bootstrap at α = 0.05. Detectors never see this label.

Detectors

Each one sees only what a production system logs.

DetectorSignalStatus
Re-embedding checkRe-encodes a sample of documents and compares them with the stored vectorsBenchmarked
KS on top-1 scoreTwo-sample KS test on each query’s best retrieval score against the reference windowBenchmarked
Retrieved-document classifierSeparates retrieved documents from the reference window’sBuilt
Query classifierSame test on query embeddings; F1 leaves queries alone, so it should not fireBuilt
PSIDistribution shift in binned retrieval scoresNov 11
MMDTwo-sample test on query and document embeddingsNov 11
Canary queriesFixed probe queries whose answers are knownNov 18
LLM judgeAn LLM rates the retrieved passages, with cost per checkNov 18

First benchmark run

NFCorpus with F1 at two strengths: the index partly re-embedded with a close relative of the index model (all-MiniLM-L6-v2) or an unrelated one (bge-small-en-v1.5). 3 seeds × 20 windows of 100 queries per severity.

Mean nDCG@10 drop, with the share of windows the bootstrap calls harmful
Severityall-MiniLMbge-small
10%0.009 (55%)0.014 (80%)
50%0.044 (100%)0.080 (100%)
90%0.065 (100%)0.195 (100%)
Detectors: AUROC with 95% CI, and harmful windows caught at 10% / 50% / 90%
DetectorReplacementAUROCCaught
KS on top-1 scoreall-MiniLM0.908 [0.895–0.932]9% / 82% / 100%
KS on top-1 scorebge-small0.880 [0.875–0.897]6% / 85% / 100%
Re-embedding checkall-MiniLM1.000 [1.000–1.000]100% / 100% / 100%
Re-embedding checkbge-small1.000 [1.000–1.000]100% / 100% / 100%

Each detector’s threshold is its highest score on clean windows, so no clean window raises an alarm. The re-embedding check flags every changed vector, harmful or not; the harmless controls still to come will show what that costs.

20% of nDCG@10 gone, zero errors

Mean nDCG@10 over all 323 test queries falls from 0.292 to 0.234 as more of the index is upgraded. At 10% the drop on a 100-query window is only 0.013, and the paired bootstrap already calls it significant (p = 0.025).

Try a query

Type any NFCorpus test query to compare its top results on the clean index and with half the index upgraded.

Try

Loading queries…

All 323 test queries, half the index upgraded

Each bar is one query’s change in nDCG@10, sorted. The white bar is the query above.

Where the project stands

The pipeline, the experiment runner and the baseline are done. A window of 100 queries lets the paired bootstrap catch an nDCG@10 drop of about 0.06 in 80% of windows. Once every injector and detector is in, everything runs once on SciFact and FiQA, and the detector-by-failure heatmap replaces this section.

Clean nDCG@10, multi-qa-MiniLM-L6-cos-v1, 256 tokens
DatasetQueriesnDCG@1095% CI
NFCorpus3230.2920.259–0.325
SciFact3000.5460.496–0.595
FiQA6480.3600.332–0.387
  1. Oct 18Baseline: model, truncation, clean CIsDone
  2. Oct 31Failure injection: F1 done; F2–F5 and negative controls nextIn progress
  3. Nov 11Drift and score-based detectors, tuned on NFCorpusNext
  4. Nov 18Canary queries, re-embedding, LLM judge, cost per checkPlanned
  5. Nov 24Held-out run on SciFact and FiQA, detector heatmapPlanned
  6. Dec 1Benchmark release, monitoring guide, reportPlanned