No errors. Worse answers.
Silent Failures is a benchmark that breaks RAG retrieval on purpose, then tests which label-free signals notice before users do.
- 1Bronchiolitis obliterans organizing pneumonia (BOOP) after thoracic radiotherapy for breas0.614relevant
- 2Bronchiolitis obliterans and consumer exposure to butter-flavored microwave popcorn: a cas0.530relevant
- 3Pyrogallol, an active compound from the medicinal plant Emblica officinalis, regulates exp0.474not relevant
- 4Lipoid pneumonia presenting as non resolving community acquired pneumonia: a case report0.473not relevant
- 5Surgical management of bisphosphonate-related osteonecrosis of the jaw in oncologic patien0.438not relevant
Real NFCorpus numbers from the benchmark’s F1 injector: part of the index re-embedded with all-MiniLM-L6-v2 while queries stay on multi-qa-MiniLM-L6-cos-v1.
How a check runs
- Queries
- 100 of 323
- Reference
- separate 100
Sample a window
100 NFCorpus test queries, with a separate 100 as the clean reference.
- Failure
- F1, 30%
- Swapped
- 1,090 of 3,633
Break the index
Inject one failure at a set severity, here F1 at 30%.
- Re-embed
- 58 / 200 off
- Top-1 mean
- 0.505
Score from logs
Detectors see queries, top-10 results and the index. Never labels.
- nDCG@10 drop
- 0.032
- p-value
- 0.001
Grade with labels
A paired bootstrap on nDCG@10 decides whether the damage was real.
Same text, new vectors
Each line is one NFCorpus document, from where the index model put it to where all-MiniLM-L6-v2 puts the same text. Both models output 384 dimensions, so nothing complains when they are mixed, yet a document’s two vectors share a cosine of only 0.61 on average.
All 3,633 documents on the clean index’s first two principal components; lines show 260 of them.
Five ways to break it
Each failure runs at 10% to 90% of the corpus or query traffic. Harmless changes run alongside to measure false alarms.
- F1Partial embedding upgrade, two models’ vectors in one index
- F2Stale documents, outdated content next to its replacement
- F3Chunking change, part of the corpus re-chunked
- F4Query shift, traffic moves to topics the corpus lacks
- F5Config mismatch, index and queries encoded differently
Harmless changes
Reordering index rows and mixing 256- and 512-token truncation are the candidates. The final set is chosen in issue #9.
Datasets
NFCorpus (3,633 medical documents) for all tuning. SciFact (5,183 abstracts) and FiQA (57,638 forum posts) are held out until the final run.
What counts as harm
A window is harmful when its nDCG@10 drop is significant under a one-sided paired bootstrap at α = 0.05. Detectors never see this label.
Detectors
Each one sees only what a production system logs.
| Detector | Signal | Status |
|---|---|---|
| Re-embedding check | Re-encodes a sample of documents and compares them with the stored vectors | Benchmarked |
| KS on top-1 score | Two-sample KS test on each query’s best retrieval score against the reference window | Benchmarked |
| Retrieved-document classifier | Separates retrieved documents from the reference window’s | Built |
| Query classifier | Same test on query embeddings; F1 leaves queries alone, so it should not fire | Built |
| PSI | Distribution shift in binned retrieval scores | Nov 11 |
| MMD | Two-sample test on query and document embeddings | Nov 11 |
| Canary queries | Fixed probe queries whose answers are known | Nov 18 |
| LLM judge | An LLM rates the retrieved passages, with cost per check | Nov 18 |
First benchmark run
NFCorpus with F1 at two strengths: the index partly re-embedded with a close relative of the index model (all-MiniLM-L6-v2) or an unrelated one (bge-small-en-v1.5). 3 seeds × 20 windows of 100 queries per severity.
| Severity | all-MiniLM | bge-small |
|---|---|---|
| 10% | 0.009 (55%) | 0.014 (80%) |
| 50% | 0.044 (100%) | 0.080 (100%) |
| 90% | 0.065 (100%) | 0.195 (100%) |
| Detector | Replacement | AUROC | Caught |
|---|---|---|---|
| KS on top-1 score | all-MiniLM | 0.908 [0.895–0.932] | 9% / 82% / 100% |
| KS on top-1 score | bge-small | 0.880 [0.875–0.897] | 6% / 85% / 100% |
| Re-embedding check | all-MiniLM | 1.000 [1.000–1.000] | 100% / 100% / 100% |
| Re-embedding check | bge-small | 1.000 [1.000–1.000] | 100% / 100% / 100% |
Each detector’s threshold is its highest score on clean windows, so no clean window raises an alarm. The re-embedding check flags every changed vector, harmful or not; the harmless controls still to come will show what that costs.
20% of nDCG@10 gone, zero errors
Mean nDCG@10 over all 323 test queries falls from 0.292 to 0.234 as more of the index is upgraded. At 10% the drop on a 100-query window is only 0.013, and the paired bootstrap already calls it significant (p = 0.025).
Share of documents upgraded. The y axis starts at 0.2.
Try a query
Type any NFCorpus test query to compare its top results on the clean index and with half the index upgraded.
Try
Loading queries…
Each bar is one query’s change in nDCG@10, sorted. The white bar is the query above.
Where the project stands
The pipeline, the experiment runner and the baseline are done. A window of 100 queries lets the paired bootstrap catch an nDCG@10 drop of about 0.06 in 80% of windows. Once every injector and detector is in, everything runs once on SciFact and FiQA, and the detector-by-failure heatmap replaces this section.
| Dataset | Queries | nDCG@10 | 95% CI |
|---|---|---|---|
| NFCorpus | 323 | 0.292 | 0.259–0.325 |
| SciFact | 300 | 0.546 | 0.496–0.595 |
| FiQA | 648 | 0.360 | 0.332–0.387 |
- Oct 18Baseline: model, truncation, clean CIsDone
- Oct 31Failure injection: F1 done; F2–F5 and negative controls nextIn progress
- Nov 11Drift and score-based detectors, tuned on NFCorpusNext
- Nov 18Canary queries, re-embedding, LLM judge, cost per checkPlanned
- Nov 24Held-out run on SciFact and FiQA, detector heatmapPlanned
- Dec 1Benchmark release, monitoring guide, reportPlanned