Agents Find the File and Miss the Cause, and the Gap Grows on Thin Evidence

:::tip TL;DR DistDebug-Bench asks agents to name the exact code block that explains a failure, instead of asking them to produce a patch that passes tests. On 120 real bugs from 11 distributed systems, the best configuration reaches the right file 83.9% of the time and the causal block only 67.7% of the time. Grade debugging agents at the block level, run each bug more than once, and route reports with no stack trace and no named method to a human first. :::

What this benchmark tests, and why it is different

Most coding benchmarks score a patch. A patch can pass a test suite while suppressing a symptom rather than fixing the cause, and a test suite can be missing entirely for the kind of failure that matters in production. DistDebug-Bench, from researchers at the University of Michigan and submitted to arXiv on 2 November 2026, scores diagnosis directly.

Each of the 120 instances gives the agent an incident report, an evidence bundle such as logs or a stack trace, and the pre-fix repository snapshot. The agent must return one or more source blocks that best explain the failure. Human annotators labelled 218 gold root-cause blocks, and the benchmark covers 11 widely used distributed systems, 5 programming languages, and 10 failure patterns.

The dataset is built to be hard in the ways production debugging is hard. In the paper's own description, 45% of bugs need multiple blocks to express the root cause, 23% span multiple files, 69% of reports do not mention the gold method, and 63% contain no usable internal stack trace. That last pair is the point of the benchmark. A debugging agent that only works when the report already points at the answer is not doing the job engineers actually need.

This is a preprint, not a peer reviewed paper, and the sample is 120 bugs. Every headline number below carries a confidence interval roughly 7 to 8 points wide. Read the rankings as directional, and read the gaps between metrics as the finding that survives that uncertainty.

Finding 1: Reaching the file is not diagnosing the bug

The paper's primary metric is block coverage: how much of the gold causal block set the agent's diagnosis covers. File coverage, function coverage, and line coverage are reported alongside it, and the spread between them is the story.

Model (Mini-SWE-Agent)	File coverage	Block coverage (95% CI)	Block precision	Flaky bugs
GPT-6 Astra	83.9%	67.7% [60.4, 74.9]	42.7%	15 (12%)
Claude Opus 5.5	82.0%	66.4% [59.2, 73.6]	45.5%	17 (14%)
GPT-5.5	84.0%	65.1% [57.8, 72.3]	34.7%	20 (17%)
GPT-5.4	79.8%	63.1% [56.0, 70.2]	39.5%	21 (18%)
GPT-5.2-Codex	75.9%	56.4% [48.5, 64.2]	43.1%	18 (15%)
Gemini 3.5 Flash	65.9%	47.5% [39.7, 55.2]	51.1%	22 (18%)
o4-mini	63.1%	39.3% [32.1, 46.8]	34.7%	30 (25%)

GPT-6 Astra has the highest block coverage at 67.7%, but its lead over Claude Opus 5.5 is 1.3 points, well inside the confidence intervals. Do not build a model choice on that gap. The durable result is that every configuration sits 15 to 25 points higher on file coverage than on block coverage. The agents reliably get to the right neighbourhood and then stop short of the causal lines.

Block precision sharpens the point. Precision is the share of predicted blocks that are actually gold blocks. GPT-6 Astra names 3.3 blocks per diagnosis on average and only 42.7% of them are gold. Gemini 3.5 Flash is the most precise at 51.1%, but its coverage is only 47.5%. There is no configuration here that is both precise and complete. A diagnosis that names the right block plus two wrong ones still sends an engineer down two dead ends.

Finding 2: Evidence quality sets the ceiling

When the report names the exact method, agents do well. The best configuration reaches 95.8% block coverage on local or adjacent method-named bugs and 94.8% on no-stack method-named bugs. When the report is opaque, meaning it names no gold method, file, or class identifier, the same best configuration falls to 65.0% on the 42 no-stack opaque bugs and 40.5% on the seven cross-component opaque bugs.

That 40.5% figure rests on seven bugs, so treat it as exploratory, not as a precise estimate. The direction is confirmed by the fixed baseline: Mini-SWE-Agent with GPT-5.2-Codex goes from 82.8% on no-stack method-named bugs to 44.7% on no-stack opaque bugs and 14.3% on cross-component opaque bugs. Newer models raise the absolute numbers without removing the gap. GPT-6 Astra improves no-stack opaque coverage to 65.0%, which is still 29.8 points below its 94.8% on no-stack method-named bugs.

Opaque reports are not a corner case. They account for 50 of the 120 bugs, including 42 without a stack trace. Any deployment plan that quotes the aggregate 67.7% figure for a queue full of vague incident reports is quoting the wrong number. The relevant number for that queue is closer to 65% at best, and far lower for cross-component failures.

Finding 3: Repeat runs disagree, most on the hard bugs

The paper defines a flaky bug as one where block coverage spreads by more than 0.2 across repeated attempts. GPT-6 Astra is flaky on 15 of 120 bugs (12%). The range across the sweep runs from 12% up to 29% for HY3, with Kimi K2.6 and DeepSeek V4 Flash each at 32 bugs (27%).

Instability concentrates exactly where you would least want it. For Mini-SWE-Agent with GPT-5.5, 11 of the 42 no-stack opaque bugs are flaky (26%), with a mean within-bug standard deviation of 9.2 points, against 2 of 16 no-stack method-named bugs (13%) and 3.7 points. The paper reports the same contrast for GPT-5.2-Codex: 11 of 42 opaque bugs flaky (26%) against 1 of 16 method-named bugs (6%).

A single run therefore cannot certify a debugging agent. On roughly one bug in eight for the best model, and one in four for the worst, a second run gives a materially different diagnosis. Any evaluation that reports one pass per bug, or any production gate that trusts one diagnosis per incident, is hiding this variance instead of measuring it.

Finding 4: The harness trades recall for precision

Holding the model fixed at GPT-5.2-Codex and swapping only the harness changes the shape of the diagnosis. Mini-SWE-Agent gives 56.4% block coverage and 43.1% precision, OpenCode gives 50.0% and 47.7%, and Codex CLI gives 46.5% and 49.3%. Blocks per diagnosis move with the tradeoff: 2.3 for Mini-SWE-Agent, 1.8 for OpenCode, 1.6 for Codex CLI.

That is a real choice, not a ranking. Where a false positive is expensive, because each wrong block costs an engineer an investigation, the precision leaning harness wastes less time. Where missing the cause is expensive, because the incident stays open, the recall leaning harness is the better bet. What no harness fixes is the evidence gap: every harness degrades sharply when the report lacks direct causal code references.

Finding 5: A stronger model can be a broader, sloppier diagnostician

GPT-5.5 improves aggregate block coverage from 56.4% to 65.1% over GPT-5.2-Codex on the same harness, and improves no-stack opaque bugs by 15.4 points. At the same time it names 3.6 blocks per diagnosis instead of 2.3, and its block precision falls from 43.1% to 34.7%. It recovers more causal code on the hard cases and bundles that gain with more irrelevant code.

Token spend tells a similar story about scaling without control. GPT-6 Astra averages 132K tokens per run and 9.0 tool calls. GPT-5.5 averages 468K tokens and 21.9 calls. MiMo v2.5 Pro reaches 58.5% block coverage at 2.6M tokens per run, and Kimi K2.6 reaches 58.3% at 2.8M tokens. More tokens buy exploration, not reliability. The paper's own summary is blunt: model scaling can improve exploration, but it is not a substitute for reliable diagnosis control.

Finding 6: Tunnel vision is the named failure mode

The paper calls the central failure mode tunnel vision. Agents commit early to a plausible local explanation and rarely recover when the initial search scope misses the causal region. In the trace proxy, only 21.0% of parseable runs showed late recovery, defined as a final diagnosis whose files do not appear among the first two inspected source files. Among successful runs, only 11.25% were late recoveries.

Two ZooKeeper traces make the mechanism concrete. In one, successful and failed runs both read the same node log early, but successful runs searched for "Exception" while failed runs searched for specific line numbers or election related logs, and missed the BufferUnderflowException that mattered. In another, the report emphasised quorum behaviour, agents narrowed early to the server quorum subdirectory, and keyword search then could no longer surface the correct class in the broader server path. A small difference in the first search term, or the first directory entered, sends the whole run into a different subsystem, and the agent then deepens a coherent but wrong explanation instead of reopening the search.

What this means for data and AI practitioners

If you run coding agents against production incidents, or you are asked to approve one, DistDebug-Bench gives you a checklist that survives its own confidence intervals:

1. Grade diagnoses at the block or function level, never at the file level. A gate that passes on file coverage will pass diagnoses that miss the cause.
2. Run each bug at least three times before trusting a verdict. The paper averages the first three valid scored attempts per bug, and the flaky rates show why one run is not enough.
3. Route by evidence quality. Incidents with no stack trace and no named method should escalate to a human first, or at minimum carry a warning that the agent's expected coverage is far below its aggregate score.
4. Pick the harness for your cost structure. Precision leaning where false leads are expensive, recall leaning where a missed cause is expensive, and measure both numbers, not just one.
5. Force a hypothesis re-check. If the agent has not converged after its first two files, require it to state a competing hypothesis and search outside the current scope before it is allowed to conclude.
6. Log the first search terms and directories, not just the final diagnosis. The ZooKeeper traces show that the failure is often decided by the first query, and you cannot fix what you do not log.

Against

The sample is 120 bugs, so the top of the leaderboard is a statistical tie: GPT-6 Astra at 67.7%, Opus 5.5 at 66.4%, and GPT-5.5 at 65.1% all sit inside overlapping intervals. The benchmark is also a preprint, and its gold labels, however carefully cross-checked by three annotators, define "cause" as the pre-fix block set. A diagnosis that points at a different valid intervention point can score zero, so the metric may understate diagnostic competence in some cases. Finally, all 120 bugs are historical public bugs, so pretraining exposure cannot be ruled out entirely, even though the paper's closed-book probes suggest repository access matters far more than memorised issue identifiers.

In favour

Three design choices make this benchmark harder to dismiss than a typical leaderboard. First, the leakage guard is tested, not just asserted: flagged attempts for GPT-5.4 fell from 102 of 360 to 4 of 360 after future commits were removed, and revealing an issue identifier in closed-book probes gave no advantage. Second, the same 120 bugs are held fixed across 13 model configurations plus the harness comparison, so model effects and harness effects are separated instead of confounded. Third, the evidence breakdown is published per cell with sample counts, including the seven-bug cell flagged as exploratory, rather than buried in an aggregate.

What would make me wrong

A larger replication, say 500 or more bugs with the same protocol, that closes the file to block gap to under 5 points would weaken the central claim. Peer review that overturns the gold labels on a substantial share of bugs would do the same. If reruns on a team's own incident queue show block coverage at or above file coverage, or flaky rates near zero on opaque reports, then this benchmark's failure modes do not generalise to that codebase, and the team's own numbers should replace these. And if a harness change, such as automatic context augmentation with related symbols and call paths, lifts cross-component opaque coverage close to method-named coverage, the tunnel vision diagnosis needs revising: the problem was tooling all along, not reasoning.

Sources

• DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? (arXiv abstract)
• DistDebug-Bench full paper text (arXiv HTML)
• DistDebug-Bench benchmark and evaluator (GitHub)
• DistDebug-Bench traces and leaderboard

What is your team's rule for trusting an agent's diagnosis: one run, three runs, or a human in the loop until the evidence is strong? And which number do you actually gate on, file, function, or block?

Related on Everyday Data Science:

• ThinkingBox: agent reliability graded on database state
• Agent context pruning: less history, better accuracy
• AgentPhishing: adaptive phishing against email agents

Sources: arXiv:2611.03523