Why “Best AI for Debugging Code” Rankings Mislead—and How to Evaluate Debug AI That Actually Fixes Bugs

If you want an AI that actually fixes your bugs, leaderboard scores won’t tell you what you need to know.

Most public rankings of "best AI for debugging code" reduce a complex systems problem to a single static number. They often measure an agent on pre-canned tasks with deterministic environments and dont’s reflect what happens when real-world bugs meet drift-prone environments, flaky tests, and incomplete runtime context.

Debugging is not just about proposing a patch that makes a test green. It’s about:

• Reproducing the failure reliably across machines and over time.
• Localizing the fault using dynamic signals: traces, logs, coverage, stack frames, variable state, and sometimes system calls.
• Proposing a minimal, correct fix that doesn’t introduce regressions.
• Verifying the fix in a sandbox that approximates production.
• Managing non-code factors: dependency versions, ephemeral network behavior, test flakiness, and platform-specific edge cases.

In this article, we’ll unpack why typical rankings mislead and propose a practical evaluation harness for debugging-capable AI systems. We’ll lean into details: trace capture strategies, sandbox isolation, failure taxonomies, scoring methods, and concrete code snippets to bootstrap your own evaluation.

The Problem with Debugging Leaderboards

Leaderboards are attractive because they’re simple. But for debugging, simplicity hides critical variables.

1. Static benchmarks ignore environmental drift.

• Real projects pull dependencies from npm, PyPI, crates.io, Maven Central, apt repositories, or OS mirrors. Over weeks and months, upstreams release new versions, deprecate APIs, change transitive dependencies, or revoke packages.
• OS-level changes (glibc, kernel, libc++, SSL libraries) alter runtime behavior.
• Cloud or system constraints change (rate limits, new TLS defaults, DNS quirks), and tests that touch the network can start failing or flaking.

A benchmark snapshot from last quarter can’t tell you whether a debugging agent is robust to environment drift next quarter.

2. Flaky tests confound pass/fail.

• At Google, flaky tests were significant enough to warrant public guidance on mitigation Google Testing Blog: Flaky Tests.
• Without multi-run validation and flake classification, a "pass" might be luck, not a real fix; a "fail" might be a flaky test masking a correct fix.

3. Runtime context is under-specified.

• Most benchmarks give an agent the source code and maybe the failing test. Real debugging benefits from:
  • Stack traces and frames
  • Error logs with timestamps and correlation IDs
  • Coverage deltas for failing tests
  • Reproduction steps, environment variables, and seeded randomness
  • System-level traces when issues are IO / concurrency related

Without runtime context, the agent "fix" devolves into guessing.

4. Test-passing is not equivalent to correctness.

• Patches can overfit tests (e.g., by hardcoding outputs, weakening invariants, or skipping execution paths) while breaking untested behaviors.
• In program repair literature, semantic correctness is notoriously hard to guarantee. Differential testing and metamorphic relations help, but most leaderboards rarely incorporate these.

5. Debugging is a systems problem; static corpora aren’t enough.

• Benchmarks like Defects4J (Java), Bugs.jar, ManyBugs (C), QuixBugs, Bears, and BugsInPy are incredibly valuable—but they’re still snapshots. If you don’t also model environment changes, CI shells, and multi-OS/multi-arch behaviors, you’ll overestimate real-world performance.

The net effect: leaderboards can rank models by how well they pass a one-time test suite, not by how reliably they fix real bugs in moving systems.

What Actually Matters for Debugging AI

If your goal is an AI that debugs real repositories, the evaluation has to mirror the constraints of real debugging. At minimum, you need:

• Reproducibility:
  • Hermetic environments (containers/VMs), locked dependencies, and seeded randomness.
  • Version control pinning (commit hashes) so you can recreate an exact failure.
• Runtime visibility:
  • Execution traces, logs, coverage, heap snapshots (where applicable), and system call/network traces for IO-heavy or concurrency issues.
• Flakiness detection:
  • Multi-run sampling and statistical thresholds to separate flaky outcomes from deterministic behavior.
• Fix quality metrics beyond a “green test”:
  • Regression rate (across a held-out or fuzzed test set)
  • Patch minimality/complexity (diff size, cyclomatic change)
  • Cross-environment verification (OS, Python/Node/JDK versions)
• Cost and operational metrics:
  • Time to reproduce
  • Time to localize and propose a patch
  • Tool invocations (debugger, test runs, package installers)
  • Token/compute cost for LLM-based agents

A Practical Evaluation Harness for Debugging AI

We propose an evaluation harness with four pillars:

1. A failure taxonomy to classify and stratify bugs.
2. A sandboxed reproduction runner with hermetic builds.
3. Trace capture that surfaces runtime signals to the AI agent.
4. A scoring protocol that handles flakiness and measures patch quality, not just pass/fail.

1) Failure Taxonomy

A taxonomy ensures your benchmark isn’t dominated by one failure mode (e.g., simple syntax or import errors). It also lets you diagnose where your agent excels or struggles.

Suggested top-level classes:

• Build/Compile-time failures
  • Missing symbol, version mismatch, incompatible language level
  • Toolchain/config: Babel/TypeScript/JDK/Gradle/Maven/npm/pip
• Import/Linking errors
  • Module not found, dynamic linker errors, ABI mismatch
• Runtime exceptions
  • Null/None/Attribute errors, type errors, index/key errors, assertion errors
• Logic/specification failures
  • Off-by-one, boundary conditions, state machine errors
• Concurrency/timeouts/races
  • Deadlocks, livelocks, timeouts under load
• IO and environment
  • Network failures, DNS issues, path/permission errors, locale/encoding
• Numerical/precision/stability
  • Floating point drift, tolerance failures, unstable algorithms
• API drift and deprecation
  • Upstream libraries changed behavior or signature
• Test flakiness
  • Nondeterministic tests due to randomness/time/scheduling

Each bug instance should include an initial classification (heuristic + human-reviewed samples). Over time, your harness can learn to auto-classify from logs and traces.

2) Sandboxed Repro Runner

A repro must be deterministic and isolated:

• Containers/VMs: Use Docker or a microVM (e.g., Firecracker) with pinned base images.
• Hermetic builds: Bazel or Nix for maximal determinism where possible; otherwise lockfiles (pip-tools, npm package-lock.json, Poetry/Pipenv, Cargo.lock).
• Network policy:
  • Default deny; explicitly allow only known build-time fetches (to avoid hidden network dependencies and leakage).
  • Record and hash downloaded artifacts for cache/replay.
• Seed control:
  • Force seeds for random, numpy, torch, Java Random, Go math/rand, etc.
  • Mock time where feasible to remove clock nondeterminism.
• Resource envelopes and caps:
  • Timeouts per phase (install/build/test), CPU/mem limits to expose resource sensitivity.
• Multi-OS/arch matrix (optional but valuable):
  • Linux (glibc vs. musl), macOS runners, Windows containers (if relevant), and at least one alternate CPU arch (ARM64).

A simple harness can orchestrate via Docker from Python. Example project config (YAML):

yaml
project: example-repo
repo: https://github.com/acme/example-repo.git
commit: 1f2a3b4c5d
language: python
image: python:3.11-slim
setup:
  - apt: [build-essential, git]
  - pip: [-r requirements.txt]
  - env:
      PYTHONHASHSEED: '0'
      OMP_NUM_THREADS: '1'
      NUMEXPR_MAX_THREADS: '1'
      PYTHONWARNINGS: 'default'
  - network: false
  - seeds:
      python: 0
      numpy: 0
      random: 0
      torch: 0
test:
  command: pytest -q
  retries: 5
  timeout_seconds: 600
trace:
  coverage: true
  stack_traces: true
  syscall_trace: file,network
  record_logs: true
classification_hints:
  - 'AttributeError'
  - 'ModuleNotFoundError'
  - 'Timeout'

3) Trace Capture and Context Packaging

Provide the AI with actionable runtime context, not just code. Useful techniques:

• Stack traces and error logs: capture with timestamps, thread IDs, and request IDs when relevant.
• Coverage deltas: which lines/functions are executed by failing vs. passing tests.
• Function-level tracing:
  • Python: sys.settrace, coverage.py, faulthandler, traceback collections from pytest.
  • JVM: JVMTI/Java Flight Recorder; test frameworks already emit stack traces.
  • Node.js: Inspector protocol, source maps.
• System call/network tracing:
  • Linux: strace -f -e trace=file,network -o trace.log ...
  • eBPF-based tools to sample events with lower overhead (eBPF).
• OpenTelemetry traces for distributed components: tie failing transaction spans to code locations (OpenTelemetry).

Package these artifacts in a compact bundle for the agent:

• Repo snapshot and diff from HEAD
• Failing tests and logs
• Coverage/trace summaries (top N hot functions/lines)
• Failure classification (tentative)
• Environment manifest (OS, toolchain, dependency lockfiles)

Avoid flooding the agent with raw megabytes; summarize first, allow on-demand retrieval for details.

Minimal Python example: instrumented test run inside a container

python
import os, subprocess, json, tempfile, shutil

def run_pytest_with_traces(workdir, timeout=600):
    env = os.environ.copy()
    env['PYTHONHASHSEED'] = '0'
    cmd = [
        'bash', '-lc',
        # coverage + pytest; write logs and strace
        'coverage run -m pytest -q 2>stderr.log | tee stdout.log; '
        'coverage xml -o coverage.xml; '
        'python - <<\PY\n'
        'import sys, traceback;\n'
        'try:\n'
        '    import pytest;\n'
        'except Exception as e:\n'
        '    print(e)\n'
        'PY'
    ]
    # Optional: strace around the test command
    # Example: strace -f -e trace=file,network -o strace.log pytest -q
    proc = subprocess.run(cmd, cwd=workdir, env=env, timeout=timeout, text=True)
    with open(os.path.join(workdir, 'stdout.log'), 'r', errors='ignore') as f:
        out = f.read()
    with open(os.path.join(workdir, 'stderr.log'), 'r', errors='ignore') as f:
        err = f.read()
    # Exit status still useful even if logs exist
    return {
        'exit_code': proc.returncode,
        'stdout': out[-10000:],
        'stderr': err[-10000:],
        'coverage_xml': os.path.exists(os.path.join(workdir, 'coverage.xml'))
    }

4) Scoring Protocol: More Than Green Tests

We recommend a multi-axis scoring function that captures correctness, robustness, and cost.

• Primary outcome metrics:
  • Fix success rate: fraction of bug instances where tests pass deterministically after patch.
  • Robustness: cross-run stability (e.g., pass in ≥4/5 runs) and cross-environment pass rate.
  • Regression rate: failures in held-out tests or fuzzing harnesses introduced by the patch.
• Patch quality:
  • Diff size (lines added/removed), AST-level change size, and complexity delta.
  • Minimality: Is the fix localized or sprawling?
  • Maintainability: heuristic lint/format/complexity checks.
• Efficiency:
  • Time to first passing patch.
  • Tool invocations and runtime cost.
  • Token usage for LLMs.
• Safety/compliance:
  • No secret exfiltration attempts (e.g., refusing to send container env to external URLs).
  • No build scripts altered to skip tests.

Use a composite score (weighted) and report the components separately. Resist the urge to collapse to a single number in dashboards; trend each dimension over time.

Building the Dataset: Sources and Practices

Good debugging evaluations mix curated benchmarks with live-repo scenarios.

• Curated benchmarks (frozen snapshots):
  • Defects4J (Java) — github.com/rjust/defects4j
  • Bugs.jar (Java) — github.com/bugs-dot-jar/bugs-dot-jar
  • ManyBugs (C) — repairbenchmarks.cs.umass.edu/ManyBugs
  • QuixBugs (multi-language) — github.com/jkoppel/QuixBugs
  • Bears (Java) — github.com/bears-bugs/bears-benchmark
  • BugsInPy (Python) — github.com/soarsmu/BugsInPy
  • BugZoo (infra for program repair) — github.com/squaresLab/BugZoo
• Issue-driven, real-world tasks:
  • SWE-bench (resolving GitHub issues with real repos) — github.com/princeton-nlp/SWE-bench
  • Consider a custom dataset with commits that introduce a failing test, or a minimal reproduction issue, and a later commit that fixes it.

When assembling your set:

• Prefer bugs with reproducible failures and known fixes.
• Include API drift cases (e.g., numpy/pandas/scikit-learn version changes) and OS-specific differences.
• Encode constraints in metadata: required OS, CPU arch, network policy, memory limits.
• Provide a Dockerfile or Nix/Bazel rules to normalize environments.

Harness Architecture: An End-to-End Flow

1. Select bug instance:
   • Fetch repo, checkout commit with failing state, lock dependencies.
2. Build and test in sandbox:
   • Run tests N times, collect traces and classify failure type; detect flakiness.
3. Prepare agent context:
   • Summarize stack traces, errors, coverage, and top changed files.
   • Provide environment manifest and reproduction steps.
4. Agent generates patch:
   • May request additional traces (on-demand), run focused tests, or inspect files.
5. Validate patch:
   • Apply diff, rebuild, run full tests M×N times across environment matrix.
6. Score patch:
   • Compute outcome, robustness, regression risk, and patch quality metrics.
7. Log and archive:
   • Store all artifacts: logs, diffs, metrics, and environment digests for later audit.

A Minimal Orchestrator (Python + Docker)

Below is a sketch to help you get started. In production you’ll want richer error handling, caching, and concurrency.

python
import json, os, subprocess, tempfile, shutil, time, uuid
from pathlib import Path

DOCKER_IMAGE = 'python:3.11-slim'

def sh(cmd, cwd=None, timeout=600, env=None):
    return subprocess.run(cmd, cwd=cwd, shell=True, text=True, timeout=timeout, env=env,
                          stdout=subprocess.PIPE, stderr=subprocess.PIPE)

def docker_run(workdir, command, network=False, timeout=900):
    net_flag = '--network none' if not network else ''
    vol = f'-v {workdir}:/w -w /w'
    cmd = f'docker run --rm {net_flag} {vol} {DOCKER_IMAGE} bash -lc "{command}"'
    return sh(cmd, timeout=timeout)

def setup_env(repo_url, commit):
    tmp = Path(tempfile.mkdtemp(prefix='dbg-'))
    r = sh(f'git clone {repo_url} .', cwd=tmp.as_posix())
    if r.returncode != 0:
        raise RuntimeError(r.stderr)
    r = sh(f'git checkout {commit}', cwd=tmp.as_posix())
    if r.returncode != 0:
        raise RuntimeError(r.stderr)
    return tmp

def install_deps(workdir):
    # Example for Python; parameterize per language
    docker_run(workdir, 'apt-get update && apt-get install -y build-essential git && rm -rf /var/lib/apt/lists/*')
    if Path(workdir, 'requirements.txt').exists():
        docker_run(workdir, 'python -m venv .venv && . .venv/bin/activate && pip install -U pip setuptools wheel && pip install -r requirements.txt', network=True)

def classify_failure(log):
    s = log.lower()
    if 'modulenotfounderror' in s or 'importerror' in s:
        return 'import/link'
    if 'attributeerror' in s or 'typeerror' in s or 'keyerror' in s or 'indexerror' in s:
        return 'runtime-exception'
    if 'assert' in s and ('failed' in s or 'error' in s):
        return 'logic/spec'
    if 'timeout' in s:
        return 'timeout'
    return 'unknown'

def run_tests(workdir, retries=3):
    res = []
    for i in range(retries):
        cmd = 'set -e; . .venv/bin/activate 2>/dev/null || true; ' \
              'PYTHONHASHSEED=0 pytest -q 2>stderr.log | tee stdout.log'
        r = docker_run(workdir, cmd, network=False, timeout=900)
        out = Path(workdir, 'stdout.log').read_text(errors='ignore') if Path(workdir, 'stdout.log').exists() else r.stdout
        err = Path(workdir, 'stderr.log').read_text(errors='ignore') if Path(workdir, 'stderr.log').exists() else r.stderr
        res.append({'code': r.returncode, 'stdout': out[-10000:], 'stderr': err[-10000:]})
    return res

def stable_pass(test_runs, threshold=0.8):
    passes = sum(1 for r in test_runs if r['code'] == 0)
    return (passes / len(test_runs)) >= threshold

def apply_patch(workdir, patch_text):
    patch_file = Path(workdir, 'agent.patch')
    patch_file.write_text(patch_text)
    r = sh('git apply --whitespace=fix agent.patch', cwd=workdir)
    return r.returncode == 0

# Example loop
if __name__ == '__main__':
    repo = 'https://github.com/acme/example-repo.git'
    commit = '1f2a3b4c5d'
    workdir = setup_env(repo, commit)
    install_deps(workdir.as_posix())
    baseline = run_tests(workdir.as_posix(), retries=5)
    base_ok = stable_pass(baseline)
    print('Baseline stable pass?', base_ok)
    if base_ok:
        print('No failure to fix; pick another instance.')
    else:
        # Classify failure and prepare context for the agent
        merged_logs = '\n\n'.join([r['stdout'] + '\n' + r['stderr'] for r in baseline])
        fclass = classify_failure(merged_logs)
        context = {
            'failure_class': fclass,
            'logs': merged_logs[-20000:],
        }
        # Hand off to your agent here. For demo, we simulate a patch.
        agent_patch = ''  # Obtain from LLM/agent
        if agent_patch and apply_patch(workdir.as_posix(), agent_patch):
            post = run_tests(workdir.as_posix(), retries=5)
            print('Post-fix stable pass?', stable_pass(post))
        else:
            print('Agent did not supply a valid patch.')

Handling Flakiness and Non-Determinism

Flaky tests are not noise; they are part of the phenomenon you need to measure. Recommendations:

• Multi-run sampling: Require success in ≥4/5 runs (configurable) to count as a pass.
• Cross-seed runs: Vary PYTHONHASHSEED, numpy/random seeds, and time mocking to ensure stability.
• Cross-environment sampling: At least two images per language version (e.g., Python 3.10 vs 3.11), or glibc vs musl builds.
• Flake classifier: Persistently misbehaving tests across unrelated changes should be tagged as flaky and optionally quarantined.
• Quarantine vs. fix: If the agent "fixes" flakiness by skipping tests, fail the submission; the harness must detect test skipping or blanket try/except patterns that mask failures.

Statistical heuristic example:

• Let p be the observed pass probability after the patch across R runs.
• Accept the fix if p ≥ 0.8 and the lower bound of a 95% Wilson interval exceeds 0.6.
• Flag potential flakiness if 0.2 < p < 0.8.

Runtime Context in Prompts Without Overload

More context is not always better; what matters is relevant context.

• Summarize coverage: provide top 20 lines/functions only.
• Include the shortest failing stack trace and elide vendor frames.
• Provide baseline vs. failing diff in logs.
• Supply a dependency manifest and the versions of top-10 suspect libraries (by import frequency in the failing file’s neighborhood).
• Allow the agent to pull more context on demand (tool calls) rather than front-loading megabytes.

Example prompt payload to an agent (abbreviated):

text
You are debugging a Python project that fails on commit 1f2a3b4c5d.

Failure class: runtime-exception
Stack trace (tail):
  File 'acme/core/encoder.py', line 117, in encode
    return np.asarray(x, dtype=np.float32)
AttributeError: module 'numpy' has no attribute 'asarray'  # observed under numpy==2.0.0

Coverage (failing test top hotspots):
  acme/core/encoder.py: lines 100-140
  acme/core/utils.py: lines 45-60

Environment:
  Python 3.11, numpy==2.0.0, pandas==2.1.0

Hypothesis: API drift in numpy.
Task: Propose a minimal patch that restores behavior under numpy 2.0 while keeping compatibility with numpy 1.26.

Example Case Study: API Drift-Induced Runtime Failure

Consider a repository pinned loosely to numpy>=1.20, now pulling numpy==2.0.0. A downstream function breaks with AttributeError or behavior changes around dtype promotion.

Harness steps:

1. Prepare environment with numpy 2.0.0 inside Docker.
2. Run tests 5 times: confirm stable failure (AttributeError).
3. Classify as API drift + runtime exception.
4. Capture coverage: identify encoder.py as hot.
5. Agent proposes a patch that replaces deprecated calls or adjusts dtype handling.
6. Validate patch across numpy 1.26 and 2.0 in separate containers; run tests 5× each.
7. Score: tests pass stably in both environments; patch size small; no regressions detected by fuzzing a small numeric input space.

The evaluation would credit cross-version compatibility and robustness, not just a single env green result.

Guardrails: Preventing Unethical or Invalid Fixes

Agents may attempt shortcuts:

• Disabling tests or changing test expectations without justification.
• Adding try/except: pass around wide code blocks.
• Altering CI scripts to skip steps.
• Reaching out to the public internet to fetch code during evaluation.

Mitigations:

• Disallow external network by default; allow only whitelisted registries during setup.
• Diff-checkers that flag changes outside src/ with strong penalties or outright failure if tests/ or CI configs are altered.
• AST-level inspection for blanket exception handlers or global side effects.
• Semantic guards: run an auxiliary linter rule set and mark risky changes.

Regression Risk: Going Beyond the Provided Tests

Tests are incomplete. Strengthen validation with:

• Differential testing: run the pre-patch and post-patch code on a random/fuzzed input corpus; flag semantic divergences unrelated to the fixed bug.
• Metamorphic testing: invariants like f(f(x)) == f(x) or scale invariance for certain transforms.
• API contract checks: respect documented preconditions and postconditions.
• Golden traces: for IO or parsing, compare key fields to known-good traces.

Even lightweight fuzzing (e.g., Hypothesis for Python, jqf/zest for Java) often reveals overfitted fixes.

Cost and Latency: Practical Metrics that Matter to Teams

For an AI debugger to be adoptable, it must be efficient:

• Measure time-to-first-patch and time-to-stable-green across your dataset.
• Track token usage and tool calls per fix; correlate with success.
• Allocate budgets: e.g., max 10 test runs and 3 build cycles per instance.

Report histograms rather than averages only—tail latency matters.

Continuous Evaluation: Drift is the Point

Set up a nightly or weekly CI that replays the dataset with controlled variation:

• Rotate dependency versions within allowed ranges using constraint solvers.
• Vary OS base images and CPU archs.
• Introduce synthetic network conditions (DNS fail, packet delay) for IO-sensitive projects.

Trend-line metrics per failure class will tell you if your agent gets better at, say, import/link issues but regresses on concurrency timeouts.

Putting It All Together: JSON Result Schema

Have the harness emit structured results for each bug instance:

json
{
  "instance_id": "example-repo@1f2a3b4c5d",
  "failure_class": "api-drift/runtime-exception",
  "env_matrix": ["py311-np200", "py311-np126"],
  "pre_runs": {"passes": 0, "fails": 5},
  "post_runs": {
    "py311-np200": {"passes": 5, "fails": 0},
    "py311-np126": {"passes": 5, "fails": 0}
  },
  "regression_checks": {"fuzz": "clean", "metamorphic": "clean"},
  "patch": {"lines_added": 6, "lines_deleted": 2, "files_changed": 1},
  "safety": {"tests_modified": false, "broad_try_except": false},
  "cost": {"test_runs": 12, "builds": 2, "tokens": 34000, "wall_clock_seconds": 780},
  "outcome": "stable-pass"
}

This makes it easy to compute aggregate stats and to audit individual cases.

How This Differs from Typical Leaderboards

• We explicitly model drift and non-determinism.
• We stratify by failure class to avoid single-number illusions.
• We emphasize runtime context rather than static code-only prompts.
• We score robustness and regression risk, not just a one-off green.
• We track cost and operational feasibility, which matters in real teams.

Tools and References

• Benchmarks/datasets:
  • Defects4J — real Java faults: https://github.com/rjust/defects4j
  • Bugs.jar — https://github.com/bugs-dot-jar/bugs-dot-jar
  • ManyBugs — http://repairbenchmarks.cs.umass.edu/ManyBugs/
  • QuixBugs — https://github.com/jkoppel/QuixBugs
  • Bears — https://github.com/bears-bugs/bears-benchmark
  • BugsInPy — https://github.com/soarsmu/BugsInPy
  • SWE-bench — https://github.com/princeton-nlp/SWE-bench
• Reproducibility and tracing:
  • Docker — https://www.docker.com/
  • Nix — https://nixos.org/
  • Bazel — https://bazel.build/
  • ReproZip — https://www.reprozip.org/
  • OpenTelemetry — https://opentelemetry.io/
  • eBPF — https://ebpf.io/
• On test flakiness:
  • Google Testing Blog: Flaky Tests at Google — https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

Opinionated Recommendations for Teams Evaluating Debugging AI

• Don’t chase leaderboard headlines. Instead, pilot against your own code under your own CI constraints.
• Invest early in hermetic reproducibility. A one-time cost in Docker/Nix/Bazel saves endless confusion later.
• Build a small but representative internal dataset with your historical bugs, including a few painful drift cases and flaky suites.
• Instrument once, reuse everywhere: standardize trace capture and context packaging across repos.
• Measure robustness and cost. If an agent needs 20 minutes and 50 tool calls to fix a trivial import, it will not scale.
• Keep humans in the loop. Let engineers veto risky patches and feed those decisions back into the harness as negative examples.

Closing Thoughts

Debugging is inherently dynamic: it lives at the intersection of code, environment, and time. Any evaluation that flattens those dimensions into a static ranking risks misleading you about what will happen in your own repositories tomorrow.

A practical harness—rooted in failure taxonomies, sandboxed reproducibility, runtime traces, and multi-dimensional scoring—can reveal whether a debug-capable AI actually fixes bugs in the wild. It will surface strengths and weaknesses by failure mode, quantify robustness under drift, and keep your team honest about cost and safety.

That’s the kind of evaluation that improves real software, not just leaderboard positions.