RAG for Code Debugging AI: Turning Logs, Traces, and Diffs into Reproducible Fixes

Retrieval-augmented generation (RAG) has proven its value for knowledge-heavy tasks. In software engineering, the knowledge artifacts that matter most during debugging are not wiki pages; they are logs, traces, test failures, diffs, build environments, and the code itself. If you want an AI that reliably reproduces, localizes, and repairs bugs — not just writes plausible-looking patches — your retrieval corpus must be designed around those artifacts.

This article lays out a practical, opinionated blueprint for building a RAG system specifically for code debugging. It is aimed at teams who already have CI/CD, observability, and version control, and who want their AI to turn messy production evidence into deterministic, validated fixes.

Highlights:

• What to index (beyond source files)
• Schema design that connects evidence to code and environments
• Embeddings vs. symbols: hybrid retrieval that actually works for debugging
• PII-safe pipelines and governance choices
• CI/CD hooks that capture repro data and gate AI patches via tests
• Concrete snippets for ingestion, retrieval, and evaluation

If you build the corpus right, the AI’s job becomes straightforward: take a failure signature, retrieve relevant code and past fixes, assemble a repro harness, propose a patch, and run the tests. The retrieval layer is the difference between speculative code generation and reliable repair.

Why RAG for Debugging Is Different

Most code-assist tools retrieve by filename or symbol name, and maybe a docstring embedding. That helps with code completion, not debugging. Debugging is inherently evidentiary and temporal:

• Failures happen in a particular environment, at a particular time, under a specific configuration.
• Logs and traces are the breadcrumbs; stack frames and diffs are the smoking gun.
• Reproduction requires recreating the exact dependency and runtime context.

A debugging-focused RAG system must therefore:

1. Prioritize evidence-carrying artifacts (stack traces, failing test output, core logs, diffs, feature-flag states).
2. Bind those artifacts to a precise build/runtime fingerprint (commit SHA, container digest, dependency lock, env vars).
3. Retrieve across multiple modalities (symbolic code graph + dense text embeddings + exact-match filters).
4. Feed the AI both how and why: the repro harness alongside the code to fix.

What to Index: A Minimal-Complete Set

Think of your retrieval corpus as a versioned graph of three classes of nodes: Evidence, Code, and Environment. You do not need to index everything; you need to index the right things with the right joins.

Evidence (observability and change data):

• Stack traces: normalized to stable hashes, with file/function/line and language.
• Error events and logs: structured fields (logger, level, error code) + text body.
• Test failures: test identifiers, assertion messages, seed/seeded RNG values, run IDs.
• Runtime traces: OpenTelemetry spans, attributes, and links (service.version, deployment.environment).
• Diffs and commit metadata: patch hunks, touched symbols, commit message, author time.
• Crash dumps and thread dumps: minimized to signatures and top N frames.
• Feature flags and config snapshots: resolved values at runtime.
• API schemas and contracts: versioned OpenAPI/GraphQL schemas; useful to surface breaking changes.
• Incident tickets and postmortems: summarized and embedded for retrieval.

Code (semantics and structure):

• Symbol index: functions, classes, methods, fields, with file path and line ranges.
• AST snippets: per function or class; used for structural matching and repair constraints.
• Control-flow/call-graph edges: who calls what, for impact analysis.
• Type signatures and interface conformance: inferred types if dynamic.
• Test-to-code coverage: mapping from tests to production lines/modules.

Environment (determinism and reproducibility):

• Build manifest: commit SHA, branch, CI job ID, toolchain versions.
• Dependency lock: exact versions and checksums (requirements.txt + lock, package-lock.json, go.sum, Cargo.lock, etc.).
• Container or Nix/Bazel derivation digest: reproducible runtime artifact.
• OS/arch, kernel, locale, timezone.
• Env vars and secrets placeholders: sanitized but kept as shape.
• Seed and randomness sources: if fuzzing or property-based tests are used.

Indexing these with proper keys allows you to reconstruct a failing context and map it to the code that likely broke. The retrieval problem becomes: given a failure signature, pull the code and environment slices needed to reproduce and fix.

Schema Design: Link Everything With First-Class Keys

A good schema is biased toward joins you will actually perform during retrieval. The core primary keys are:

• run_id: unique execution or test run
• commit_sha: code version
• workspace_digest: container or build artifact hash
• error_signature: normalized failure key
• stack_signature: deterministic hash of top frames
• test_id: stable test identifier
• diff_id: change-set identifier
• symbol_id: fully qualified symbol (lang aware)

A normalized, language-agnostic event schema (in Python/pydantic style) can look like this:

python
from pydantic import BaseModel, Field
from typing import List, Dict, Optional
import hashlib

class StackFrame(BaseModel):
    language: str
    file_path: str
    function: str
    line: int
    module: Optional[str] = None
    symbol_id: Optional[str] = None  # like py:pkg.module.Class.method

class StackTrace(BaseModel):
    frames: List[StackFrame]
    exception_type: Optional[str] = None
    message: Optional[str] = None
    def signature(self) -> str:
        # hash of top K (file_path,function) pairs; ignore line numbers for stability
        key = '|'.join(f'{f.file_path}:{f.function}' for f in self.frames[:6])
        return hashlib.sha256(key.encode()).hexdigest()[:16]

class LogEvent(BaseModel):
    run_id: str
    commit_sha: str
    workspace_digest: str
    service: str
    level: str
    logger: Optional[str]
    message: str
    ts: float
    attributes: Dict[str, str] = {}
    stack: Optional[StackTrace]
    error_signature: Optional[str] = None

class TestFailure(BaseModel):
    run_id: str
    commit_sha: str
    test_id: str
    seed: Optional[int]
    stderr: Optional[str]
    stdout: Optional[str]
    stack: Optional[StackTrace]
    error_signature: Optional[str]
    coverage_modules: List[str] = []

class DiffHunk(BaseModel):
    diff_id: str
    file_path: str
    language: str
    added: List[str]
    removed: List[str]
    touched_symbols: List[str] = []

class EnvFingerprint(BaseModel):
    workspace_digest: str
    os: str
    arch: str
    toolchains: Dict[str, str]  # 'python':'3.11.6', 'gcc':'12.3'
    dependencies: Dict[str, str]  # 'requests':'2.31.0'
    env_vars: Dict[str, str]  # sanitized

Two design choices here are crucial:

• Stable signatures: Normalize stack traces to signatures that do not include line numbers. You want the same logical error to coalesce across commits.
• Language-aware symbol identifiers: No free-text function names. Include full qualification and language, and if possible a file:line range. This unlocks joins from stack frames to AST nodes and diffs.

Store embeddings sparingly and deterministically. Every entity that contains free text (log message, assertion, commit message) can have a cached embedding keyed by its content hash, not run_id. That way repeated messages are indexed once.

Embeddings vs. Symbols: Use Both, on Purpose

It is a mistake to treat debugging retrieval as a purely semantic text problem. The sweet spot is hybrid retrieval with three legs:

• Symbolic retrieval: exact or fuzzy matches on module/file/symbol names; call graph proximity; coverage overlap with failing test.
• Lexical retrieval: BM25 or equivalent on tokens for stack frames, filenames, error codes, and log phrases.
• Dense retrieval: embeddings for free-form messages, commit explanations, and incident write-ups.

When to use which:

• Stack traces: start with symbolic (frame-to-symbol) and lexical (file path) matching. Embed the exception message only as a re-ranking signal.
• Diffs: primarily symbolic (touched_symbols, file paths). Use embeddings on commit messages for context.
• Logs: semantic embeddings shine on multi-line messages and user input echoes (after redaction), but respect exact-match filters like error_code.
• Code snippets: AST and symbol indices should be primary; if you embed code, consider pooling at function granularity with identifier-preserving tokenizers.

A pragmatic scoring recipe:

• Hard filters: language == 'python', service == 'payments', time window around failure.
• Symbol overlap score: Jaccard similarity between stack trace symbols and diff touched_symbols.
• Path proximity score: longest common subpath between stack frame paths and diff paths.
• BM25 score: on error messages and assertion text.
• Embedding cosine: between failure description and commit messages / incident summaries.
• Time-decay: exponential decay on older evidence, but never zero if a signature matches.

Final score = 0.35 * symbol_overlap + 0.2 * path_proximity + 0.2 * BM25 + 0.2 * embedding + 0.05 * time_decay

Tune the weights per repo; automated learning-to-rank with historical fix outcomes can learn better weights.

PII-Safe Pipelines Without Ruining the Signal

Your logs and traces likely contain sensitive data. A debugging RAG without privacy-by-design will be shut down by your security team. You need to remove sensitive payloads while preserving structure and error signal.

Principles:

• Structured redaction first: field-aware scrubbing beats regex-only. Keep keys, hash values.
• Pseudonymize identifiers: deterministic hashing lets the same user_id correlate across events without revealing identity.
• Language/locale safe: be robust to non-ASCII and multi-language messages.
• Secrets detection: run dedicated detectors for tokens and keys (high recall, then human review for false positives during canaries).
• Vaulted reversibility (optional): only for analysts with break-glass approval; AI indexes never receive reversible tokens.

Example: streaming redactor for JSON-ish logs:

python
SALT = b'static-or-rotating-salt'

SENSITIVE_KEYS = {'email','ssn','token','authorization','password','card','cvv','address','phone','user_id'}

def dhash(value: str) -> str:
    import hmac, hashlib
    return hmac.new(SALT, value.encode('utf-8'), hashlib.sha256).hexdigest()[:16]

def redact(obj):
    if isinstance(obj, dict):
        out = {}
        for k, v in obj.items():
            lk = k.lower()
            if lk in SENSITIVE_KEYS:
                out[k] = f'<redacted:{lk}:{dhash(str(v))}>'
            else:
                out[k] = redact(v)
        return out
    if isinstance(obj, list):
        return [redact(v) for v in obj]
    if isinstance(obj, str):
        # strip long base64-like tokens
        if len(obj) > 64 and any(c in obj for c in '+/=_-'):
            return f'<redacted:blob:{dhash(obj)}>'
        return obj
    return obj

PII policy tips:

• Maintain a contract test suite showing that redaction does not break failure clustering quality (e.g., stack signatures and error codes unaffected).
• Split storage: raw logs in a secure lake with tight ACLs; AI index only consumes redacted streams.
• Data residency: keep embeddings and vectors in-region. If using managed vector DBs, pin region.

CI/CD Hooks That Capture Everything You Need to Reproduce

Helpless AI is born from missing context. The easiest way to guarantee reproducibility is to make your CI/CD pipeline capture it automatically when something fails.

Key hooks:

• Test runner wrapper: on any failure, emit a TestFailure event with stack trace, seed, coverage, and env fingerprint.
• Build step: emit EnvFingerprint for every build artifact with a workspace_digest.
• Post-merge job: compute symbol index and call graph for the new commit; compute DiffHunk(s) with touched symbols.
• Runtime hook: ship structured logs and traces with commit_sha and workspace_digest tags.

Example GitHub Actions snippets to persist repro context to an S3-like store and push metadata to an indexer:

yaml
name: ci
on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install
        run: |
          pip install -r requirements.txt
          pip install pytest pytest-cov
      - name: Run tests with repro capture
        env:
          COMMIT_SHA: ${{ github.sha }}
        run: |
          pytest -q --maxfail=1 --disable-warnings \
            --cov=myapp --cov-report=term-missing \
            || echo 'tests failed'
      - name: Collect artifacts on failure
        if: failure()
        run: |
          mkdir -p artifacts
          # capture dependency lock, env, and coverage
          pip freeze > artifacts/requirements.lock
          env | sort > artifacts/env.txt
          cp .coverage artifacts/.coverage || true
      - name: Upload artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: repro-${{ github.sha }}
          path: artifacts/

  index:
    needs: [ test ]
    runs-on: ubuntu-latest
    if: always()
    steps:
      - uses: actions/checkout@v4
      - name: Push metadata to indexer
        run: |
          python scripts/push_index.py --sha ${{ github.sha }} --artifacts artifacts/

The indexer (called by push_index.py) should read artifacts, compute stack signatures for any failures, compute the symbol index, and upload both to the retrieval store.

A Reference Storage and Indexing Architecture

• Metadata and joins: Postgres with strong relational keys (run_id, commit_sha, symbol_id). Add pg_trgm or text search for BM25-like ranking.
• Vector store: pgvector extension in Postgres or Milvus/Weaviate if you need distributed scale.
• Blob store: S3-compatible for artifacts (core dumps, coverage, container digests manifest).
• Search: Elasticsearch/OpenSearch for logs if you already run it; otherwise, rely on Postgres text search plus structured columns.

Simple table sketch:

• events_log(run_id, ts, level, logger, message, attributes_jsonb, commit_sha, workspace_digest, stack_signature, embedding_vector)
• test_failures(run_id, test_id, commit_sha, stack_signature, error_signature, coverage_modules, seed)
• diffs(diff_id, commit_sha, file_path, touched_symbols, hunk_text, commit_message_embedding)
• symbols(symbol_id, commit_sha, language, file_path, start_line, end_line, callers, callees, ast_blob)
• env_fingerprints(workspace_digest, os, arch, toolchains_jsonb, dependencies_jsonb)

With this layout, retrieval statements can join test_failures.stack_signature to events_log, then map stack frames to symbols.symbol_id, then to diffs.touched_symbols for the suspect change-set.

Retrieval Recipes That Work in Practice

1. Given a failing test (test_id, run_id):

• Pull the TestFailure row and its stack_signature.
• Join to the most recent diffs whose touched_symbols intersect with the top stack frames or coverage_modules.
• Join to similar past failures by stack_signature and to any incidents linked by error_signature.
• Retrieve corresponding EnvFingerprint and container digest.
• Return: top 5 suspect diffs + code snippets of affected functions + repro harness.

2. Given a production error log snippet:

• Normalize to a stack_signature and error_signature.
• Search logs for the same signature in the last N days; cluster by commit_sha.
• For the dominant commit_sha, pull diffs and symbol index. If multiple services are involved (traces), intersect by service.version.
• Return: probable change window, affected symbols, and the smallest environment delta.

3. Given a patch in a PR that causes flaky tests:

• Compute touched_symbols.
• Look up historical failures where test_id coverage overlapped touched_symbols.
• Retrieve prior fixes and their commit messages; show common patterns (race detection, timeouts, missing await, etc.).

These flows are amenable to simple API endpoints that your LLM agent can call to fetch structured packs of context.

From Retrieval to Reproduction: Make It One Call

Your agent should not guess how to set up the environment. Provide a single structured artifact called a Repro Pack that the agent can request by run_id or error_signature.

Repro Pack contents:

• workspace_digest or container image ref
• commands to run tests or a specific failing test
• env var diff vs. baseline
• seed and test flags
• minimal dataset or fixture references
• code files and line ranges to inspect (based on retrieval)

Example spec (YAML-ish):

yaml
repro:
  workspace_digest: sha256:abc123... # OCI image or Nix derivation
  base_command: pytest -q
  failing_tests:
    - test_id: tests/test_invoice.py::test_decimal_serialization
      seed: 1729
      args: --maxfail=1 -k test_decimal_serialization -s
  env:
    TZ: UTC
    FEATURE_FLAGS: invoice_v2=false
  data_fixtures:
    - s3://debug-bucket/fixtures/invoices_small.ndjson
  files_of_interest:
    - path: app/serialization/json.py
      lines: 1-140
    - path: app/routes/invoices.py
      lines: 40-120

Your CI should be able to materialize this pack locally via a Make target or a CLI, and your AI agent should simply reference it.

Worked Example: Turning Logs and Traces into a Fix

Scenario: A Python microservice begins returning 500s on an endpoint that serializes Decimal values in invoices. Logs show errors like: Object of type Decimal is not JSON serializable.

What the retrieval system does:

• Normalize the error to error_signature: python:TypeError:DecimalNotSerializable and stack_signature based on frames in app/serialization/json.py: dumps, and in app/routes/invoices.py: get_invoice.
• Join to the most recent diffs touching app/serialization/json.py — a commit refactoring JSON dumps to use the stdlib json.dumps without a default handler.
• Retrieve past incidents where Decimal issues appeared; pull patches that added a default serializer using str(x).
• Pull EnvFingerprint confirming Python 3.11.6 and no third-party JSON libraries.
• Assemble Repro Pack: run pytest with the failing test test_decimal_serialization using a provided fixture file.

Agent’s repair strategy, enabled by retrieval:

• Open app/serialization/json.py, highlight dumps function.
• Propose a default serializer that handles Decimal and date/datetime.
• Add unit test to reproduce and guard the behavior.

Illustrative patch (conceptual):

diff
--- a/app/serialization/json.py
+++ b/app/serialization/json.py
@@
-from json import dumps as _dumps
+from json import dumps as _dumps
+from decimal import Decimal
+import datetime as _dt
+
+def _default(o):
+    if isinstance(o, Decimal):
+        return str(o)
+    if isinstance(o, (_dt.date, _dt.datetime)):
+        return o.isoformat()
+    raise TypeError(f'Object of type {type(o).__name__} is not JSON serializable')
 
-def dumps(obj: object) -> str:
-    return _dumps(obj)
+def dumps(obj: object) -> str:
+    return _dumps(obj, default=_default)

And a focused test (guardrail) added to tests/test_serialization.py:

python
from decimal import Decimal
from app.serialization.json import dumps

def test_decimal_serialization():
    assert dumps({'amount': Decimal('10.50')}) == '{"amount": "10.50"}'

The agent applies the patch, runs the Repro Pack’s command, passes the test, and the CI runs full regression. This is not hypothetical; it is precisely the class of bug for which hybrid retrieval excels: logs identify the failing path, diffs highlight the regression window, code snippets localize the repair site, and the repro harness makes it deterministic.

Code Ingestion and Indexing Snippets

A compact ingestion worker that reads from a log topic, performs redaction, computes signatures and embeddings, and stores records:

python
import json, time
import psycopg
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

def embed(text: str):
    return model.encode([text])[0]

conn = psycopg.connect('postgresql://ai:***@db/ai')

INSERT_LOG = '''
insert into events_log(run_id, ts, level, logger, message, attributes_jsonb,
  commit_sha, workspace_digest, stack_signature, embedding_vector)
values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
'''

def process_log(rec):
    log = LogEvent(**rec)  # from earlier schema
    msg = log.message
    # compute or reuse embedding via content hash
    emb = embed(msg[:4096]) if msg else None
    with conn.cursor() as cur:
        cur.execute(INSERT_LOG, (
            log.run_id, log.ts, log.level, log.logger or '', msg,
            json.dumps(log.attributes), log.commit_sha, log.workspace_digest,
            log.stack.signature() if log.stack else None, emb))
    conn.commit()

Indexing symbols using tree-sitter (example in Python) and storing symbol_id rows:

python
from tree_sitter import Language, Parser
# assume languages.so built with tree-sitter-python, -js, -go, etc.
PY_LANG = Language('build/languages.so', 'python')
parser = Parser(); parser.set_language(PY_LANG)

def list_symbols(code: str, path: str):
    tree = parser.parse(bytes(code, 'utf8'))
    # walk the AST to collect function and class definitions with start/end positions
    # pseudo-implementation; use a proper query in real code
    return [
        {
            'symbol_id': f'py:{path}:{name}',
            'file_path': path,
            'language': 'python',
            'start_line': start,
            'end_line': end
        }
        for (name, start, end) in extract_defs(tree, code)
    ]

Diff-to-symbol mapping with GumTree-like differencing (or simple heuristics by line ranges) gives you touched_symbols so you can prioritize likely fix sites.

Evaluating Retrieval for Debugging

You can (and should) quantify whether retrieval helps your AI fix bugs. Build an offline benchmark:

• Dataset: historical failures with known fixes (link run_id/error_signature to a fixing commit). Public corpora like Defects4J (Java), BugsInPy (Python), ManySStuBs4J (small-stub fixes) and SWE-bench (Python) are good starting points.
• Queries: construct queries from failure artifacts (stack traces, log snippets, failing test names).
• Relevant set: ground-truth diffs, files, or symbols touched by the fix.
• Metrics: nDCG@k or recall@k for files/symbols; MRR for pulling the fixing commit; time-to-repro (seconds to materialize a passing repro harness).

Track the following in CI:

• % of failures for which a Repro Pack can be constructed automatically.
• % of AI-proposed patches that compile and pass the failing test.
• % of patches that pass full regression.
• Mean number of retrieval artifacts consumed per successful fix (cost awareness).

Common Failure Modes and How to Avoid Them

• High-cardinality logs drown embeddings: fix by heavy use of hard filters (service, env, commit window) before any vector search.
• Over-chunking code kills locality: index at function/class granularity, not 200-line chunks.
• Line-number brittle signatures: ignore line numbers in stack signatures; include file path and symbol name instead.
• PII redaction breaks error matching: keep deterministic hashes for identifiers and keep error codes intact.
• Non-hermetic repro: freeze your runtime via container or Nix/Bazel; capture toolchain versions in EnvFingerprint.
• Agent patch proposes API changes: constrain generation to only touched files/symbols unless tests demand otherwise; ask the agent to produce minimal edits.

Tooling Choices and Trade-offs

• Vector DB: Start with pgvector if your data is <100M embeddings and you want transactional joins; otherwise Milvus or Weaviate for scale-out and HNSW/IVF indexes.
• Search: If you already run OpenSearch/Elasticsearch, leverage it for logs and BM25; otherwise Postgres full-text is enough for many teams.
• Symbol extraction: tree-sitter covers many languages with robust parsing. For security-heavy code analysis, consider code property graphs (e.g., Joern) to get data/control flow.
• Tracing: OpenTelemetry is the default. Ensure span attributes include commit_sha and service.version.
• Build determinism: Docker with image digests is OK; Nix or Bazel provides stronger hermetic reproducibility.

Governance and Access Control

• Principle of least privilege: AI agents can read indices and sanitized artifacts only; no write access to production.
• Secrets boundary: ensure your ingestion pipelines validate that no secrets reach embeddings; reject on detection with alerts.
• Retention and TTL: keep high-resolution logs for days, signatures and embeddings for months; compress or summarize older data.
• Auditability: every AI patch should link back to retrieval artifacts used and include an explanation of why files were modified (grounded in evidence).

Extending to Concurrency, Performance, and Memory Leaks

Not all bugs are simple exceptions. For concurrency deadlocks, performance regressions, or leaks, emphasize these artifacts:

• Thread dumps and lock graphs: index contention pairs and lock order anomalies.
• Flamegraphs and span timelines: store as lightweight profiles and index with service.version.
• Allocation sites: for leak detectors, map stack traces to symbols and diffs.

Retrieval should prioritize recent diffs touching synchronization primitives, shared caches, or allocation-heavy paths.

A Lightweight Retrieval API for Your Agent

Expose a simple HTTP/gRPC interface that returns structured packs, not just text:

• GET /repro/by-run/{run_id} -> Repro Pack
• POST /search/failure -> returns suspect diffs, files_of_interest, env
• POST /search/symbols -> returns call graph neighborhood for a symbol
• POST /patch/validate -> apply patch in sandbox, run repro tests, return results

Example failure search request:

json
{
  "service": "invoices",
  "language": "python",
  "log_snippet": "TypeError: Object of type Decimal is not JSON serializable",
  "stack_frames": [
    {"file": "app/serialization/json.py", "function": "dumps"},
    {"file": "app/routes/invoices.py", "function": "get_invoice"}
  ],
  "time_window": {"start": 1730566400, "end": 1730568200}
}

And an abbreviated response:

json
{
  "suspect_diffs": [
    {"diff_id": "b1f2...", "score": 0.83, "files": ["app/serialization/json.py"]}
  ],
  "files_of_interest": [
    {"path": "app/serialization/json.py", "lines": [1,140]}
  ],
  "env": {"workspace_digest": "sha256:abc123...", "toolchains": {"python": "3.11.6"}}
}

CI as the Arbiter of Truth: Patch Validation

Even with perfect retrieval, AI patches must face reality:

• Apply patch in a clean workspace at commit_sha.
• Build with the captured toolchain and dependencies (workspace_digest).
• Run the Repro Pack test(s); if green, run a focused regression subset (tests touching the same modules) before full CI.
• Only then open or update the PR with the patch and validation artifacts attached (logs, coverage, Repro Pack reference).

This turns the AI into a junior engineer who always attaches a repro and proves their fix.

Quick Start Checklist (Opinionated)

• Instrumentation:
  • Add commit_sha and workspace_digest to all logs and spans.
  • Wrap test runners to emit TestFailure with stack signatures and seeds.
• Indexing:
  • Build a symbol index with tree-sitter per commit.
  • Compute touched_symbols for each diff.
  • Store EnvFingerprint for each build.
• Retrieval:
  • Implement hybrid scoring: symbol overlap + BM25 + embeddings + time-decay.
  • Expose Repro Pack API.
• Privacy:
  • Ship a deterministic redactor; prove via tests that clustering quality is intact.
  • Keep embeddings in-region; separate raw from sanitized stores.
• CI/CD:
  • On failure, upload artifacts and push metadata to the indexer.
  • Provide a Make target to materialize a Repro Pack locally.
• Evaluation:
  • Create a historical benchmark; track recall@k for suspect files and success rate of AI patches.

References and Pointers

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP (2020) — foundational RAG concept.
• Falleri et al., Fine-grained and Accurate Source Code Differencing (2014) — GumTree algorithm.
• Defects4J (Java), BugsInPy (Python), ManySStuBs4J — bug-fix corpora for evaluation.
• SWE-bench and SWE-bench Verified — end-to-end software engineering benchmarks with reproduction.
• OpenTelemetry — tracing/logging standard to attach commit_sha and service.version.
• tree-sitter — fast parsers for symbol extraction across languages.
• Joern — code property graphs for deep code analysis.

Closing

RAG for debugging is not about stuffing your vector database with code files. It is about curating a graph of evidence linked to code and environments so the AI can reproduce failures and propose targeted fixes. Do the unglamorous work — stable signatures, symbol indices, deterministic repro, PII hygiene — and the payoff is compound: faster mean-time-to-repair, safer automation, and an engineering team that trusts the AI because it always brings the receipts.