Shadow Mode for Code Debugging AI: From Read-Only Traces to Safe Auto-Fixes

Modern engineering teams want AI that does more than autocomplete. They want AI that reads logs and traces, finds the root cause, and proposes fixes that preserve service level objectives (SLOs). But handing a production codebase to an auto-fixing model is playing with fire. The practical path is shadow mode: an AI that observes, replays, diagnoses, and drafts safe change sets under watchful guardrails. Only once it proves itself against SLO gates and formal checks should it graduate to controlled rollouts.

This article lays out a concrete, opinionated playbook for deploying a code debugging AI in shadow mode: ingest telemetry, build deterministic replays, triage and root-cause incidents, propose safe patches, then validate through SLO gates and canary rollouts with automatic rollback and exhaustive audit trails. The audience is technical; expect details, examples, and code.

Executive Summary

• Start with read-only telemetry: logs, traces, metrics, and crash dumps via OpenTelemetry and your APM. Do not grant write privileges to production systems.
• Make reproduction boring. Deterministic replay is non-negotiable: record relevant inputs, isolate side effects, and force stable execution to recreate incidents locally and in CI.
• Align AI outputs with SLOs, not vibes. The AI proposes patches that must demonstrate SLO improvements in shadow canaries and regression tests.
• Use layered guardrails: static analysis, property checks, unit and fuzz tests, probabilistic SLO checks, and feature-flagged rollouts with automatic rollback.
• Keep a tamper-evident audit trail: inputs, model versions, prompts, diffs, reviewers, approvals, and rollout outcomes.
• Grow trust in phases: read-only analysis, then PR suggestions, then controlled auto-fixes behind rings and SLO gates. Never jump straight to auto-merge.

Opinion: deterministic replay and SLO-gated safety checks are the difference between a useful debugging AI and a source of novel incidents. Most real-world bugs involve concurrency, time, and mutable state. If you cannot reproduce deterministically, you cannot fix safely.

Why Shadow Mode for Debugging AI

Shadow mode means the AI runs beside your system without controlling it:

• It consumes the same telemetry as on-call humans.
• It reproduces incidents in a sandbox to gather ground truth.
• It proposes patches and mitigation runbooks.
• It never mutates production state or configuration directly.

The benefits are substantial:

• Lower blast radius: the AI cannot break prod when it is wrong.
• Faster learning loop: measure proposal quality against past incidents without rollouts.
• Measurable ROI: compare time-to-diagnosis and mean time to recovery (MTTR) with and without AI suggestions.
• Regulatory alignment: auditable artifacts for every suggestion, aligned with SOC 2, ISO 27001, and internal engineering standards.

The constraint is latency. Shadow systems must analyze quickly enough to matter during an incident. That is why deterministic replay and aggressive data reduction (slices of code and traces) are core.

Architecture: From Telemetry to Safe Patch

A pragmatic architecture for shadow-mode debugging AI comprises nine layers.

1. Telemetry Ingestion

• Collect logs, metrics, and traces using OpenTelemetry (OTel) SDKs and collectors.
• Store structured logs with request IDs, correlation IDs, and span IDs.
• Normalize stack traces and error codes; enrich with commit SHA, feature flag states, service version, and environment.

2. Deterministic Replay

• Record inputs: HTTP/gRPC requests, message payloads, DB query result snapshots, time, and RNG seeds.
• Virtualize time and randomness; stub external calls; isolate file system.
• Reproduce in hermetic containers; pin toolchains (Nix, Bazel, or locked Docker images).

3. Incident Triage

• Detect SLO violations: error rate, latency, saturation, or specific error signatures.
• Rank incidents by blast radius and SLO budgets.
• Select representative traces and inputs for replay.

4. Root Cause Analysis (RCA)

• Use program slicing: map spans and stack frames to code regions.
• Correlate deployments and config changes with error spikes.
• Identify minimal fix surfaces: code diffs, library upgrades, or configuration tweaks.

5. Patch Proposal Generation

• Input: code slice, failing replay, stack traces, and relevant dependency metadata.
• Output: a small diff, tests reproducing and verifying the fix, and a rationale.
• Constraints: code style, secure coding policies, and architectural invariants.

6. Safety Gates

• Static checks: linters, type checkers, secret scanners, license scanners.
• Tests: unit, property-based, fuzzing, and replayed scenarios.
• SLO gates: shadow traffic evaluation and statistical checks.

7. Controlled Rollout

• Create a PR with labels and an approval policy.
• Run canary release behind a feature flag or ring; mirror traffic.
• Auto-rollback on SLO breach.

8. Audit Trail

• Persist inputs, prompts, model IDs, diffs, tests, approvals, rollout metrics, and rollback events.
• Cryptographically sign build and release artifacts (Sigstore, Rekor) for provenance.

9. Privacy and Safety

• Redact PII from prompts; use reversible tokens if needed.
• Restrict AI to principle of least privilege; isolate networks and secrets.

Reference Flow

• A PagerDuty incident is created when error rate exceeds the 30-minute SLO budget.
• The system selects top N failing traces with distinct signatures and pulls the corresponding inputs.
• The replay harness reproduces the failures in containers.
• The AI proposes a minimal code patch and a unit test that fails before the patch and passes after.
• CI runs: static analysis, unit and integration tests, and replay tests.
• A shadow canary mirrors 5% of traffic to the patched service (dark launch) and compares SLO metrics against control.
• If improvements are significant, a PR moves to human review; otherwise, it is auto-closed with feedback logged.

Deterministic Replay: The Hard Part You Cannot Skip

Most teams underestimate how much non-determinism leaks into production bugs. Time, randomness, concurrency, network ordering, and third-party services all conspire to make reproduction flaky. The debugging AI cannot reason reliably unless it can run the exact failing path.

Techniques by runtime:

• Linux native and C/C++: rr (record and replay) captures syscalls and ordering; pair with perf and eBPF uprobes for minimal overhead in production sampling. Replay in CI for instruction-level determinism. Pernosco provides visual analysis.
• JVM: Java Flight Recorder, async-profiler, and deterministic random seeds. Use classpath locking and pinned container images.
• Go: -race and execution tracing; use time.Now() abstraction; stub network with net/http/httptest and recorded fixtures; freeze goroutine scheduling using GOMAXPROCS=1 for replay.
• Python: pytest with freezegun for time, vcrpy or respx for HTTP capture, deterministic random via random.seed; isolate with virtualenv and pinned wheels.
• Node.js: nock or Polly.js for HTTP capture; fake timers; lockfile with exact versions; Docker with reproducible builds.

Side-effect isolation:

• Database: capture result snapshots for read-only queries and run against ephemeral DB clones; use transaction snapshots or logical replication to seed state.
• Message buses: persist inbound messages and offsets; replay to a shadow consumer; stub outbound with idempotency keys to avoid duplicate effects.
• Filesystem: mount ephemeral volumes; snapshot fixtures.

Network virtualization:

• Service mesh (Istio, Linkerd) to mirror packets to a shadow deployment.
• Local proxies (Hoverfly, WireMock, Mountebank) to serve recorded responses.

Minimal Replay Harness Example (Python)

python
# replay_harness.py
import json
import os
import random
from contextlib import contextmanager
from freezegun import freeze_time
import requests

@contextmanager
def stub_network(fixtures_dir):
    # Monkeypatch requests to serve recorded responses
    import requests.adapters
    class FixtureAdapter(requests.adapters.HTTPAdapter):
        def send(self, request, **kwargs):
            path = request.path_url.replace('/', '_')
            fname = os.path.join(fixtures_dir, f'{request.method}_{path}.json')
            with open(fname, 'r') as f:
                payload = json.load(f)
            from requests import Response
            resp = Response()
            resp.status_code = payload['status']
            resp._content = json.dumps(payload['body']).encode('utf-8')
            resp.headers.update(payload.get('headers', {}))
            return resp
    s = requests.Session()
    s.mount('http://', FixtureAdapter())
    s.mount('https://', FixtureAdapter())
    yield s

@freeze_time('2025-01-01T00:00:00Z')
def run_replay(entrypoint, input_fixture):
    random.seed(1337)
    with open(input_fixture, 'r') as f:
        event = json.load(f)
    with stub_network('fixtures') as sess:
        return entrypoint(event, sess)

This harness makes time and random deterministic, and replaces network calls with recorded fixtures. Your entrypoint function should accept inputs and a session-like object. You will still need to handle DB state snapshots and concurrency separately.

Incident Triage with Telemetry

Triage should be SLO-aware and trace-aware.

• Define SLOs for error rate, tail latency, and saturation per service and endpoint.
• Trigger incident candidates when burn rate exceeds thresholds (see Google SRE Workbook). Prioritize by error budget consumption.
• Use trace exemplars: for each cluster of similar failures (stack trace + span path), pick the top exemplars with the highest request volume or recency.

A ClickHouse or BigQuery query can identify hot errors quickly.

Example BigQuery-style query over OTel traces in a lakehouse:

sql
-- Find dominant error signatures in the last 30 minutes
SELECT
  service_name,
  span_name,
  REGEXP_EXTRACT(stack, r'^(.*?)(\n|$)') AS top_frame,
  COUNT(*) AS errors,
  APPROX_TOP_COUNT(http_target, 5) AS top_paths
FROM `otel.traces`
WHERE
  end_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 MINUTE)
  AND status_code = 'ERROR'
GROUP BY 1,2,3
ORDER BY errors DESC
LIMIT 20;

This drives the replay selection by mapping dominant error signatures to real traces you can fetch and reproduce.

Root Cause Analysis: Mapping Traces to Code

The AI does not need the entire repository to fix a bug. It needs the relevant slice.

• Build a symbol map from spans and stack frames to files and functions using debug info and source maps.
• Use program slicing: static and dynamic analysis to narrow the dependency graph to code that touches erroneous spans.
• Cross-check with recent changes: commits in the deployment window and configuration diffs.
• Generate a minimal reproducer: the input event and initial state that triggers the failure.

This yields input for the model: the failing function, the sequence of calls, and the exact assertion or exception that fails in replay.

Example RC Prompt Structure

Rather than raw text, send a structured prompt to your model. Even with general LLMs, a fixed schema improves reliability.

json
{
  "service": "payments-api",
  "commit": "2f19c8d",
  "language": "python",
  "stack": [
    {"file": "handlers/charges.py", "func": "create_charge", "line": 137},
    {"file": "lib/serializer.py", "func": "to_json", "line": 45}
  ],
  "error": {
    "type": "TypeError",
    "message": "Object of type Decimal is not JSON serializable"
  },
  "repro": {
    "fixture": "fixtures/charge_request.json",
    "replay_cmd": "pytest -q tests/replay/test_charge.py::test_charge_decimal"
  },
  "constraints": {
    "max_diff_lines": 25,
    "no_api_contract_change": true,
    "perf_regression_budget_ms": 5
  }
}

The model can respond with a diff, test patch, and rationale in the same schema.

Patch Proposal Generation

A debugging AI should output:

• A minimal, type-safe diff that fixes the failing replay.
• A test that fails before the patch and passes after.
• A migration or config change if necessary, with fallbacks.
• A rationale referencing the trace and stack evidence.

Avoid free-form prose diffs. Use AST-aware edits where possible:

• tree-sitter or LibCST for Python
• comby for structural search and replace across languages
• go/ast for Go; javac Tree API for Java

This reduces syntactic errors and improves alignment with code style and formatters.

Example: Python Decimal serialization bug

Suppose the stack shows TypeError due to Decimal in a JSON response. The AI should:

• Propose a custom encoder or pre-convert Decimals to str.
• Add a unit test reproducing the failure.
• Confirm that response schemas remain unchanged except for stable number formatting.

Proposed patch (simplified):

diff
--- a/lib/serializer.py
+++ b/lib/serializer.py
@@
-import json
+import json
+from decimal import Decimal
 
-def to_json(obj):
-    return json.dumps(obj)
+def _convert(obj):
+    if isinstance(obj, Decimal):
+        return str(obj)
+    if isinstance(obj, dict):
+        return {k: _convert(v) for k, v in obj.items()}
+    if isinstance(obj, (list, tuple)):
+        return [ _convert(x) for x in obj ]
+    return obj
+
+def to_json(obj):
+    return json.dumps(_convert(obj))

Accompanying test:

python
def test_decimal_serialization():
    from decimal import Decimal
    payload = {'amount': Decimal('12.34')}
    s = to_json(payload)
    assert '12.34' in s

The replay harness should pass for the failing trace after this change.

Safety Gates: From Static Checks to SLO Compliance

A patch that passes unit tests is not necessarily safe. We add layered gates.

• Static checks: run linters (flake8, ESLint), type checkers (mypy, ts), secret scanners (gitleaks, trufflehog), and license scanners.
• Test checks: unit, integration, replay tests, and fuzzing on affected functions.
• Performance checks: microbench the changed functions and ensure no regressions beyond a budget.
• SLO canary checks: mirror traffic to a shadow or canary deployment and compare error/latency against control.

Shadow Traffic and Canary

Use a service mesh to mirror production traffic while preventing side effects.

Istio VirtualService example to mirror 5% of traffic to a shadow deployment without responses affecting clients:

yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments-api
spec:
  hosts:
  - payments
  http:
  - route:
    - destination:
        host: payments
        subset: stable
      weight: 100
    mirror:
      host: payments
      subset: shadow-fixed
    mirrorPercentage:
      value: 5.0

Ensure the shadow deployment uses stubbed outbound integrations to avoid double-charging or sending real emails.

SLO Gate as a Statistical Test

Do not trust eyeballing graphs. Treat shadow results as an A/B test.

• Define success metrics: error rate p and tail latency L95.
• Baseline from stable deployment using the same time window and request mix.
• Apply a sequential test (e.g., Wald SPRT) or Bayesian test to decide if the candidate is at least non-inferior.

Pseudocode for a simple non-inferiority check on error rate:

python
def non_inferior_errors(control, candidate, delta=0.001, alpha=0.05):
    # control and candidate are tuples (failures, total)
    from statsmodels.stats.proportion import proportions_ztest
    c_fail, c_total = control
    t_fail, t_total = candidate
    # H0: candidate - control >= delta (worse)
    count = [t_fail, c_fail]
    nobs = [t_total, c_total]
    stat, p = proportions_ztest(count, nobs, value=delta, alternative='smaller')
    return p < alpha

Carry a similar gate for latency using Mann-Whitney U or bootstrap.

Controlled Rollout and Automatic Rollback

Once gates pass, the patch still should not go global without control.

• Create a PR with labels and require codeowner and SRE approvals.
• Gate merges behind CI status, static checks, and canary success signals.
• Use feature flags to decouple deploy from release; progressively ramp exposure.
• Auto-rollback if SLOs regress beyond thresholds.

Argo Rollouts or Flagger can handle progressive delivery with metrics hooks to Prometheus or Datadog. Configure rollback on metric alarms.

Flagger example for a canary on Kubernetes with Prometheus metrics:

yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: payments-api
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 5
    stepWeight: 10
    maxWeight: 50
    metrics:
    - name: error-rate
      templateRef:
        name: error-rate
      thresholdRange:
        max: 0.02
      interval: 30s
    - name: latency-p95
      templateRef:
        name: latency-p95
      thresholdRange:
        max: 250
      interval: 30s

The AI does not control Flagger; it only supplies candidate deployments and watches outcomes.

Auto-Revert Strategy

Revert should be cheap and automatic:

• GitHub: enable auto-revert on failing post-merge checks; the AI can open a revert PR with context.
• Feature flags: default to off and roll back by flipping the flag in seconds.
• Immutable deployments: keep the last N images warm for instant rollback.

A small GitHub Action to auto-revert on post-deploy failure:

yaml
name: auto-revert
on:
  workflow_run:
    workflows: [post-deploy-checks]
    types: [completed]
jobs:
  revert:
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Revert last commit
      run: |
        git revert --no-edit HEAD
        git push origin HEAD

End-to-End Example: Fixing a 500 in a FastAPI service

Scenario: The payments-api emits 500s for create_charge in production. The on-call sees an SLO burn alert.

1. Detection

• SLO burn rate 10x over 30 minutes on POST /charges.
• Top error signature: TypeError in serializer.py line 45.

2. Trace selection

• Choose 20 exemplars with different request parameters.
• Pull HTTP request bodies, headers, and the DB snapshot for the customer rows involved.

3. Replay

• Spawn a hermetic container with pinned dependencies.
• Freeze time and random; stub outbound calls to gateway and email.
• Reproduce the error locally: Decimal not serializable in the JSON serializer.

4. RCA and proposal

• Model receives structured prompt with code slice and failing test.
• Proposes Decimal to string conversion via a custom converter plus tests.
• Diff is 18 lines; no API contract change; performance budget < 5 ms maintained.

5. Safety gates

• Static checks: pass (mypy, flake8, secret scan).
• Unit and integration tests: pass.
• Replay tests: failing exemplars now pass.
• Shadow canary: mirror 5% traffic; error rate drops from 2.8% to 0.0% for candidate; latency unchanged.
• Statistical non-inferiority passes for error rate and latency.

6. Rollout

• PR created with rationale and links to replay artifacts.
• Human reviewer approves; canary ramps 10% -> 50% -> 100% over 30 minutes.
• No SLO breaches. Release completes.

7. Audit

• All artifacts stored: prompts, model version, patch, tests, CI logs, canary metrics, approvals.
• SBOM and provenance signed with Sigstore; entry recorded in Rekor.

Result: MTTR reduced from hours to under 30 minutes, with a minimal, safe patch.

Audit Trails: Make It Tamper-Evident

If you cannot reconstruct what the AI saw, proposed, and decided, you cannot trust the system.

• Record every input: selected logs, traces, fixtures, and state digests used in replay.
• Record every output: diffs, tests, rationales, and risk scores.
• Record model identity: model name, version or hash, temperature, decoding params.
• Record reviewers, approvals, and policy gates that passed or failed.
• Sign artifacts and store immutable hashes. Use Sigstore cosign to attach provenance to container images and PR artifacts.

Example provenance payload (SLSA-inspired):

json
{
  "subject": {"repo": "github.com/acme/payments", "commit": "2f19c8d"},
  "ai": {"model": "code-llm-x", "version": "2025-03", "params": {"temp": 0.1}},
  "inputs": {"traces": ["trace-abc", "trace-def"], "fixtures": ["charge_1.json"]},
  "outputs": {"diff": "git-patch-sha", "tests": ["test_decimal_serialization"]},
  "checks": {"static": "pass", "replay": "pass", "slo": "pass"},
  "approvals": ["devoncall", "sre"]
}

Store this alongside build artifacts; write to an append-only log like Rekor or a WORM bucket.

Privacy, Security, and Compliance

• Minimize and redact: remove PII and secrets before prompts; tokenize if necessary and detokenize after.
• Isolation: run the AI in a segregated network segment without access to production secrets.
• Principle of least privilege: read-only access to telemetry and repo; no direct write to prod. Only CI bots perform merges and deploys under policy.
• Model governance: whitelisted model versions; freeze and canary new models like any other dependency.
• Compliance mapping: align artifacts with SOC 2 CC series and ISO 27001 A.12 change management controls.

Implementation Blueprint

Data Schemas

Define clear schemas for inputs and outputs.

• Event fixture

json
{
  "request": {
    "method": "POST",
    "path": "/charges",
    "headers": {"x-request-id": "abc123"},
    "body": {"amount": "12.34", "currency": "USD"}
  },
  "db_snapshot": "s3://bucket/snapshots/payments/abc123.json",
  "time": "2025-01-01T00:00:00Z",
  "rand_seed": 1337
}

• Candidate fix

json
{
  "diff": "patch-contents-or-ref",
  "tests": ["tests/replay/test_charge.py::test_decimal_serialization"],
  "rationale": "Convert Decimal to str to allow JSON serialization",
  "risk": {"surface": "serializer", "expected_perf_ms": 1.2}
}

Orchestrator Pipeline

A simple CI pipeline definition tying steps together:

yaml
name: ai-debugger-pipeline
on:
  workflow_dispatch:
  schedule:
    - cron: '*/10 * * * *'
jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Select incidents
      run: python tools/select_incidents.py --window 30m --top 5 --out incidents.json
    - name: Build fixtures
      run: python tools/build_fixtures.py incidents.json fixtures/
    - name: Run replays
      run: pytest -q tests/replay
    - name: Propose fixes
      run: python tools/propose_fixes.py fixtures/ --out candidates/
    - name: Apply diffs
      run: python tools/apply_diffs.py candidates/
    - name: Static checks
      run: make lint typecheck
    - name: Tests
      run: make test
    - name: Shadow canary analysis
      run: python tools/slo_gate.py --candidate sha --baseline stable --alpha 0.05
    - name: Open PRs
      if: success()
      run: python tools/open_prs.py candidates/

Policy as Code

Use OPA to enforce guardrails:

rego
package codefix.policy

# Require tests for any code fix touching prod code
require_tests[msg] {
  input.diff.touched_paths[_] == p
  startswith(p, "services/")
  count(input.tests) == 0
  msg := sprintf("missing tests for path %s", [p])
}

# Deny direct changes to infra unless labeled
deny[msg] {
  input.diff.touched_paths[_] == p
  startswith(p, "infra/")
  not input.labels[_] == "infra-approved"
  msg := sprintf("infra change without approval: %s", [p])
}

Cost and Latency Controls

• Pre-compute slices: do not send the whole repo to the model; ship only relevant files and spans.
• Cache embeddings and code indices per commit.
• Use a smaller local model for triage and a bigger one for final patch proposals.
• Batch replay and testing across incidents to amortize CI costs.

Anti-Patterns and Failure Modes

• Skipping replay: relying on logs alone leads to brittle fixes and high false positives.
• Overly broad diffs: refactors disguised as fixes increase risk and review time.
• No SLO gates: deploying based on intuition instead of measured improvement invites regressions.
• Silent prompt drift: changing prompts or model versions without versioning makes RCA audits impossible.
• Leaky shadow traffic: failing to stub side effects leads to duplicate actions in prod (payments, emails). Always double-insulate.

Measuring ROI

Track the following before and after adopting shadow-mode AI:

• Time to first plausible root-cause hypothesis
• MTTR for incident classes that the AI covers
• PR acceptance rate for AI-generated patches
• SLO-impacting regressions caught in shadow versus in prod
• Human review time per AI PR

Expect early wins in repetitive incident classes (serialization, nil checks, off-by-one, flaky tests, configuration misalignments). Harder categories include distributed transactions and data consistency bugs; still, AI can often propose mitigations and diagnostics.

Phased Adoption Plan

• Phase 0: Read-only adviser
  • The AI summarizes incidents and suggests lines of inquiry based on traces and code slices. No code changes.
• Phase 1: Replay-backed proposals
  • The AI generates diffs and tests that pass in CI. Human merges only.
• Phase 2: Shadow canaries
  • PRs auto-launch shadow traffic canaries with SLO gates. Human approves production rollout.
• Phase 3: Guarded auto-fixes
  • Low-risk classes (serialization, pure functions) can auto-merge if all gates pass, with instant rollback.
• Phase 4: Organization-wide templates
  • Standardize policies, prompts, and replay harnesses across services.

Tools and References

• Telemetry and replay
  • OpenTelemetry, Jaeger, Tempo, Honeycomb
  • rr, Pernosco, eBPF
  • JVM Flight Recorder, async-profiler
  • pytest, freezegun, vcrpy; nock, Polly.js; Hoverfly, WireMock
• Program analysis and diffs
  • tree-sitter, LibCST, comby, go/ast
• Delivery and rollbacks
  • Argo Rollouts, Flagger, Spinnaker; Istio; LaunchDarkly/Unleash feature flags
  • Sigstore, Rekor for provenance
• Stats and testing
  • statsmodels, scipy; hypothesis for property-based testing

A few good reads:

• Google SRE Workbook: practical SLO and burn rate alerts
• Practical record/replay with rr and Pernosco
• Deterministic builds with Nix and Bazel

Conclusion

Shadow mode is not a half-measure; it is the engineering discipline that makes AI-driven debugging real. Ingest the right telemetry, make replay deterministic, and let SLOs arbitrate whether a proposed patch is better than today. Add layered safety gates, progressive delivery, and audit trails that stand up to scrutiny. If you follow this path, your AI stops being a novelty and starts being a reliable, measurable accelerant for incident response and bug fixing.

The takeaway is simple: make reproduction boring, make fixes small, and make safety measurable. Do that, and you can responsibly graduate from read-only traces to safe auto-fixes.