A checkout bug gets reported on Monday morning. Users can add items to the cart, apply a coupon, and click Pay. The button spins. The order page briefly flashes a success state. Then nothing. No confirmation email. No charge in the payment provider. No order in the back office.

An AI agent picks up the ticket, reads the stack trace, and patches the frontend error handling. The null reference is gone. Unit tests pass. The pull request is green. The deploy goes out.

By Tuesday, support volume doubles.

The original exception was never the real bug. It was just the visible symptom. The actual failure lived in the handoff between systems: the checkout app redirected before the payment session was fully persisted, the webhook arrived before the order record existed, and the “success” page rendered off a local optimistic state instead of a confirmed payment event. The agent fixed the code path it could see and broke the workflow it could not.

That pattern is getting more common.

As AI generates more production code, more pull requests are technically plausible, locally coherent, and dangerously incomplete. They close tickets by repairing isolated logic while quietly violating contracts at the seams: frontend to auth callback, app to email provider, CRM to support queue, backend to warehouse sync, product analytics to billing events. Traditional testing misses these failures because most teams still validate code paths, not user outcomes.

That is the uncomfortable shift engineering teams need to face: in modern systems, reliability is not mostly about whether a function returns the right value. It is about whether an action completes across multiple services, with real timing, redirects, retries, side effects, and asynchronous state changes.

If your verification strategy stops at unit tests, mocked APIs, and green CI/CD checks, you are not proving the workflow works. You are proving your assumptions are internally consistent.

The real problem is not bad code. It is broken handoffs.

Most production incidents in software-heavy companies do not come from spectacular algorithmic failures. They come from ordinary boundary mistakes:

A payment intent is created but the return URL is malformed.
An auth provider callback succeeds but the frontend loses session state during redirect.
A signup flow writes the user row but fails to enqueue the welcome email event.
A support form submits in the UI but the CRM field mapping changed, so tickets land without account IDs.
A cancellation event reaches billing but never propagates to feature entitlements.
An internal API starts returning a nullable field that downstream code assumed was always present.

These bugs are boring in the worst possible way. They are not deep computer science problems. They are operational seams where one system hands work to another.

AI coding agents are especially prone to creating or missing them for a simple reason: most agent loops optimize for local evidence. They inspect the repository, infer likely intent from nearby code, run the available tests, and stop when the checks pass. That is often enough to fix syntax, refactor duplication, update interfaces, or patch straightforward business logic. It is not enough to verify that a real customer action still succeeds when it crosses service boundaries.

The handoff bug is where local reasoning breaks down.

A function can be “correct” and still fail the business action. A route handler can return 200 and still lose the order. A modal can show “Success” and still not create the ticket. A test suite can be 100% green and still certify a dead workflow.

That is why so many teams feel confused after incidents: every component looked healthy in isolation.

Why AI-generated changes fail at integration seams

This is not because AI is uniquely sloppy. It is because integration seams are under-specified, under-tested, and often invisible in code review.

AI amplifies that weakness because it produces change faster than teams improve verification.

Here are the common failure modes.

1. The contract exists in behavior, not types

Engineers love explicit contracts: TypeScript interfaces, OpenAPI schemas, protobuf definitions. Those matter. But real system contracts often extend far beyond shape.

The true handoff contract may include:

event ordering n- idempotency requirements
redirect timing
retry semantics
required headers
metadata fields used by downstream automations
webhook signature validation
timeout tolerances
eventual consistency windows
side effects triggered only after a status transition

An AI agent can update the payload shape and keep the compiler happy while still breaking the behavioral contract.

For example, consider a payment flow:

ts
// Before
await payments.createCheckoutSession({
  customerId,
  orderId,
  successUrl: `${APP_URL}/checkout/success?order=${orderId}`,
  cancelUrl: `${APP_URL}/checkout/cancel`,
});

An agent “simplifies” the code:

ts
await payments.createCheckoutSession({
  customerId,
  orderId,
  successUrl: `${APP_URL}/success`,
  cancelUrl: `${APP_URL}/cancel`,
});

Locally, nothing crashes. The provider accepts the request. Tests mocked against createCheckoutSession still pass.

But downstream, the success page depended on the order query param to reconcile the returning customer with the pending order. The webhook might eventually repair the state, or it might race and fail if the order isn’t found. You do not get a clean exception in CI. You get a customer who paid and a support ticket three hours later.

2. Mocks preserve assumptions, not reality

Mocks are useful. Overused mocks are dangerous.

The whole point of a mock is to simulate a dependency in a way that makes tests deterministic and fast. But once your system’s risk shifts from local logic to inter-service behavior, mocks become an instrument for certifying your own fiction.

A typical mocked test looks like this:

ts
it('sends welcome email after signup', async () => {
  emailClient.send = vi.fn().mockResolvedValue({ messageId: '123' });

  const result = await signupUser({ email: 'a@example.com' });

  expect(result.ok).toBe(true);
  expect(emailClient.send).toHaveBeenCalledWith(
    expect.objectContaining({ template: 'welcome' })
  );
});

What does this actually prove?

That your code invoked a function.
That your code passed fields matching your expectation.
That your chosen fake returned success.

What does it not prove?

That the real provider accepted the payload.
That required merge variables exist.
That the sender domain is verified.
That rate limiting did not block the call.
That your queue worker actually executed.
That the email event was not suppressed by account state.
That the customer received the email in the flow where it matters.

This gap gets worse when AI edits code. The agent sees the mock, infers the contract from the test, and updates implementation to satisfy it. But the mock often captures only a thin slice of the real integration.

In practice, teams accidentally train both humans and agents to optimize for simulated dependencies rather than production behavior.

3. CI/CD gives binary signals for non-binary systems

A green CI/CD pipeline feels authoritative because it compresses complexity into a single decision: ship or do not ship.

The problem is that real workflows are not binary and not synchronous.

A modern user action might involve:

Browser state updates
API request to your app
Insert into primary database
Message published to queue
Worker consumes event
Call to third-party API
Redirect through identity or payment provider
Webhook back into your system
Background reconciliation
Email or CRM side effect
UI eventually reflecting final state

Most CI pipelines do not execute that chain end to end. They run linting, unit tests, container builds, maybe a few integration tests against local dependencies, and then declare success.

That is not verification of the workflow. That is static confidence wrapped in automation.

A representative GitHub Actions setup often looks like this:

yaml
name: ci

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
        ports: ['5432:5432']
      redis:
        image: redis:7
        ports: ['6379:6379']
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run lint
      - run: npm run test
      - run: npm run test:integration

This is fine as far as it goes. But notice what it does not do:

no real browser redirect through auth or payment
no real webhook loop
no real provider-side validation
no proof that support, CRM, or email systems received the expected side effect
no measurement of eventual consistency or race conditions

The CI/CD check is green because the environment was shaped to be green.

4. QA can click through a path without validating the action

Manual QA still catches plenty of issues. But it also has structural blind spots for handoff failures.

A tester can complete a checkout UI flow in staging, see a success page, and mark the ticket verified. Meanwhile:

the payment provider in staging is stubbed
the email provider is disabled
the CRM integration points to a sandbox nobody monitors
the webhook tunnel is flaky
the callback timing differs from production
feature flags alter the path for real users

What got validated was the appearance of success, not the completion of the workflow.

This distinction matters more now because AI-generated changes can preserve the visible affordances while changing the semantics underneath. A button still clicks. A redirect still happens. A toast still appears. The side effect chain is what broke.

The core insight: correctness must move from code-level assertions to action-level proof

This is the shift teams need to make.

The question is no longer just:

Does this function return the right value?
Does this route handler respond correctly?
Does this component render the expected state?

The more important question is:

Can a user complete the action, across real boundaries, and can we prove the intended outcome happened?

That means testing has to center on actions like:

user signs up and receives a usable session
customer checks out and order is actually paid
password reset email is sent and token completes the flow
support request appears in the right queue with the right metadata
CRM lead is created and ownership rules are applied
subscription cancellation revokes access within the expected window

These are not code paths. They are business workflows.

The test oracle cannot just be “the API returned 200” or “the DOM contained Success.” The oracle has to include observable effects across systems.

This is where a lot of debugging effort should move. Not more assertions inside isolated modules. More evidence at boundaries.

What action-level verification looks like

Start by expressing workflows as outcomes.

Instead of this kind of test:

ts
it('creates checkout session', async () => {
  const session = await createCheckoutSession(order);
  expect(session.url).toContain('checkout');
});

Aim for something closer to this:

ts
import { test, expect } from '@playwright/test';

test('customer can complete checkout and order is paid', async ({ page, request }) => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();

  await page.getByLabel('Email').fill('buyer@example.com');
  await page.getByRole('button', { name: 'Pay now' }).click();

  // Real redirect flow or provider-hosted test flow
  await page.waitForURL(/\/checkout\/success/);

  await expect(page.getByText('Thanks for your order')).toBeVisible();

  const orderResponse = await request.get('/api/test/orders/latest?email=buyer@example.com');
  const order = await orderResponse.json();

  expect(order.status).toBe('paid');
  expect(order.paymentProviderStatus).toBe('succeeded');
  expect(order.confirmationEmailSent).toBe(true);
});

This is still only part of the answer, but it is pointed in the right direction. The flow asserts on business outcomes, not just local return values.

For async systems, action-level proof often requires polling rather than immediate assertions.

ts
async function waitFor(fn, { timeout = 15000, interval = 500 } = {}) {
  const start = Date.now();
  while (Date.now() - start < timeout) {
    const value = await fn();
    if (value) return value;
    await new Promise(r => setTimeout(r, interval));
  }
  throw new Error('Timed out waiting for condition');
}

Then:

ts
const ticket = await waitFor(async () => {
  const res = await request.get('/api/test/support/latest?email=user@example.com');
  const data = await res.json();
  return data?.queue === 'priority' ? data : null;
});

expect(ticket.accountId).toBeTruthy();

That style of testing acknowledges reality: distributed workflows are delayed, retried, reordered, and eventually consistent.

Practical example: auth callback bugs that unit tests miss

A classic seam is frontend to auth provider callback.

Suppose an AI agent refactors your login callback handler.

ts
// Next.js route handler
export async function GET(req: Request) {
  const url = new URL(req.url);
  const code = url.searchParams.get('code');

  if (!code) {
    return Response.redirect('/login?error=missing_code');
  }

  const session = await exchangeCodeForSession(code);
  await saveSession(session);

  return Response.redirect('/app');
}

The agent “improves performance” by making session persistence lazy:

ts
export async function GET(req: Request) {
  const url = new URL(req.url);
  const code = url.searchParams.get('code');

  if (!code) {
    return Response.redirect('/login?error=missing_code');
  }

  const session = await exchangeCodeForSession(code);
  void saveSession(session);

  return Response.redirect('/app');
}

Unit tests may still pass:

ts
it('redirects to app after auth callback', async () => {
  const res = await GET(new Request('https://app/callback?code=abc'));
  expect(res.status).toBe(302);
});

But real users now intermittently land on /app before the session is available in storage. The app flashes logged out, retries, and may strand the user depending on client behavior. This is a workflow failure created by a locally reasonable optimization.

A better verification setup would run the actual browser redirect flow and assert authenticated state after navigation.

ts
test('user can login through provider callback and reach authenticated app', async ({ page }) => {
  await page.goto('/login');
  await page.getByRole('button', { name: 'Continue with Auth' }).click();

  // complete provider sandbox login here

  await page.waitForURL(/\/app/);
  await expect(page.getByText('Welcome back')).toBeVisible();
  await expect(page.getByTestId('user-avatar')).toBeVisible();
});

Not because browser tests are trendy, but because the bug only exists in the handoff.

Practical example: backend side effects with Python

The same issue appears in backend systems. Consider a support escalation flow.

python
# app/support.py
from crm import create_ticket
from queueing import publish_event

def escalate_account(account_id: str, reason: str) -> dict:
    ticket = create_ticket(account_id=account_id, reason=reason)
    publish_event("support.ticket.created", {
        "account_id": account_id,
        "ticket_id": ticket["id"],
        "reason": reason,
    })
    return ticket

An AI agent sees duplicate logic and decides to make the event asynchronous through a background task. Fine in principle. But it also changes the payload key because another nearby function used accountId.

python
# subtle break
publish_event("support.ticket.created", {
    "accountId": account_id,
    "ticket_id": ticket["id"],
    "reason": reason,
})

Unit tests with mocks still pass if they only verify the publish call happened.

python
def test_escalate_account(mocker):
    mock_create = mocker.patch("app.support.create_ticket", return_value={"id": "T1"})
    mock_publish = mocker.patch("app.support.publish_event")

    result = escalate_account("A1", "fraud-review")

    assert result["id"] == "T1"
    mock_publish.assert_called_once()

Meanwhile, the downstream consumer routes tickets into the default queue because it relies on account_id for enrichment. Nobody notices until support SLAs degrade.

A stronger test would inspect the resulting state in the receiving system, not just the fact that an internal function was called.

Tools comparison: what each layer is good for

This is not an argument to throw away unit tests or mocks. It is an argument to stop pretending they cover workflow risk.

Here is the practical breakdown.

Unit tests

Best for:

pure business logic
edge cases in parsing, pricing, validation, transformation
fast feedback during development
regression coverage for deterministic code

Weak at:

redirects
timing
side effects
external provider behavior
async workflow completion
contract drift across services

Bottom line: essential, but insufficient for handoff reliability.

Integration tests with local dependencies

Best for:

repository-to-database behavior
API-layer wiring
serialization/deserialization
queue usage against local brokers
service internals

Weak at:

real third-party semantics
hosted auth/payment flows
production-like retries and webhooks
provider-side validation rules

Bottom line: better than unit tests for wiring, still often blind to real boundaries.

End-to-end browser tests

Best for:

user-visible workflows
redirects
session/auth issues
form completion and navigation
validating action completion from the user perspective

Weak at:

invisible downstream effects unless you add observability hooks
coverage breadth if used for every edge case
debugging when systems are flaky and evidence is poor

Bottom line: critical for workflow verification, especially at frontend-to-service seams.

Synthetic monitoring in production-like environments

Best for:

continuous validation of critical workflows
detecting expired credentials, broken callbacks, webhook failures
catching config drift after deployment

Weak at:

deep debugging without trace correlation
broad scenario exploration

Bottom line: one of the few ways to know the system still works after ship.

Contract testing

Best for:

explicit API/schema compatibility
consumer-provider alignment for internal services
detecting payload shape drift

Weak at:

behavior beyond schema
timing/order guarantees
full workflow semantics

Bottom line: useful guardrail, not full verification.

Observability and trace-based debugging

Best for:

understanding where workflow handoffs fail
correlating requests, jobs, callbacks, and external calls
reducing mean time to resolution

Weak at:

prevention unless paired with tests and monitors

Bottom line: if you cannot trace the workflow, you cannot reliably debug it.

Actionable practices for teams shipping AI-assisted code

If AI is writing more of your code, your testing strategy has to get more selective and more outcome-oriented.

1. Define critical workflows as first-class assets

Make a short list of business-critical user actions:

signup
login
checkout
password reset
contact support
cancel subscription
invite teammate
export data

For each one, define the minimum proof of success across boundaries. Not just “the button works,” but what state changes must exist in which systems.

Example:

Checkout proof might require:

browser reaches success page
order status is paid
payment provider status is succeeded
confirmation email event exists
duplicate charge did not occur

That becomes the basis for your highest-value automated verification.

2. Add seam-aware end-to-end tests, not just UI smoke tests

A smoke test that loads pages is not enough. Your tests should cross the actual seam and assert the resulting effect.

For example, use Playwright for the browser path and add test-only verification endpoints or database fixtures for outcome checks.

Be disciplined: do not make these tests assert every pixel. Make them prove the workflow completed.

3. Test against real providers where the risk is highest

Not for everything. But for auth, payments, email delivery, and webhook-heavy systems, a provider sandbox is often worth the complexity.

A fake provider gives clean feedback. A sandbox provider gives useful truth.

The teams with the best developer productivity are not the teams with the fastest green tests. They are the teams whose verification matches where failures actually occur.

4. Stop over-mocking side effects in critical paths

If a path exists mainly to coordinate side effects, mocking all those side effects removes the thing that matters.

Keep unit tests for logic. But add at least one higher-level test per critical path that exercises the real boundary or a much more faithful environment.

If your checkout tests never perform a real redirect or webhook loop, you do not have checkout verification.

5. Add observability to every workflow handoff

Every critical action should have a correlation ID that follows it across services:

browser request
backend request
queue message
webhook callback
provider response
email/ticket/CRM artifact

That enables real debugging when the workflow breaks.

Without correlation, teams stare at separate logs and argue about ownership. With correlation, you can answer the only question that matters: where did the handoff fail?

6. Make CI/CD report workflow evidence, not just test counts

A mature pipeline should surface signals like:

checkout E2E passed against payment sandbox
auth callback flow passed
support ticket creation verified in CRM sandbox
average webhook completion time
flaky handoff retries exceeded threshold

You do not need 500 brittle end-to-end tests. You need a handful of trusted workflow proofs for the paths that create revenue, trust, and support load.

A second workflow job in CI might look like this:

yaml
jobs:
  workflow-proof:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npm run test:workflow
        env:
          BASE_URL: ${{ secrets.STAGING_BASE_URL }}
          PAYMENT_SANDBOX_KEY: ${{ secrets.PAYMENT_SANDBOX_KEY }}
          EMAIL_TEST_INBOX: ${{ secrets.EMAIL_TEST_INBOX }}

This is slower than unit tests. Good. It is measuring something more real.

7. Review AI-generated changes for boundary assumptions

Code review for AI-assisted work should explicitly ask:

Did this change alter payload shape, metadata, IDs, or query params?
Did it change sync to async behavior?
Did it modify redirect destinations or callback handling?
Did it remove retries, waits, idempotency keys, or ordering guarantees?
Did it preserve side effects required by downstream systems?
Is there proof of the workflow, not just proof of the code path?

That review checklist catches a class of bugs that style comments never will.

8. Build “debuggability” into tests

When workflow tests fail, they need to produce evidence:

browser trace
screenshots
HAR/network logs
backend request logs
provider event IDs
webhook payloads
correlation IDs

Playwright already helps here. Use traces and videos sparingly but effectively.

ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
});

Debugging distributed workflow failures without evidence is slow and political. Evidence makes it engineering again.

9. Measure escaped seam failures separately

Track incidents caused by integration seams as their own category:

webhook race
redirect/callback bug
contract mismatch
missing side effect
async completion failure
environment/config drift

You will probably find these account for a disproportionate amount of user pain relative to the lines of code involved. That data justifies investment in workflow testing better than abstract quality arguments.

10. Accept that speed without proof is fake productivity

This is the broader point.

AI absolutely improves throughput. Teams can write more code, patch more tickets, and ship more PRs. But if verification remains code-centric while failures are workflow-centric, you are accelerating the rate at which plausible bugs reach production.

That is not developer productivity. It is deferred debugging.

Real developer productivity means engineers spend less time diagnosing why a “successful” change created support issues three systems away. The way to get there is not more ceremony. It is better proof.

Conclusion

AI agents are very good at fixing what they can see: local logic, nearby patterns, failing assertions, inconsistent interfaces. The problem is that many of the most expensive production failures do not live there.

They live at the handoff.

Between your app and the payment provider. Between signup and email. Between CRM and support routing. Between redirect and session. Between webhook arrival and durable state. Between “the code passed” and “the user actually succeeded.”

That is why traditional testing increasingly gives false confidence. Unit tests validate internals. Mocked dependencies validate assumptions. CI/CD validates that your isolated system is self-consistent. None of that is the same as proving a user action completed across real service boundaries.

So the standard has to change.

Do not ask only whether the code is correct. Ask whether the workflow is complete.

Do not settle for synthetic success signals. Require action-level proof.

Do not treat seam failures as edge cases. In distributed software, they are the main event.

The teams that adapt to AI-assisted development will not win by generating the most code. They will win by building the shortest path from change to trustworthy evidence. That means better debugging, smarter testing, more honest CI/CD, and a definition of quality grounded in user outcomes rather than green checkmarks.

Your agent may have fixed the code. The only question that matters is whether the handoff still works.