Your PR Is Green. The Checkout Is Broken: Why CI Needs Workflow-Level Verification for AI-Written Code

A pull request goes green. Type checks pass. Unit tests pass. Integration tests pass. The preview deploy looks healthy. The author is an AI coding agent, or a human moving fast with AI assistance, and the diff looks reasonable enough to merge.

Then someone tries to buy something.

The cart loads. The shipping form submits. The payment intent is created. But the final confirmation screen never appears because a client-side state transition stopped listening to the backend event that marks the order complete. Or a permissions refactor means the payment webhook succeeds but the order service can’t write the fulfillment record. Or the billing address validator normalizes a field differently than the tax service expects, so checkout only fails for a subset of countries.

Everything important is broken, and your CI/CD pipeline still gave you a green check.

That is the new failure mode of AI-assisted development. Not obviously bad code. Plausible code. Locally correct changes. Diffs that satisfy the tests you already had, preserve types, and fit the surrounding style, while silently breaking the actual user outcome.

The problem is not that AI writes uniquely terrible code. The problem is that AI makes it much cheaper to produce code that is syntactically valid, structurally convincing, and semantically incomplete. Traditional testing was already weak at proving that a product works end to end. AI scales that weakness. It lets teams generate many more changes that pass artifact-level validation without proving workflow-level correctness.

If your release process still treats green CI as a proxy for product reliability, you are optimizing for the wrong thing.

The failure isn’t in the code review. It’s in what CI is verifying

Most CI/CD systems validate artifacts, not outcomes.

They answer questions like:

• Does the code compile?
• Do the types line up?
• Do these functions return expected values for known inputs?
• Do service boundaries behave as mocked in tests?
• Does the app boot in a controlled environment?

Those are useful questions. None of them answer the one your business actually cares about:

• Can a user still complete the workflow that creates value?

That gap has always existed, but AI-assisted development makes it dangerous in a new way.

A human engineer often carries implicit product context while coding. They know checkout is fragile. They know signup involves analytics, email verification, fraud checks, role provisioning, and feature flags. They might still break something, but they often have a mental model of the journey.

An AI agent usually does not. It optimizes for the requested diff and the visible signals in the repository. If the tests are narrow, it learns narrow success criteria. If CI rewards passing snapshots and mock-heavy integration tests, it will produce code that passes snapshots and mock-heavy integration tests.

The result is a class of failures that look like this:

• The frontend form submits, but the next route never renders because a state machine transition changed.
• The backend returns 200, but the side effect the UI depends on never occurs.
• A permission change is correct for one service and wrong for the end-to-end journey.
• Idempotency logic works in isolation but duplicates orders under a real retry sequence.
• A background job is enqueued correctly, but the user-visible status never reflects completion.
• Two individually safe refactors combine into a broken multi-step flow.

This is not a testing edge case. This is the natural outcome of validating components while shipping workflows.

Why current approaches fail: unit tests, integration tests, QA, and the illusion of safety

Teams usually respond to production failures by saying they need more tests. Usually what they mean is more of the same kind of tests.

That is rarely enough.

Unit tests prove local behavior, not user success

Unit tests are excellent for constraining logic. They are fast, cheap, and valuable for debugging regressions in pure behavior. But unit tests are also the easiest tests for AI to satisfy without understanding the system.

An agent can update the implementation and the test together. It can preserve assertions while changing assumptions. It can lock in the wrong behavior if the wrong behavior still looks internally consistent.

A unit test can tell you that calculateTax() still returns expected values for a fixture. It cannot tell you that checkout still completes for a logged-in user with a saved address, a discount code, and a 3DS challenge.

That is not a criticism of unit tests. It is a reminder to stop asking them to prove what they cannot prove.

Integration tests often stop at service seams

A lot of so-called integration tests are really contract tests with mocks, test doubles, and controlled assumptions. Again: useful. But narrow.

You might verify that the billing service emits an invoice.paid event. You might verify that the order service can consume that event under an expected schema. You might even verify that the frontend handles a mocked “payment successful” response.

And still fail in reality because:

• the event arrives before a record exists,
• a feature flag changes the UI branch,
• the preview environment lacks one dependency,
• the user session expires between steps,
• a webhook retry creates a duplicate state,
• the frontend waits for a field the backend no longer returns.

Each piece can pass its own test while the workflow fails as a sequence.

Manual QA does not scale to the speed of AI-generated change

The old escape hatch was QA. If automated testing doesn’t prove the journey, a human can click through it.

The problem is throughput.

AI-assisted teams can produce dramatically more diffs, more refactors, more “small” changes, more dependency churn, and more broad edits across layers. Manual QA becomes the bottleneck instantly. Worse, it gets pushed to the end of the cycle, when context is lost and failures are expensive.

QA still matters, especially for exploratory testing and weird edge cases. But relying on humans to catch every workflow regression created by a machine that can generate 20 pull requests before lunch is a losing strategy.

Preview deployments are not verification

A preview deploy is only a place where verification could happen. On its own, it proves almost nothing.

Many teams confuse “the branch deployed successfully” with “the product works.” Those are different statements.

A healthy preview URL tells you the app booted. It does not tell you a user can sign up, confirm email, create a team, add a card, invite a teammate, assign permissions, and complete checkout.

Without workflow assertions, preview environments are just prettier logs.

The core insight: stop gating merges on test counts and start gating them on completed workflows

If the highest-value risk in your system lives in user journeys, your CI/CD process should evaluate those journeys directly.

This is the shift:

From:

• Did the code satisfy enough unit and integration checks?

To:

• Can a real user workflow complete in an ephemeral environment, with expected UI state transitions and expected cross-service side effects?

That means CI should run action-level verification for a small set of critical product journeys on every risky pull request, and on main before release.

Not hundreds of brittle browser tests. Not a giant end-to-end suite that nobody trusts. Not screenshot theater.

A targeted set of business-critical workflows.

Examples:

• Signup: create account, verify email, land in authenticated app state.
• Billing: upgrade plan, create payment method, receive entitlement.
• Permissions: invite user, assign role, confirm access changes in UI and backend.
• Checkout: add item, submit address, complete payment, confirm order record and user-visible success.
• Onboarding: connect integration, backfill initial data, render first successful dashboard state.

The point is not coverage in the abstract. The point is verifying the flows that matter most to revenue, activation, retention, and support load.

What workflow-level verification actually means

Workflow-level verification is not just “run Playwright.” It is a layered assertion model:

1. Reproduce the user journey in a realistic environment.
2. Assert key UI transitions, not just element existence.
3. Assert backend side effects across services.
4. Correlate steps with traceable identifiers.
5. Fail the PR if the workflow does not complete.

That last part matters. If it is non-blocking, it will be ignored under deadline pressure.

A good workflow check answers:

• Did the user reach the intended end state?
• Did each critical transition occur?
• Did the system produce the expected side effects?
• If not, where exactly did the flow break?

This is where debugging and developer productivity improve together. A failing workflow check should tell an engineer more than “timeout on selector.” It should narrow the failure to “payment intent created, webhook processed, order record missing, confirmation UI never advanced.”

That is actionable.

A concrete example: checkout that passes tests but fails users

Suppose you have a React frontend, Node API, payment provider webhook handler, and order service.

An AI agent refactors checkout state handling to reduce duplicated logic and updates a backend schema name for consistency.

Everything passes:

• TypeScript compiles.
• Unit tests around reducers pass.
• API integration tests pass with mocked responses.
• Webhook contract tests pass.
• Preview environment deploys.

But real checkout is broken because the frontend now waits for order.status === 'confirmed', while the backend writes completed. The order exists. Payment succeeded. The UI spinner never exits.

This kind of bug is common because each local component is reasonable. The mismatch only appears in the workflow.

Here is the kind of Playwright verification that catches it.

ts
import { test, expect } from '@playwright/test';

const runId = `ci-${Date.now()}`;

test('guest checkout completes and shows confirmation', async ({ page, request }) => {
  await page.goto(process.env.APP_URL!);

  await page.getByTestId('product-card-basic-plan').click();
  await page.getByTestId('add-to-cart').click();
  await page.getByTestId('start-checkout').click();

  await page.getByLabel('Email').fill(`buyer+${runId}@example.com`);
  await page.getByLabel('First name').fill('CI');
  await page.getByLabel('Last name').fill('Buyer');
  await page.getByLabel('Address').fill('1 Market St');
  await page.getByLabel('City').fill('San Francisco');
  await page.getByLabel('Postal code').fill('94105');

  await page.getByTestId('continue-to-payment').click();

  await expect(page.getByTestId('payment-step')).toBeVisible();
  await expect(page.getByTestId('checkout-progress')).toContainText('Payment');

  // Test payment method in sandbox
  await page.frameLocator('iframe[title="Secure payment input frame"]').getByPlaceholder('Card number').fill('4242424242424242');
  await page.getByLabel('Expiration date').fill('12/34');
  await page.getByLabel('CVC').fill('123');

  await page.getByTestId('submit-payment').click();

  await expect(page.getByTestId('order-processing')).toBeVisible();
  await expect(page.getByTestId('order-confirmation')).toBeVisible({ timeout: 30000 });
  await expect(page.getByTestId('order-confirmation')).toContainText('Thank you');

  // Verify side effects through internal verification endpoint
  const response = await request.get(`${process.env.APP_URL}/__verify/order?runId=${runId}`);
  expect(response.ok()).toBeTruthy();

  const payload = await response.json();
  expect(payload.paymentStatus).toBe('succeeded');
  expect(payload.orderStatus).toBe('confirmed');
  expect(payload.fulfillmentQueued).toBe(true);
});

This test does three important things:

• It executes the actual user flow.
• It validates UI state progression.
• It verifies side effects, not just frontend rendering.

That is the difference between “the button worked” and “checkout worked.”

Add observable verification points instead of trusting black-box timeouts

A common reason end-to-end testing becomes flaky is that teams treat the application like a black box and only wait for arbitrary UI elements.

A better approach is to instrument verification points specifically for CI environments.

For example, attach a runId to the workflow and expose a protected verification endpoint in ephemeral environments only.

ts
// Express example
app.get('/__verify/order', requireCiAuth, async (req, res) => {
  const runId = String(req.query.runId);

  const order = await orderRepository.findByRunId(runId);
  const payment = order ? await paymentRepository.findByOrderId(order.id) : null;
  const fulfillment = order ? await fulfillmentQueue.findLatest(order.id) : null;

  res.json({
    orderId: order?.id ?? null,
    orderStatus: order?.status ?? null,
    paymentStatus: payment?.status ?? null,
    fulfillmentQueued: Boolean(fulfillment),
  });
});

This is not an excuse to pollute production. Keep it gated to ephemeral environments, authenticated, and intentionally scoped. The goal is not to add test-only behavior. The goal is to make system outcomes observable enough for reliable debugging and testing.

If you can’t inspect whether the order was created, the entitlement granted, or the invitation accepted, your workflow checks will fail with vague symptoms and developers will stop trusting them.

Python example: verify cross-service side effects directly

In some stacks, the best place to assert side effects is outside the browser test. For example, after Playwright completes the visible journey, a Python verifier can query internal services or databases used in staging.

python
import os
import requests

APP_URL = os.environ["APP_URL"]
RUN_ID = os.environ["RUN_ID"]
CI_TOKEN = os.environ["CI_TOKEN"]


def verify_checkout_workflow():
    resp = requests.get(
        f"{APP_URL}/__verify/order",
        params={"runId": RUN_ID},
        headers={"Authorization": f"Bearer {CI_TOKEN}"},
        timeout=20,
    )
    resp.raise_for_status()
    data = resp.json()

    assert data["paymentStatus"] == "succeeded", f"unexpected payment status: {data}"
    assert data["orderStatus"] == "confirmed", f"unexpected order status: {data}"
    assert data["fulfillmentQueued"] is True, f"fulfillment not queued: {data}"


if __name__ == "__main__":
    verify_checkout_workflow()
    print("workflow verification passed")

Separating journey execution from state verification can be useful when multiple services need different credentials or APIs. The important point is still the same: the merge gate should depend on workflow completion, not just test process completion.

What to run in CI/CD: a practical pipeline design

You do not need to run every workflow on every pull request. That is how teams create slow, noisy pipelines nobody respects.

Instead, define a workflow verification tier.

A sane CI/CD structure looks like this:

• Fast checks on every PR: lint, types, unit tests, narrow integration tests.
• Workflow checks on risky PRs: browser-driven journeys plus side-effect verification in ephemeral environments.
• Full critical-path workflows on main or release branches.
• Nightly broader regression runs for lower-value journeys and edge cases.

Here is an example GitHub Actions workflow.

yaml
name: pr-validation

on:
  pull_request:

jobs:
  fast-checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test -- --runInBand

  deploy-preview:
    needs: fast-checks
    runs-on: ubuntu-latest
    outputs:
      app_url: ${{ steps.deploy.outputs.app_url }}
    steps:
      - uses: actions/checkout@v4
      - id: deploy
        run: |
          APP_URL=$(./scripts/deploy-preview.sh)
          echo "app_url=$APP_URL" >> $GITHUB_OUTPUT

  workflow-checkout:
    needs: deploy-preview
    runs-on: ubuntu-latest
    if: contains(join(github.event.pull_request.labels.*.name, ','), 'workflow-checkout') || contains(github.event.pull_request.changed_files, 'checkout')
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npm run test:workflow:checkout
        env:
          APP_URL: ${{ needs.deploy-preview.outputs.app_url }}
          CI_TOKEN: ${{ secrets.CI_TOKEN }}

  workflow-signup:
    needs: deploy-preview
    runs-on: ubuntu-latest
    if: contains(join(github.event.pull_request.labels.*.name, ','), 'workflow-signup')
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npm run test:workflow:signup
        env:
          APP_URL: ${{ needs.deploy-preview.outputs.app_url }}
          CI_TOKEN: ${{ secrets.CI_TOKEN }}

There is one caveat here: github.event.pull_request.changed_files is not actually a direct string of file paths in Actions expressions, so in practice you would compute changed areas in a prior step and output booleans. The important design idea is selective workflow gating based on risk.

A more realistic pattern uses a path classifier job.

yaml
jobs:
  classify:
    runs-on: ubuntu-latest
    outputs:
      checkout_changed: ${{ steps.filter.outputs.checkout }}
      billing_changed: ${{ steps.filter.outputs.billing }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            checkout:
              - 'apps/web/src/routes/checkout/**'
              - 'services/orders/**'
              - 'services/payments/**'
            billing:
              - 'services/billing/**'
              - 'apps/web/src/routes/settings/billing/**'

Then gate workflow jobs based on those outputs.

This is how you keep CI/CD efficient without pretending that all code changes are equally risky.

How to make workflow tests trustworthy instead of flaky theater

Engineers hate end-to-end tests for good reasons. Many suites are slow, brittle, and noisy. The answer is not to avoid workflow verification. The answer is to design it properly.

Practice 1: Test state transitions, not pixels

Do not write assertions like “button exists” or “page contains text.” Those are weak signals.

Write assertions like:

• cart count increments,
• payment step becomes active,
• submit button disables during processing,
• confirmation screen appears only after order state changes,
• entitlement banner appears after plan upgrade.

Those map to workflow semantics.

Practice 2: Verify side effects explicitly

A checkout flow is not complete because the browser navigated. It is complete because:

• payment succeeded,
• order was persisted,
• confirmation state was reached,
• downstream processing was triggered.

Pick 2–4 side effects per critical workflow and assert them directly.

Practice 3: Use ephemeral environments with realistic dependencies

If your workflow tests run against mocks, they will inherit mock confidence.

Use preview or ephemeral environments wired to sandbox versions of external providers and isolated backing services where possible. The environment should be close enough to reality that sequencing, auth, webhooks, and async jobs behave meaningfully.

Practice 4: Add testability hooks intentionally

Test IDs. Correlation IDs. Verification endpoints. Structured logs. Trace links. These are not hacks. They are observability for delivery.

Teams invest in runtime observability for production debugging but often neglect observability for CI debugging. That is a mistake. A merge-blocking workflow check needs first-class diagnostics.

Practice 5: Keep the critical set small

You do not need 300 workflow tests. You probably need 5 to 15.

Ask:

• What workflows create revenue?
• What workflows create user activation?
• What workflows create permissions or security risk?
• What workflows, when broken, generate executive escalations?

Start there.

Practice 6: Route checks by risk

Not every copy change needs checkout verification. But changes touching shared state management, auth, routing, billing, event schemas, or service orchestration often do.

Use path filters, labels, ownership metadata, or change impact analysis to decide which workflows run.

Practice 7: Fail with diagnostics, not just screenshots

When a workflow check fails, attach:

• browser trace,
• console logs,
• network logs,
• correlated backend events,
• final verification payload,
• step-by-step timeline.

This shortens debugging loops dramatically and improves developer productivity. Engineers will accept strict gates if failures are understandable.

AI-written code makes locally correct / globally wrong changes more common

This deserves emphasis because it is the heart of the problem.

AI is very good at producing code that looks complete at the scope of the prompt. It is much worse at preserving invisible product invariants unless those invariants are encoded into tests or system constraints.

A typical AI-generated change can be:

• syntactically valid,
• type-safe,
• stylistically consistent,
• accompanied by updated tests,
• still wrong for the actual workflow.

Why? Because workflows depend on hidden coupling:

• event timing,
• route transitions,
• long-lived session state,
• retries,
• feature flags,
• permissions propagation,
• async jobs,
• third-party behavior,
• cross-service schema meaning, not just shape.

An agent sees files. A user experiences sequences.

So if your CI only checks files, it will systematically miss sequence failures.

This is why AI-assisted development should push teams toward stronger workflow verification, not just more code generation and larger test suites.

Tools comparison: where different testing layers fit

No single tool solves this. The right approach is a stack.

Unit test frameworks: Jest, Vitest, Pytest

Best for:

• pure business logic,
• edge-case coverage,
• fast feedback,
• refactoring safety inside components.

Weak at:

• proving product workflows,
• catching state orchestration bugs,
• validating cross-service outcomes.

Verdict: necessary, not sufficient.

API/integration testing: Supertest, pytest + requests, contract testing tools

Best for:

• validating service behavior,
• checking schemas and contracts,
• catching narrow integration regressions.

Weak at:

• user-visible sequencing,
• client/server interaction timing,
• real browser state transitions.

Verdict: useful for service confidence, still not outcome verification.

Browser automation: Playwright, Cypress

Best for:

• critical user journeys,
• asserting workflow progression,
• detecting UI orchestration issues,
• collecting strong debugging artifacts.

Weak at:

• broad coverage if overused,
• maintainability without disciplined scope.

Verdict: the best entry point for workflow-level verification, especially Playwright because of tracing, network controls, and parallel CI support.

Synthetic monitoring and production probes

Best for:

• catching issues after deploy,
• verifying production health continuously,
• measuring real availability.

Weak at:

• preventing bad merges,
• providing fast PR feedback.

Verdict: complementary, not a substitute for pre-merge verification.

Visual regression tools

Best for:

• catching unintended visual changes,
• validating rendering consistency.

Weak at:

• proving checkout, signup, or permissions workflows actually complete.

Verdict: useful accessory, not a workflow gate.

Internal orchestration and observability tooling

Best for:

• correlation IDs,
• event tracing,
• side-effect verification,
• debugging failing workflow checks.

Weak at:

• direct user journey execution on their own.

Verdict: essential force multiplier for reliable workflow testing.

A better merge contract for modern teams

The merge contract for AI-assisted development should be explicit.

Not:

• all tests passed.

But:

• the diff passed local correctness checks,
• the affected critical workflows completed in an ephemeral environment,
• expected side effects occurred,
• debugging artifacts are available if verification failed.

That is a much stronger statement. It is also closer to what engineering leaders think they are getting from CI today, but often are not.

How to adopt this without boiling the ocean

You do not need a platform team and six months of migration to start.

Do this in order.

Step 1: Pick three critical workflows

Choose the flows where failure is expensive and obvious:

• signup,
• checkout,
• team invitation/permissions.

If you are a SaaS product, those three alone cover a huge amount of business risk.

Step 2: Define success as an outcome, not a page load

For each workflow, write down:

• start trigger,
• key transitions,
• final user-visible success state,
• required backend side effects.

This becomes the spec for verification.

Step 3: Add correlation and observability

Introduce a workflow run ID propagated through frontend requests, service logs, events, and persistence where feasible. This makes debugging dramatically easier.

Step 4: Build one Playwright journey per workflow

Do not start with edge cases. Start with the happy path that must never break.

If your happy path is flaky, your product or your environment likely has real determinism problems worth fixing anyway.

Step 5: Add one verifier endpoint or internal verification script

Assert the side effects that matter. Keep it simple.

Step 6: Gate risky PRs

Use path-based or ownership-based triggers so only relevant changes run workflow verification.

Step 7: Make failures easy to debug

Publish traces, logs, and verification payloads in CI artifacts.

Step 8: Expand carefully

Only after the critical set is stable should you add secondary journeys or edge cases.

This sequencing matters because teams often fail by trying to automate every path before they can reliably automate one.

The organizational benefit: better incentives for code review and AI usage

There is a second-order effect here that matters.

When workflow verification becomes the merge gate, engineers and AI agents alike are incentivized toward product correctness instead of artifact compliance.

That changes behavior.

Reviewers ask better questions:

• What user journey does this affect?
• Which workflow check covers it?
• What side effects should we verify?
• Are we changing workflow semantics or only implementation details?

AI usage also becomes safer because the system no longer assumes that plausible diffs are enough. The bar becomes: can the change survive contact with a real product journey?

That is the right standard.

Conclusion

The green pull request is becoming a weaker signal.

Not because unit tests, integration tests, or CI/CD pipelines are useless. They are still necessary. But they were designed to validate code artifacts, and AI-assisted development is increasing the volume of changes that can satisfy artifact-level checks while breaking outcome-level reality.

That is why teams keep seeing the same painful pattern: the PR is green, the deploy is clean, and the actual workflow is broken.

If you care about reliability, debugging speed, and developer productivity, stop treating test counts as a release strategy. Start identifying the handful of user journeys that matter most, replay them in ephemeral environments, assert their UI transitions and side effects, and gate merges on workflow completion.

Because your business does not run on passing tests.

It runs on users successfully finishing the flows that matter.