Your PR Passed. The User Journey Still Fails: Why AI Coding Needs Action-Level Verification in CI

A team merges a PR on Friday afternoon. CI is green. Unit tests passed. Type checks passed. Linting passed. Even the new integration tests passed because every external dependency was mocked exactly as expected. By evening, support gets the first ticket: new users can sign up, but they never receive access to the workspace they just created. Auth succeeds. The frontend redirects. The database row exists. But the background job that provisions the team never runs under the feature-flag combination used in production. Nothing in the PR checks caught it.

That is the new failure pattern of AI-assisted development.

The problem is not that AI writes bad code. The problem is that AI can produce a lot of plausible code, very quickly, along with equally plausible tests that verify local assumptions instead of real product behavior. The velocity goes up. The surface area of change goes up. The confidence signal from traditional testing often does not.

If your CI/CD pipeline still treats code-level assertions as the finish line, you are optimizing for the wrong layer. In modern products, reliability lives in action sequences: signup, invite, upgrade, import, retry, recover, cancel, restore. Users do not care whether your reducer updated state correctly or whether a mocked webhook returned 200 in a test harness. They care whether they can complete the job they came to do.

The next required layer in CI is action-level verification: test the sequence of actions a real user takes, against an environment that behaves enough like production to expose failures at handoff points between frontend state, auth, background jobs, feature flags, persistence, and third-party systems.

The new blind spot: AI increased code throughput, not behavioral confidence

AI coding tools changed the economics of shipping software. A single engineer can now generate a migration, API handler, background worker, React component, and test scaffolding in one sitting. That is useful. It is also dangerous in a very specific way.

Traditional engineering bottlenecks were often code production bottlenecks. Now the bottleneck is validation. Teams can change more files, more layers, and more assumptions per PR than their test strategy was designed to absorb.

That matters because most existing testing stacks were built for a world where changes were smaller and more manually reasoned about. In that world, a healthy mix of unit tests, some integration tests, and a QA pass might be enough. In the AI-assisted world, a single PR can subtly alter:

• frontend state transitions
• API contracts
• retry behavior
• background job triggers
• flag-dependent code paths
• auth session handling
• analytics side effects
• idempotency behavior
• third-party payload formats

And all of it can still look clean in review.

The result is a dangerous asymmetry: code generation got faster, but end-to-end behavioral verification did not. So teams merge changes that are syntactically sound, locally tested, and operationally wrong.

That is why debugging production failures increasingly feels absurd. You inspect a broken journey and find that every individual layer “worked” according to its own tests. The frontend did dispatch the action. The API did return success. The worker did pass unit tests. The integration mock did behave correctly. Yet the user journey failed.

Not because one function was obviously broken. Because the system behavior across boundaries was never verified as a whole.

Why PR checks miss the failures users actually see

Most PR pipelines are great at verifying artifacts of implementation. They are much worse at verifying completion of intent.

That distinction matters.

A user intent is: “I signed up and invited my team.”

An implementation artifact is: “The invite button rendered and a POST request returned 201.”

Those are not the same thing.

Here are the common reasons PR checks miss real workflow failures.

Unit tests validate logic in isolation

Unit tests are still valuable. They catch regressions cheaply, help with debugging, and make refactoring safer. But they are structurally incapable of proving a workflow works across system boundaries.

A unit test can tell you that a function builds the correct invite payload. It cannot tell you that:

• the auth token used by the browser is accepted by the downstream service
• the invite event is blocked by a feature flag rollout rule
• the background job consumer is listening to the right queue in CI or staging
• the email provider accepted the template version currently configured
• the invited user lands in the correct post-acceptance state

Yet these are exactly the failures that show up in production.

Integration tests often mock away the risk

A lot of so-called integration tests are “integration-shaped unit tests.” They spin up part of the app, then mock the unstable or expensive edges. That is understandable. Real integrations are slow and flaky if implemented badly.

But the cost of heavy mocking is false confidence.

If your import flow depends on:

• uploading a file from the browser
• creating a record in your app
• enqueuing a job
• polling job status
• handling provider rate limiting
• updating billing usage
• rendering a completion state

then mocking the provider, queue, and polling layer may produce a stable test suite that verifies almost nothing meaningful.

The dangerous part is not that mocks are inaccurate in theory. It is that they drift silently. The real API starts returning a nullable field. Your retry semantics change. The feature flag gates the job path for enterprise accounts only. Your mocked tests stay green because they encode an idealized version of the system.

CI/CD pipelines reward speed over behavioral realism

This is a pipeline design issue. Most teams optimize CI for throughput, which is reasonable. They run checks that are:

• deterministic
• parallelizable
• cheap to maintain
• easy to cache

That usually means static analysis, unit tests, and mocked integration tests. It does not mean environment-aware verification of user workflows with real services or realistic substitutes.

So CI becomes an implementation correctness filter, not a user outcome filter.

That is a useful layer. It is not a sufficient one.

Manual QA cannot keep up with change volume

The fallback for many organizations is still some form of QA, whether dedicated testers or ad hoc product checks. But manual verification breaks under modern release velocity.

AI-assisted development makes this worse. More small changes ship more often. More hidden assumptions shift. More combinations of role, plan, flag, region, and integration become relevant.

No human QA process will reliably cover:

• signup under a new experiment bucket
• import under a legacy account tier
• retry after a transient third-party 429
• upgrade with a preexisting unpaid invoice
• passwordless auth with expired magic links
• invite acceptance when SSO enforcement flips mid-flow

Manual QA remains useful for exploratory testing and UX judgment. It is not a scalable guardrail for workflow correctness between merge and deploy.

The core insight: verify user intent as action sequences, not code paths

The right mental model is simple: stop asking only whether code paths executed, and start asking whether user intents completed.

That means your CI pipeline needs tests that exercise business actions end to end, through the interfaces users and systems actually use.

Examples of action-level verification:

• A visitor signs up, creates a workspace, and lands in a usable initial state.
• An admin invites a teammate, the teammate accepts, and permissions are correct.
• A customer upgrades plans, billing state updates, and gated features unlock.
• A user imports data, a background job completes, and records become queryable.
• A failed sync is retried, duplicate side effects are avoided, and the account recovers.
• A user toggles a feature that requires re-auth, completes the auth handoff, and returns to the expected state.

These are not just end-to-end UI tests in the shallow sense. They are business outcome verifications.

The difference is important. A brittle UI test checks whether a button exists or a text node matches. An action-level test checks whether the sequence produced the intended state transition in the product.

That may include browser automation, but it also includes backend assertions, job completion checks, network observability, and state seeding.

What action-level verification looks like in practice

A useful action-level test has a few traits:

1. It begins from a realistic starting state.
2. It executes user actions through the browser or public interfaces.
3. It allows asynchronous system behavior to complete.
4. It asserts business outcomes, not just UI cosmetics.
5. It runs in an environment where config, flags, and integrations resemble production enough to matter.

Let’s make that concrete.

Example: signup and workspace provisioning with Playwright

A common failure pattern is successful authentication followed by broken provisioning. The user is technically logged in, but their account or workspace is not usable.

A weak test might do this:

js
import { test, expect } from '@playwright/test';

test('signup shows welcome screen', async ({ page }) => {
  await page.goto('/signup');
  await page.fill('[name=email]', 'new@example.com');
  await page.fill('[name=password]', 'Password123!');
  await page.click('button[type=submit]');
  await expect(page.getByText('Welcome')).toBeVisible();
});

This proves very little. The redirect may succeed even if provisioning failed.

A better action-level test verifies the actual outcome:

js
import { test, expect } from '@playwright/test';

async function waitForWorkspaceReady(page) {
  await expect.poll(async () => {
    const response = await page.request.get('/api/me/workspace');
    if (!response.ok()) return { ready: false };
    return response.json();
  }, {
    timeout: 30000,
    intervals: [1000, 2000, 5000]
  }).toMatchObject({
    ready: true,
    role: 'owner'
  });
}

test('signup provisions a usable workspace', async ({ page }) => {
  const email = `user-${Date.now()}@example.test`;

  await page.goto('/signup');
  await page.fill('[name=email]', email);
  await page.fill('[name=password]', 'Password123!');
  await page.click('button[type=submit]');

  await page.waitForURL('**/getting-started');
  await waitForWorkspaceReady(page);

  const workspaceResponse = await page.request.get('/api/me/workspace');
  const workspace = await workspaceResponse.json();

  expect(workspace.ready).toBe(true);
  expect(workspace.plan).toBe('free');
  expect(workspace.members).toHaveLength(1);

  await page.goto('/app/projects/new');
  await page.fill('[name=projectName]', 'First Project');
  await page.click('button:text("Create project")');

  const projectsResponse = await page.request.get('/api/projects');
  const projects = await projectsResponse.json();
  expect(projects.some(p => p.name === 'First Project')).toBe(true);
});

This test does more than click through a screen. It verifies that the user can continue into a meaningful next action. That is the real standard: can the user proceed?

Seed backend state instead of forcing the UI to do all setup

One reason end-to-end suites become slow and brittle is that teams insist every precondition be created through the UI. That is unnecessary.

Action-level verification is not about romantic purity. It is about verifying critical transitions realistically and efficiently.

If you need an account with a feature flag enabled, seeded billing status, and an existing import record, set that state directly through factories or seed APIs.

Here is a simple Node helper for seeding state:

js
export async function createSeededAccount(request, overrides = {}) {
  const response = await request.post('/test-support/seed/account', {
    data: {
      plan: 'pro',
      flags: ['new_import_flow'],
      billingStatus: 'active',
      ...overrides,
    },
  });

  if (!response.ok()) {
    throw new Error('Failed to seed account');
  }

  return response.json();
}

And in Playwright:

js
import { test, expect } from '@playwright/test';
import { createSeededAccount } from './helpers/seeds';

test('pro user can import and search records', async ({ page, request }) => {
  const account = await createSeededAccount(request, {
    flags: ['new_import_flow', 'search_v2'],
  });

  await page.goto(`/test-login/${account.loginToken}`);
  await page.goto('/app/import');

  await page.setInputFiles('input[type=file]', 'fixtures/customers.csv');
  await page.click('button:text("Start import")');

  await expect.poll(async () => {
    const res = await page.request.get(`/api/imports/${account.latestImportId}`);
    return res.json();
  }, { timeout: 60000 }).toMatchObject({ status: 'completed' });

  await page.goto('/app/search');
  await page.fill('[placeholder="Search customers"]', 'Acme');
  await page.press('[placeholder="Search customers"]', 'Enter');

  await expect(page.getByText('Acme Corp')).toBeVisible();
});

That is the right compromise. Seed setup. Exercise the user-critical path through the actual product. Assert the business result.

Assert outcomes in the backend, not only the DOM

DOM assertions are often the weakest part of browser-based testing. They are easy to write and easy to overvalue.

If the business action is “upgrade succeeded,” then the right assertions may be:

• subscription tier changed in your system
• provider customer record is linked
• entitlements recalculated
• premium feature becomes available
• invoice state is correct

The UI matters, but it is often just one manifestation of the outcome.

Here is a Python example using Playwright and API assertions after a billing upgrade:

python
from playwright.sync_api import sync_playwright, expect
import requests
import time

BASE_URL = "http://localhost:3000"
API_URL = "http://localhost:3000/api"

def wait_for_entitlement(session, token, feature_key, timeout=30):
    deadline = time.time() + timeout
    while time.time() < deadline:
        r = session.get(
            f"{API_URL}/me/entitlements",
            headers={"Authorization": f"Bearer {token}"}
        )
        r.raise_for_status()
        data = r.json()
        if feature_key in data.get("enabled", []):
            return True
        time.sleep(2)
    raise TimeoutError("Entitlement did not become active")

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()

    seed = requests.post(f"{BASE_URL}/test-support/seed/account", json={
        "plan": "free",
        "billing_status": "none"
    }).json()

    login_token = seed["loginToken"]
    api_token = seed["apiToken"]

    page.goto(f"{BASE_URL}/test-login/{login_token}")
    page.goto(f"{BASE_URL}/app/billing")

    page.click("text=Upgrade to Pro")
    page.fill("[name=cardNumber]", "4242424242424242")
    page.fill("[name=expiry]", "12/30")
    page.fill("[name=cvc]", "123")
    page.click("button:has-text('Confirm upgrade')")

    expect(page.get_by_text("Plan: Pro")).to_be_visible()

    session = requests.Session()
    wait_for_entitlement(session, api_token, "advanced_exports")

    r = session.get(
        f"{API_URL}/me/subscription",
        headers={"Authorization": f"Bearer {api_token}"}
    )
    r.raise_for_status()
    subscription = r.json()

    assert subscription["plan"] == "pro"
    assert subscription["status"] == "active"

    browser.close()

This style of testing is far better for debugging too. When it fails, you can narrow the break: UI submission, billing handoff, webhook processing, entitlement update, or post-upgrade state rendering.

Environment-aware automation matters more than perfect mocks

There is no requirement that every CI run hit every real third-party service. That would be expensive and often unstable. But your action-level layer must be environment-aware enough to catch real integration behavior.

That usually means choosing one of three modes per dependency:

1. Real service in non-production mode: best for critical providers that offer stable test environments, like payments.
2. Contract-faithful simulator: acceptable if maintained against observed real behavior and versioned with the app.
3. Internal stub with explicit limits: only for low-risk dependencies where the business outcome does not materially depend on the provider edge cases.

The mistake is treating all dependencies the same and mocking everything into idealized success.

For example:

• Payments: use the real provider’s test mode whenever possible.
• Email delivery: maybe stub actual send, but verify template rendering, enqueueing, and status recording.
• File import parsing: use real parsing stack and real fixture files.
• Search indexing: run the actual indexer in CI for critical paths, even if on reduced dataset size.
• Feature flags: load real evaluation rules or a close equivalent, not hardcoded booleans scattered in tests.

CI configuration: where workflow verification belongs

A practical pipeline usually has multiple layers.

• Fast PR checks: lint, typecheck, unit tests, narrow integration tests.
• Action-level verification: critical user workflows against a realistic environment.
• Deployment gate: run a focused suite on release candidates or pre-deploy environments.
• Post-deploy smoke: validate top workflows in production-safe mode.

The mistake is expecting the first layer to provide confidence that only the second and third layers can provide.

Here is a sample GitHub Actions setup:

yaml
name: ci

on:
  pull_request:
  push:
    branches: [main]

jobs:
  fast-checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test:unit
      - run: npm run test:integration

  workflow-verification:
    runs-on: ubuntu-latest
    needs: fast-checks
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
      redis:
        image: redis:7
        ports:
          - 6379:6379
    env:
      APP_ENV: ci
      FEATURE_FLAGS_SOURCE: seeded
      STRIPE_MODE: test
      DATABASE_URL: postgresql://postgres:postgres@localhost:5432/app
      REDIS_URL: redis://localhost:6379
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run db:migrate
      - run: npm run start:ci &
      - run: npm run worker:ci &
      - run: npx playwright install --with-deps
      - run: npm run test:workflows

  predeploy-gate:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    needs: workflow-verification
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy-preview.sh
      - run: ./scripts/run-release-workflows.sh

The details will vary. The key is architectural: action-level verification is a first-class stage, not a nice-to-have afterthought.

What to test first: focus on irreversible or high-support workflows

Teams often stall because “test all workflows” is overwhelming. Don’t start there.

Start with journeys that are:

• revenue-related
• account-access related
• import/export related
• multi-step and async
• hard to manually debug
• common sources of support tickets
• vulnerable to feature flag or role interactions

A strong first set is usually:

1. Signup and first-run provisioning
2. Invite and accept with correct permissions
3. Upgrade or checkout with entitlement activation
4. Import and completion visibility
5. Retry or recovery after a transient failure
6. Password reset or magic link login
7. Feature access under role/plan constraints

If these flows are stable, you eliminate a huge class of embarrassing failures.

Tools comparison: what each layer is good for

Here is the blunt version.

Unit tests

Best for:

• pure logic
• edge case enumeration
• fast debugging
• regression protection during refactors

Bad for:

• proving user journeys work
• proving integrations behave correctly
• proving async orchestration completes

Mock-heavy integration tests

Best for:

• contract checks inside your codebase
• component interactions under controlled conditions
• error-path simulation that is hard to trigger externally

Bad for:

• validating real-world provider behavior
• catching config drift
• catching environment-specific failures

Browser automation with Playwright or similar

Best for:

• user-visible workflows
• auth and state transitions
• frontend-backend handoffs
• debugging sequence failures with traces and screenshots

Bad for:

• replacing all lower-level tests
• exhaustive business logic coverage
• workflows that are poorly isolated or require unstable test data unless you invest in seeding

Manual QA

Best for:

• exploratory testing
• UX feedback
• release confidence on novel features

Bad for:

• repeatable CI gates
• high-frequency regression detection
• broad matrix coverage at speed

Observability and production monitoring

Best for:

• detecting real-world failures
• measuring user impact
• debugging environment-specific issues

Bad for:

• preventing broken journeys before deploy

No single layer solves reliability. But if your stack lacks action-level verification, there is a structural hole right where modern failures live.

Actionable practices for teams shipping AI-assisted code

Here is the practical playbook.

1. Define workflows as business contracts

Write down your top user journeys in business terms, not implementation terms.

Bad:

• clicking submit calls POST /api/workspaces

Better:

• a new user can create an account and successfully start using the product within 60 seconds

This changes what you assert and what counts as failure.

2. Add seed APIs or test factories

If you do not invest in controlled state setup, your workflow tests will become slow and flaky. Build internal-only endpoints, fixtures, or direct factory layers for creating accounts, flags, subscriptions, and async jobs.

This is not cheating. It is test infrastructure.

3. Wait on state transitions, not arbitrary sleeps

Most flaky end-to-end tests are synchronization bugs. Replace sleeps with polling on observable state:

• job status endpoints
• database-backed API state
• entitlements
• queue drain indicators
• page URL or app readiness markers

This improves both reliability and debugging.

4. Assert business outcomes across boundaries

For every workflow test, ask: what persisted state or capability proves success?

Examples:

• workspace exists and user role is owner
• invitee membership is active
• subscription status is active and premium feature accessible
• import produced searchable records
• retry did not create duplicates

5. Run workflow checks in an environment close enough to production

You do not need a perfect replica. You do need the classes of behavior that often break:

• auth configuration
• worker execution
• real flag evaluation
• persistence
• webhook handling
• realistic network boundaries

6. Make failures debuggable by default

Collect traces, screenshots, console logs, network logs, and app-side correlation IDs. Action-level testing is only useful if failed runs point engineers toward the break quickly.

Playwright already helps a lot here. Use traces aggressively.

7. Gate deploys on a focused workflow suite, not an enormous flaky pyramid

Do not build a 600-test UI suite and call it strategy. Build a small, sharp set of high-value workflow checks that are treated as release-critical.

Ten trustworthy workflow tests are worth more than two hundred brittle screenshot checks.

8. Track workflow coverage as a reliability metric

If AI increases change velocity, your testing strategy needs a matching metric. Track how many critical user intents have automated verification between merge and deploy.

That is a better signal than raw unit test count.

A note on debugging: action-level verification shortens the blast radius

There is a secondary benefit here beyond testing quality: better debugging.

When a production issue happens today, teams often start with logs and guesses because no one has a faithful automated reproduction of the business workflow. But if you already have action-level tests, you have executable descriptions of the most important journeys.

That means when “invite acceptance fails for enterprise SSO users under the new billing rollout,” you can:

• reproduce the exact flow
• seed the relevant account state
• inspect browser and API traces
• isolate whether the fault is auth, role mapping, worker execution, or flag evaluation

This is a major developer productivity gain. The same artifacts that catch regressions pre-deploy also reduce time-to-resolution after incidents.

That matters in an AI-heavy workflow because the codebase changes more often, by more people, with more machine-generated glue. Clear reproduction paths become much more valuable.

The uncomfortable truth: green CI is often a local maximum of confidence theater

A lot of teams know this already, even if they do not say it directly. They have green pipelines and recurring workflow failures. They trust CI enough to merge and not enough to relax.

That is confidence theater.

The issue is not that unit tests are bad or that CI/CD is broken. The issue is category error. We ask implementation-level tests to answer product-level questions.

“Did this function behave?” is not the same question as “Can the user finish the task?”

AI-assisted development widens that gap because the amount of changed implementation per PR grows faster than the amount of human behavioral scrutiny. So the old signals degrade. A PR can look polished, complete, and well-tested while still breaking the only thing that matters: the journey.

If you are responsible for engineering quality, platform reliability, or developer productivity, this is the shift to internalize. The missing layer is not more generated tests. It is a better definition of done.

Done means the user action works.

Not the function. Not the component. Not the mocked integration.

The action.

Conclusion

AI has made it easier to produce code, tests, and plausible implementation correctness. It has not made it easier to guarantee that a user can sign up, invite, upgrade, import, recover, and continue using the product without hitting a broken handoff.

That is why action-level verification belongs in CI now.

Not as a replacement for unit tests. Not as a giant brittle browser suite. As a focused, environment-aware layer that verifies critical user intents across the system boundaries where modern software actually fails.

If your PR pipeline ends at code-level assertions, it is giving you at best partial confidence. In many teams, it is giving you false confidence.

The fix is straightforward, even if it takes discipline: identify the workflows that matter, seed realistic state, execute real action sequences, wait for asynchronous behavior, and assert business outcomes.

Your PR passed. Good.

Now prove the user journey does too.