Your Agent Opened the PR. Who Proved the Onboarding Flow Still Works?

A bot opens a pull request at 2:13 AM. The diff is tidy. TypeScript passes. Unit tests are green. CI/CD reports success in eight minutes.

By 8:40 AM, your conversion rate is down 18%.

Nothing is obviously “broken” in the code review sense. The forms still render. The API still returns 200s. The auth package upgrade looked routine. The copy change was harmless. The refactor improved readability. The dependency bump fixed a security advisory.

And yet new users can no longer complete onboarding because the “Continue” button is below the fold on smaller laptop screens after a layout shift triggered by a validation message that only appears when a workspace name collides with an existing slug.

That is the new reliability problem.

As AI coding agents generate more code, more refactors, and more “safe” maintenance changes, engineering teams are quietly losing the old relationship between passing tests and working software. The issue is not that agents write uniquely bad code. The issue is that they can produce locally correct changes faster than humans can reason about product-level consequences. Reviewers see plausible diffs. CI sees passing checks. Users hit dead workflows.

The question is no longer “Did the code compile?” or even “Did the tests pass?”

The real question is: who proved the onboarding flow still works?

The failure mode has changed

Traditional software failures were often implementation failures. Null pointer exceptions. Missing imports. Broken SQL migrations. Obvious crashes. Bad merge conflicts.

Those still exist, but modern teams have become reasonably good at catching them. Linters, type systems, contract tests, unit tests, staging environments, feature flags, and observability all reduce the chance that completely invalid code reaches production.

What’s growing faster now is a different class of failure: workflow regression hidden inside technically correct changes.

Examples:

• Signup succeeds, but the welcome email link expires because a background job queue name changed in one environment.
• Checkout renders, but the coupon field steals focus on mobile and breaks the payment submit sequence.
• Password reset works for email/password users, but not for users who originally signed up with Google and later set a password.
• Team invite links are generated correctly, but accepted users land in a default workspace instead of the invited workspace.
• A dependency upgrade changes browser autofill behavior, causing a two-factor auth field to submit incomplete values.
• A caching optimization delays profile creation by 500 ms, and the next page assumes the profile already exists.

No single line in the diff screams “production outage.” In fact, many of these PRs would look responsible, even high quality.

That’s why the old confidence model is breaking.

When AI increases change volume, review quality does not scale linearly. Humans do not get 5x better at predicting downstream behavior because the author is now a coding agent. If anything, reviewers become more vulnerable to plausibility bias: the code looks coherent, the patterns match the codebase, the checks are green, so the change feels safe.

It is not safe. It is merely unproven.

Why code review is a poor proxy for user reality

Code review is good at catching certain kinds of issues:

• obvious logic errors
• bad abstractions
• style drift
• security footguns
• missing edge-case handling when reviewers know where to look

Code review is terrible at validating lived product behavior across multi-step journeys.

A reviewer does not mentally execute your full onboarding flow across viewport sizes, auth states, race conditions, email confirmations, redirects, third-party integrations, asynchronous UI updates, and persisted browser state. They inspect symbols and intent. Users experience sequences.

That distinction matters.

An onboarding flow is not a function. It is a distributed interaction between frontend, backend, browser, network, state management, timing, and external systems. The thing users care about is not whether each piece seems reasonable in isolation. They care whether they can go from “land on site” to “invite my team” without friction.

Code review mostly inspects static artifacts. Product reliability emerges from dynamic behavior.

AI agents make this mismatch worse because they are good at producing coherent local changes. They can update a component, adjust a validation schema, and wire a new API response shape with impressive consistency. But they do not inherently provide evidence that the user journey still completes end to end. And unless your system requires that evidence, the PR can merge based on confidence theater.

Why green CI/CD can still mean a broken product

Many teams say, “That’s what CI is for.” But in a lot of organizations, CI/CD is mostly a syntax-and-regression gate for code-level correctness, not workflow-level correctness.

A typical green pipeline means some combination of:

• dependencies installed
• build succeeded
• lint passed
• unit tests passed
• integration tests passed
• maybe a smoke deploy worked

Useful? Absolutely.

Sufficient? Not even close.

Here’s the hard truth: a green build often means your code still satisfies the assumptions encoded in your test suite. It does not mean your software still satisfies the assumptions encoded in your product.

That gap exists because most test suites overrepresent low-level assertions and underrepresent user outcomes.

For example:

• A unit test proves createWorkspace() returns the right shape.
• An integration test proves /api/invites/accept responds correctly.
• A component test proves the form renders the error state.

None of those prove a brand-new user can sign up, verify email, create a workspace, invite a teammate, and both users land in the right place.

CI/CD becomes dangerous when teams treat it as a verdict instead of a mechanism. A pipeline is only as meaningful as the checks inside it. If the checks don’t replay reality, the green badge is just well-structured optimism.

Why unit tests, integration tests, and manual QA all miss this differently

This is not an argument against unit testing. Unit tests still matter. Integration tests still matter. Manual QA still matters.

The problem is what happens when teams confuse those methods for complete evidence.

Unit tests optimize for local correctness

Unit tests are excellent for verifying isolated logic:

• validation rules
• state transitions
• utility functions
• pricing calculations
• serializer behavior
• permission checks

They are fast, deterministic, and helpful during debugging. They improve developer productivity because they narrow the blast radius of simple mistakes.

But they encode a model of the system where boundaries are mocked, timing is simplified, and user behavior is abstracted away. That is exactly why they are maintainable.

It is also why they cannot carry the burden of proving critical workflows.

A great unit test suite can coexist with a broken signup flow.

Integration tests optimize for contracts, not journeys

Integration tests help validate that subsystems interact correctly. They catch request/response mismatches, database assumptions, service boundaries, and event sequencing better than unit tests.

But many integration tests still stop short of what matters in production. They verify API-level correctness while skipping the browser, the redirect chain, the rendering layer, the session store, the real auth callback, the email link, the loading state, or the second user role.

That means they often prove the parts are compatible without proving the experience is successful.

Manual QA does not scale to agent-driven change volume

Manual QA can catch exactly the kinds of weird product regressions automation misses, especially around visual issues, odd browser behavior, copy problems, and role-based edge cases.

But manual QA is expensive, inconsistent, and usually too slow for the volume and velocity of AI-assisted development.

If agents can open PRs for refactors, framework upgrades, generated CRUD screens, flaky test fixes, localization changes, and dependency updates all day long, a human testing pass on every meaningful workflow becomes the bottleneck. Worse, it becomes selective. Teams test what looks risky and skip what looks routine.

That is how “safe” changes break onboarding.

The core insight: test actions, not just code paths

If your product depends on critical workflows, then reliability has to be defined at the workflow level.

That means your CI/CD system should not only ask:

• Did the code build?
• Did the unit tests pass?
• Did the APIs behave as expected?

It should also ask:

• Can a new user complete onboarding?
• Can an invited user join the correct workspace?
• Can a customer finish checkout?
• Can a locked-out user recover access?

These are action-level questions. They are phrased in terms of user outcomes, not internal implementation.

That shift sounds simple, but it changes how teams think about testing.

Instead of treating end-to-end verification as a nice-to-have safety net, you treat a small number of critical workflows as release criteria. Not every click path. Not every permutation. The flows that materially define whether the product works.

For a SaaS product, those are usually things like:

• signup and first-run onboarding
• login and MFA
• password reset
• checkout and subscription activation
• team invite and seat provisioning
• permissions and role changes
• key integration setup paths

If a coding agent changes anything that can affect these paths directly or indirectly, your system should replay them automatically and attach evidence to the PR.

Evidence, not assumption.

What action-level verification looks like in practice

Action-level verification means using browser-driven or API-assisted end-to-end tests to simulate the actual user journey in an ephemeral or isolated environment.

This usually includes:

• real browser automation
• seeded or disposable test data
• deterministic environment setup
• assertions on visible outcomes
• traces, screenshots, and logs for debugging
• CI execution on every meaningful change

Playwright is a strong fit here because it combines modern browser automation with useful debugging artifacts and stable primitives for waiting on real UI behavior.

Here is a simplified example of an onboarding workflow test in JavaScript.

js
import { test, expect } from '@playwright/test';

function uniqueEmail() {
  return `new-user-${Date.now()}@example.test`;
}

test('new user can sign up, create workspace, and reach dashboard', async ({ page }) => {
  const email = uniqueEmail();

  await page.goto(process.env.APP_URL);
  await page.getByRole('link', { name: /sign up/i }).click();

  await page.getByLabel(/work email/i).fill(email);
  await page.getByLabel(/password/i).fill('StrongPass123!');
  await page.getByRole('button', { name: /create account/i }).click();

  await expect(page.getByText(/check your email/i)).toBeVisible();

  const verificationLink = await fetchVerificationLink(email);
  await page.goto(verificationLink);

  await page.getByLabel(/workspace name/i).fill('Acme QA Workspace');
  await page.getByRole('button', { name: /continue/i }).click();

  await expect(page).toHaveURL(/.*dashboard/);
  await expect(page.getByRole('heading', { name: /welcome/i })).toBeVisible();
});

async function fetchVerificationLink(email) {
  const response = await fetch(`${process.env.TEST_HELPER_URL}/email-link?email=${email}`);
  const data = await response.json();
  return data.link;
}

This test is not trying to inspect implementation details. It is proving an outcome: a new user can reach the dashboard through the intended flow.

Now extend that to a team invite flow.

js
import { test, expect } from '@playwright/test';

test('invited user joins the correct workspace', async ({ browser }) => {
  const ownerContext = await browser.newContext();
  const inviteeContext = await browser.newContext();

  const ownerPage = await ownerContext.newPage();
  const inviteePage = await inviteeContext.newPage();

  const inviteeEmail = `invitee-${Date.now()}@example.test`;

  await loginAsSeedUser(ownerPage, 'owner@example.test');
  await ownerPage.goto(`${process.env.APP_URL}/settings/members`);
  await ownerPage.getByLabel(/email/i).fill(inviteeEmail);
  await ownerPage.getByRole('button', { name: /send invite/i }).click();

  await expect(ownerPage.getByText(/invite sent/i)).toBeVisible();

  const inviteLink = await fetchInviteLink(inviteeEmail);
  await inviteePage.goto(inviteLink);
  await inviteePage.getByLabel(/full name/i).fill('Invited User');
  await inviteePage.getByLabel(/password/i).fill('StrongPass123!');
  await inviteePage.getByRole('button', { name: /accept invite/i }).click();

  await expect(inviteePage).toHaveURL(/.*workspace/);
  await expect(inviteePage.getByText(/acme inc/i)).toBeVisible();
});

That kind of test catches failures that no reviewer can infer reliably from a diff.

Using Python for supporting test setup and diagnostics

A lot of real workflow testing needs support scripts: seeding data, polling mailboxes, generating signed links, cleaning test accounts, or validating side effects.

Python is often a practical choice for these helper tasks.

python
import os
import time
import requests

TEST_HELPER_URL = os.environ["TEST_HELPER_URL"]

def wait_for_email_link(email: str, timeout: int = 30) -> str:
    deadline = time.time() + timeout
    while time.time() < deadline:
        response = requests.get(f"{TEST_HELPER_URL}/email-link", params={"email": email})
        response.raise_for_status()
        payload = response.json()
        if payload.get("link"):
            return payload["link"]
        time.sleep(1)
    raise TimeoutError(f"No verification link found for {email}")


def reset_test_account(email: str) -> None:
    response = requests.post(f"{TEST_HELPER_URL}/reset-user", json={"email": email})
    response.raise_for_status()

These utilities matter because reliable end-to-end testing is mostly about environment control. When teams say browser tests are flaky, what they often mean is that the environment is nondeterministic, the setup is brittle, or the assertions are poorly synchronized.

That is a debugging and systems design problem, not proof that workflow testing is inherently doomed.

What to run in CI/CD on every PR

Not every browser test belongs on every pull request. But every critical workflow should have a fast, stable “proof test” version that can run in CI/CD as a merge gate.

A practical strategy looks like this:

1. Keep unit and integration suites for broad regression coverage.
2. Define 5–15 mission-critical user journeys.
3. Build one stable end-to-end test for each journey.
4. Run those proof tests on every PR that can affect product behavior.
5. Run expanded cross-browser or edge-case suites on merge, nightly, or pre-release.
6. Attach traces, screenshots, video, and logs to failures for fast debugging.

Example GitHub Actions workflow:

yaml
name: pr-checks

on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER: app
          POSTGRES_PASSWORD: app
          POSTGRES_DB: app_test
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U app"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium

      - name: Start app
        run: |
          npm run migrate:test
          npm run seed:test
          npm run start:test &
          npx wait-on http://localhost:3000
        env:
          DATABASE_URL: postgres://app:app@localhost:5432/app_test

      - name: Run unit and integration tests
        run: npm test

      - name: Run critical workflow tests
        run: npx playwright test tests/critical-flows --project=chromium
        env:
          APP_URL: http://localhost:3000
          TEST_HELPER_URL: http://localhost:3000/test-helpers

      - name: Upload Playwright report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/

This is where CI/CD becomes meaningful again. Not because it is green, but because the green status includes evidence that your product’s essential paths were replayed successfully.

How to make workflow tests less flaky and more useful

People dismiss end-to-end tests because they remember brittle Selenium suites from a decade ago. Some of that criticism was earned. But most modern workflow testing failures come from poor discipline, not the concept itself.

Here are the practices that actually help.

1. Test outcomes, not implementation trivia

Bad:

• asserting on CSS classes
• clicking fragile nth-child selectors
• depending on animation timing
• reaching into internals instead of using user-facing semantics

Good:

• use accessible roles and labels
• assert URLs, visible text, and completed states
• prefer stable test IDs only where semantics are insufficient

2. Control data explicitly

Critical workflow tests should not depend on ambient shared state.

Use:

• unique emails and workspace names
• isolated databases or schemas
• deterministic seed users
• cleanup routines
• disposable environments per PR when feasible

Data collisions are a major source of false failures.

3. Eliminate arbitrary sleeps

Nothing rots browser tests faster than wait(5000).

Use Playwright’s built-in waiting on:

• visible elements
• network responses
• URL changes
• locators reaching expected state

The test should synchronize with product behavior, not with guesses.

4. Capture artifacts for debugging

If a workflow test fails in CI/CD and you cannot explain why within minutes, the test system is incomplete.

Capture:

• trace files
• screenshots on failure
• videos when useful
• console logs
• network logs
• server logs correlated to the test run

Fast debugging is part of test quality.

5. Keep the critical set intentionally small

You do not need 400 browser tests to get confidence. You need a compact set of high-value journeys that represent the product’s operational truth.

Ask:

• If this breaks, do users fail to activate, pay, collaborate, or recover access?
• Would a reviewer be unlikely to catch this from the diff?
• Could multiple subsystems contribute to failure?

If yes, it belongs in the critical workflow suite.

6. Treat failed workflow tests as product incidents, not annoying flakes

A broken onboarding proof test is not “just QA noise.” It is often the earliest evidence that your product no longer works for new users.

That should change the response pattern. Teams that take workflow failures seriously get better debugging habits, better observability, and better release confidence.

Tool comparison: what each layer is actually good for

Here is the blunt version.

The mistake is not using the wrong tool. The mistake is asking one tool to answer a question it cannot answer.

Unit tests cannot prove checkout works. Manual QA cannot keep up with agent-generated change volume. Code review cannot simulate every state transition in a multi-step onboarding journey. Browser-based action-level verification is the right instrument for that question.

Why this matters more with AI-generated code

AI changes the economics of software change.

More code gets written. More refactors get attempted. More dependency updates get proposed. More “small” improvements land. More surface area shifts without deep human understanding.

That does not automatically mean quality goes down. But it absolutely means your old heuristics become less trustworthy.

When human authors wrote every line, the pace of change itself limited exposure. Reviewers could sometimes rely on tribal knowledge: “Sarah touched auth, she probably tested it locally.” That model already had cracks, but it functioned socially.

An agent-generated PR has no such social guarantee. It may have valid code and invalid product behavior. It may satisfy every existing assertion while violating an unstated workflow dependency no one encoded.

So the governance model has to evolve. If agents can author changes, your pipeline must demand stronger proof.

Not because AI is magic. Because software change without user-level verification is speculation, regardless of who wrote it.

Actionable practices to implement this quarter

If you want to improve reliability without boiling the ocean, start here.

1. Identify your top 10 revenue- or activation-critical flows

Do this with engineering, product, support, and growth together.

Examples:

• create account
• verify email
• first workspace creation
• invite teammate
• start trial
• upgrade plan
• password reset
• login with SSO
• enable MFA
• import first dataset

If one of these breaks, it matters immediately.

2. Turn each flow into one canonical proof test

Keep it narrow and outcome-focused. One happy-path proof per critical workflow is better than twenty half-maintained scenario tests.

The goal is not exhaustive coverage. The goal is confidence that the product still basically works where it counts.

3. Add proof tests as required CI/CD checks for agent-authored PRs

Realistically, they should be required for all meaningful PRs, but if you need political leverage, start with agent-authored changes, dependency updates, auth changes, billing changes, and frontend refactors.

If the workflow can be affected, replay it.

4. Build a thin test-helper layer

Most teams fail here by trying to automate email inboxes, OTP apps, and external billing systems in ad hoc ways.

Create internal test helpers for:

• fetching magic links and verification emails
• generating OTP seeds or bypass tokens in test envs
• seeding accounts and workspaces
• resetting state
• exposing deterministic fixtures

This will do more for test stability than arguing about test philosophy.

5. Require artifacts on failure

No exception. If a critical flow fails in CI/CD, the PR should show enough evidence to debug quickly.

At minimum:

• screenshot
• Playwright trace
• browser console
• relevant app logs

This dramatically improves trust in the suite.

6. Track workflow pass rates like product health indicators

Do not bury these tests under generic automation metrics.

Track:

• onboarding flow pass rate
• checkout flow pass rate
• invite flow pass rate
• auth recovery flow pass rate
• average time to diagnose failures

These are operational indicators of real reliability.

7. Prune low-value tests to afford high-value ones

If your pipeline is overloaded, remove redundant low-signal tests instead of refusing to add workflow proofs.

Ten brittle component snapshots are often less valuable than one end-to-end onboarding verification.

A realistic rollout model

You do not need a giant quality transformation program. A pragmatic rollout can happen in stages.

Stage 1: visibility

• list critical user journeys
• map existing coverage
• identify where confidence currently comes from and where it is imaginary

Stage 2: proof

• automate 3 to 5 highest-value workflows
• run them in CI/CD on pull requests
• fix infrastructure issues until failures are diagnosable

Stage 3: enforcement

• make workflow proofs required checks
• add ownership for maintaining them
• define when a change can bypass them, if ever

Stage 4: optimization

• shard tests for speed
• provision preview environments
• add nightly expanded suites
• correlate test failures with production incidents and support tickets

This is manageable. The hard part is not the tooling. The hard part is admitting that your current green pipeline may not mean what you say it means.

Conclusion

The dangerous PR is no longer the one with obviously bad code. It is the one that looks correct, passes tests, and quietly breaks a real workflow.

That is the world AI coding agents accelerate.

If agents can open pull requests faster than reviewers can reason about product impact, then code review cannot be your final source of truth. Passing unit tests cannot be your final source of truth. A green CI/CD badge built on narrow checks cannot be your final source of truth.

User reality has to be replayed.

The teams that adapt will stop treating testing as a proxy for software quality and start using it to generate evidence. They will identify the flows that define product viability, automate them at the browser level, run them on every meaningful change, and use failures to improve debugging, observability, and release discipline.

That is what real confidence looks like in the age of agent-driven development.

Not “the diff seems fine.”

Not “CI passed.”

But: “We proved the onboarding flow still works.”