The PR Review Mirage: Why AI-Generated Diffs Hide Broken User Flows

A team merges a pull request on Friday afternoon. The diff is tidy. The summary is crisp. The tests are green. The reviewer scans the changed files, sees a sensible refactor, and approves.

On Monday, support gets a ticket: users can no longer complete checkout if they edit their shipping address after selecting express delivery. Nothing crashed. No obvious exception surfaced. The PR did not remove the checkout flow. It only “cleaned up state handling,” “standardized selectors,” and “reused an existing permission helper.” Every individual change looked reasonable in isolation. The pull request was readable. The application was not reliable.

This is the PR review mirage.

As AI generates more code, more pull requests will look polished enough to approve. That is the dangerous part. The problem is not low-quality code that screams for attention. The problem is plausible code that passes review rituals built for human-written diffs and code-level reasoning, while breaking the workflows users actually depend on.

Traditional review is optimized for reading changes. Reliability depends on executing behavior.

Those are not the same thing.

The real problem is not missing tests. It is the wrong unit of review.

When teams talk about quality, they often default to a checklist:

• unit tests pass
• integration tests pass
• CI/CD is green
• linters are clean
• reviewer approved
• QA will catch anything else

That checklist sounds disciplined. In practice, it often measures whether code is internally consistent, not whether the product still works for a real user.

This gap gets wider with AI-assisted development.

AI tools are very good at producing diffs that satisfy local expectations:

• code looks stylistically consistent
• variable names are reasonable
• tests are updated to match implementation
• components are refactored with minimal noise
• PR descriptions sound coherent

But users do not experience local expectations. They experience workflows.

A user signs in, lands on a dashboard, opens a modal, uploads a file, waits for background processing, refreshes, navigates to billing, upgrades a plan, comes back, retries an action, and expects state, permissions, and timing to still make sense.

A pull request can preserve code quality at the file level while breaking that full chain in subtle ways:

• a loading spinner resolves earlier than before
• a selector now matches the wrong duplicated element
• a permission check shifts from page load to action time
• cached state persists after navigation when it previously reset
• a background mutation races with a transition
• a URL parameter is normalized differently across routes
• an optimistic update hides a failed backend write

None of those failures needs to look dramatic in a diff. Most of them look like acceptable implementation details.

That is why saying “we need more tests” is incomplete. The deeper issue is that the team is reviewing code artifacts while users interact with action sequences.

Why AI-assisted development makes this worse

AI does not merely accelerate coding. It increases the volume of believable changes.

That matters because review capacity is finite.

A senior engineer can deeply reason about a few risky changes. They cannot fully simulate every side effect of a stream of clean, scoped, AI-assisted pull requests. So teams lean even harder on heuristics:

• the diff is small
• naming is clean
• tests were updated
• no risky schema migration
• reviewer recognized the pattern
• CI/CD passed

These heuristics were already imperfect. Under AI-generated throughput, they become dangerous.

AI can produce changes that are syntactically correct, semantically plausible, and operationally wrong. In fact, that is the default failure mode. The code often “makes sense” while the product behavior drifts.

The result is a new class of review failure: humans approve changes they would never have trusted if they had replayed the user task.

This is the critical shift. With more generated code, the bottleneck moves from code production to behavior verification.

If your review process still centers on reading diffs, you are optimizing the cheapest part of the system and ignoring the most expensive failure: broken user flows in production.

Why diff-based review fails for workflow regressions

Code review works best when the risk is visible in code structure:

• obvious logic bug
• missing null check
• insecure API usage
• incorrect transaction boundary
• dangerous migration
• broken abstraction

It works much worse when the risk emerges only during interaction over time.

Consider a React change that “simplifies” form state synchronization:

javascript
// before
useEffect(() => {
  if (!isEditing) {
    setFormData(serverData)
  }
}, [serverData, isEditing])

// after
useEffect(() => {
  setFormData(serverData)
}, [serverData])

The after version may even look cleaner. A reviewer may think: fewer branches, easier to follow.

But now a background refresh can overwrite the user’s in-progress edits while they are filling out the form.

The diff does not show the broken workflow. The workflow only appears when a user edits a field, waits for a polling update, and then tries to submit.

Here is another example involving selector cleanup in Playwright tests:

javascript
// before
await page.getByRole('button', { name: 'Continue to payment' }).click()

// after
await page.locator('.primary-button').click()

Maybe the UI library standardized classes. Maybe the generated test update still passes in CI. But on a real page with multiple primary actions, the test might click the wrong button depending on timing, viewport, or hidden elements.

The PR looks “test-covered.” The test is now weaker.

Or a backend permission refactor in Python:

python
# before
if user.is_admin or document.owner_id == user.id:
    allow_edit = True

# after
allow_edit = permissions.can_edit_document(user, document)

This looks like a maintainability improvement. The risk is invisible unless the helper changes context assumptions: maybe it checks organization role but not temporary delegated access; maybe it caches stale permission data across requests; maybe it behaves differently after a subscription downgrade.

Again: readable diff, broken workflow.

Reviewers are being asked to infer dynamic behavior from static representation. That works only up to a point.

Why CI/CD gives false confidence here

CI/CD is valuable. It catches syntax errors, dependency issues, obvious regressions, and plenty of integration failures. But green CI often gets misinterpreted as evidence that user behavior is intact.

That leap is where teams fool themselves.

Most CI pipelines verify code health, not workflow truth.

A typical pipeline looks like this:

yaml
name: ci

on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test
      - run: npm run test:integration

This pipeline may be perfectly healthy. It still does not prove that a user can:

• sign up with Google
• create a workspace
• invite a teammate
• upload a CSV
• map columns
• process the import
• resolve validation errors
• retry failed rows
• see updated analytics

CI/CD often validates components of the journey without validating the journey itself.

Worse, AI-generated changes frequently update tests so they continue passing against the new implementation. That sounds harmless until you realize the tests may be adapting to the wrong behavior.

A generated PR might:

• update snapshots to match a broken UI state
• relax assertions to remove flakiness caused by a race it introduced
• mock an API path that no longer reflects production timing
• replace semantic selectors with implementation-coupled selectors
• narrow tests to the changed function while ignoring downstream flow impact

So the signal becomes: “the changed code is internally compatible with the changed tests.”

That is not the same as: “the product still supports the user’s task.”

Why unit tests and QA also miss this class of failure

Unit tests are designed to isolate logic. That isolation is their strength and their limitation.

If the failure lives in cross-page behavior, timing interactions, or state transitions between systems, unit tests will happily pass while users fail.

For example, each of these can be individually correct:

• form reducer logic
• API client response handling
• route guard permission check
• modal open/close state
• analytics event dispatch

But together, they can produce a broken experience:

1. User opens edit modal.
2. Route-level refresh updates parent data.
3. Local draft resets.
4. User clicks save.
5. Optimistic update fires.
6. Permission check fails server-side after plan change.
7. Error toast renders behind modal overlay.
8. Navigation occurs anyway.

No single unit test sees the whole failure.

Manual QA is not a reliable backstop either, especially in high-throughput teams. QA usually works from explicit scenarios and release timing. AI-generated diffs increase the number of “probably safe” changes merged between full exploratory passes. By the time the workflow regression is noticed, the causal PR is buried in a stack of innocuous approvals.

And because the broken behavior often requires specific sequencing, QA may miss it unless they reproduce the exact task under the right timing and state conditions.

The issue is not that QA is bad. The issue is that post-merge QA is too late and too narrow to compensate for a review model that never required workflow proof in the first place.

The core insight: review actions, not just code

If user-facing reliability is the goal, then the artifact under review cannot be only the diff.

The PR needs evidence that key user workflows were actually executed.

This is the shift teams need to make:

From:

• “Does the code look right?”
• “Did CI pass?”
• “Are there tests?”

To:

• “Which user tasks could this change affect?”
• “What proof do we have those tasks were executed?”
• “Can the reviewer inspect behavior, not just implementation?”

That proof can take multiple forms:

• browser workflow replays attached to the PR
• trace artifacts from Playwright or Cypress
• screenshots at key state transitions
• logs proving cross-service action completion
• structured checklists tied to critical journeys
• environment links with seeded data and exact repro steps
• machine-verifiable evidence that target flows ran successfully

The point is not ceremony. The point is moving review closer to actual product behavior.

When a PR changes anything that might affect timing, selectors, permissions, navigation, background jobs, local state, or cross-page continuity, code review alone is insufficient.

You need action-level verification.

What action-level verification looks like in practice

Start by defining workflow contracts.

A workflow contract is a short list of user tasks that must remain true for a feature area. For example, for billing:

• user can upgrade from trial to paid plan
• user can change seat count and see updated invoice preview
• owner can add payment method and retry failed invoice
• non-owner cannot access billing settings

For onboarding:

• user can sign up, verify email, create workspace, and land on dashboard
• invited user can accept invite and join existing workspace
• user can upload starter data and complete initial setup

Then map PR risk to workflows. A reviewer should be able to ask:

• what user task might this touch?
• where is the replay or execution artifact?
• was the workflow run in a realistic environment?

Example: Playwright workflow evidence in CI

A simple Playwright config can produce trace files, videos, and screenshots that become review artifacts.

javascript
import { defineConfig } from '@playwright/test'

export default defineConfig({
  testDir: './e2e',
  retries: 1,
  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    trace: 'on',
    video: 'retain-on-failure',
    screenshot: 'only-on-failure'
  },
  reporter: [
    ['html'],
    ['json', { outputFile: 'playwright-report/results.json' }]
  ]
})

A workflow test should reflect an actual task, not just page availability:

javascript
import { test, expect } from '@playwright/test'

test('user can edit shipping address after selecting express delivery', async ({ page }) => {
  await page.goto('/checkout')

  await page.getByLabel('Email').fill('buyer@example.com')
  await page.getByLabel('Address line 1').fill('123 Main St')
  await page.getByRole('radio', { name: 'Express delivery' }).check()

  await page.getByRole('button', { name: 'Edit address' }).click()
  await page.getByLabel('Address line 1').fill('456 Market St')
  await page.getByRole('button', { name: 'Save address' }).click()

  await expect(page.getByText('456 Market St')).toBeVisible()
  await expect(page.getByText('Express delivery')).toBeVisible()

  await page.getByRole('button', { name: 'Continue to payment' }).click()
  await expect(page).toHaveURL(/payment/)
})

Notice what matters here: the test verifies continuity of state across actions. It checks the workflow a user cares about, not just whether the shipping component renders.

Example: Attach workflow artifacts to pull requests

In GitHub Actions, publish artifacts for reviewers:

yaml
name: pr-workflow-verification

on:
  pull_request:

jobs:
  e2e-critical-flows:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npm run start:test &
      - run: npx wait-on http://localhost:3000
      - run: npm run test:e2e:critical
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-artifacts
          path: |
            playwright-report/
            test-results/

Better yet, comment on the PR with links to:

• passing workflow list
• failed workflow list
• trace viewer artifact
• preview environment URL
• seed data reference

Now review is not just “LGTM.” It is “I saw evidence the changed behavior still supports the user task.”

Add cross-system proof when frontend replay is not enough

Some failures are invisible in the browser alone. For example:

• permission updates lag behind UI state
• queue jobs fail after navigation succeeds
• analytics or audit logs are malformed
• billing actions partially complete across services

In those cases, attach backend or system-level assertions.

A Python example for a workflow verification helper:

python
import requests

def assert_import_workflow_completed(base_url: str, job_id: str, token: str):
    headers = {"Authorization": f"Bearer {token}"}

    job = requests.get(f"{base_url}/api/import-jobs/{job_id}", headers=headers).json()
    assert job["status"] == "completed", f"Unexpected status: {job['status']}"

    rows = requests.get(f"{base_url}/api/import-jobs/{job_id}/rows", headers=headers).json()
    assert rows["failed"] == 0, f"Failed rows present: {rows['failed']}"

    events = requests.get(f"{base_url}/api/audit-events?job_id={job_id}", headers=headers).json()
    assert any(e["type"] == "import.completed" for e in events), "Missing audit event"

This is not glamorous. It is reliable. It proves the workflow finished beyond what the UI happened to display.

What reviewers should explicitly look for in AI-generated pull requests

AI-generated diffs deserve a different kind of skepticism than obviously messy code.

The reviewer should not only ask whether the code is understandable. They should ask whether the change modifies any hidden workflow assumptions.

High-risk categories include:

• state synchronization changes
• effect dependency changes
• selector rewrites
• permission helper centralization
• route guard refactors
• retries, debouncing, polling, or timeout changes
• component extraction that moves ownership of state
• optimistic UI behavior changes
• API response shape adaptation
• cache invalidation changes
• background job orchestration updates
• test changes that relax assertions or change selectors

For these changes, the burden of proof should rise.

A good PR template for AI-assisted work might include:

markdown
## User workflows potentially affected
- Checkout: address edit after shipping method selection
- Billing: owner seat update and invoice preview
- Dashboard: filter persistence across navigation

## Verification evidence
- [ ] Critical flow replay attached
- [ ] Playwright trace attached
- [ ] Preview URL included
- [ ] Seed data / account state documented
- [ ] Cross-system assertions included where relevant

## Test changes
- List any assertions removed or weakened
- List any selector changes
- Explain any mocking updates

## AI assistance disclosure
- Which files or tests were AI-generated or AI-modified?
- What behavior was manually replayed before requesting review?

This is not bureaucracy for its own sake. It forces the author to think in workflows and makes hidden risk easier to review.

Tools comparison: what helps and what does not

No single tool solves this problem. But some tools align much better with action-level verification than others.

Unit test frameworks: Jest, Vitest, Pytest

Strengths:

• fast feedback
• excellent for logic correctness
• supports refactoring safely at function/module level
• foundational for developer productivity

Weaknesses:

• poor visibility into cross-page user flows
• easy to overfit to implementation
• cannot prove critical journeys still work

Verdict: necessary, not sufficient.

Integration tests

Strengths:

• validate interactions between modules/services
• catch API contract issues
• more realistic than unit tests

Weaknesses:

• still often stop short of full user tasks
• commonly use mocks that hide production timing/state issues
• hard to interpret as workflow evidence for reviewers

Verdict: useful middle layer, but not enough for PR confidence.

End-to-end frameworks: Playwright, Cypress

Strengths:

• closest to actual user actions
• can capture traces, screenshots, and videos
• ideal for replayable workflow proof in PRs
• strong debugging ergonomics when failures occur

Weaknesses:

• slower
• can become flaky if poorly written
• needs discipline around test data, selectors, and environment stability

Verdict: best fit for workflow verification when scoped to critical paths.

Session replay / synthetic monitoring tools

Strengths:

• useful for post-deploy visibility
• helps debugging real failures
• can reveal edge cases missed in test environments

Weaknesses:

• mostly reactive
• not a substitute for pre-merge verification

Verdict: important for operations, but too late for PR review proof.

Preview environments

Strengths:

• lets reviewers touch the actual change
• enables realistic end-to-end execution
• supports product, design, and QA collaboration before merge

Weaknesses:

• often underused in code review
• can be hard to keep data stable and reproducible
• “available” does not mean “verified”

Verdict: very valuable, but only when paired with explicit workflow execution.

Actionable practices teams should adopt now

Here is the practical version.

1. Define 5–15 critical user workflows

Do not start with hundreds of scenarios. Start with the journeys that matter most to revenue, activation, retention, and support volume.

Examples:

• sign up and create workspace
• invite teammate and accept invite
• checkout and payment completion
• import data and resolve validation errors
• upgrade subscription and change seats
• create, edit, save, and publish core resource

If these are not continuously verified, your testing strategy is upside down.

2. Require workflow evidence in pull requests that touch risky areas

Not every doc typo needs a replay. But if a PR touches state, navigation, permissions, timing, selectors, background jobs, or critical UX paths, require proof.

That proof can be:

• artifact from automated replay
• manual recording from preview environment
• trace plus backend assertions

The key is that the reviewer can inspect behavior.

3. Treat removed assertions as risk, not cleanup

AI often “fixes” flaky tests by making them less specific. Reviewers should flag:

• broader selectors
• deleted waits without replacement reasoning
• weaker assertions
• more mocks in place of real interactions
• snapshot updates with no behavioral explanation

Test diffs need as much skepticism as app diffs.

4. Build a thin but trustworthy critical-flow suite

Do not try to convert your whole test pyramid into expensive end-to-end coverage. That is how teams create brittle suites nobody trusts.

Instead, maintain a narrow set of workflow tests for business-critical paths. Keep them:

• realistic
• stable
• traceable
• visible in PRs
• owned by product area teams

This gives you high-value behavioral coverage without drowning CI/CD.

5. Make artifacts reviewer-friendly

A failing trace hidden in CI logs is not useful.

Publish:

• named workflow results
• direct trace links
• screenshots at checkpoints
• video for failures
• preview URL
• exact test account/seed state

Lower the cost of behavior review.

6. Add risk labeling for AI-generated PRs

Not because AI is magical. Because AI increases change volume and plausibility.

Useful labels:

• ai-assisted
• workflow-risk
• selector-change
• permission-change
• state-management-change
• critical-path-touched

These labels tell reviewers when code reading is especially likely to miss behavior drift.

7. Measure escaped workflow regressions

If a bug reaches production, classify it properly.

Do not just write “missing test.” Ask:

• Was there workflow evidence in the PR?
• Did review rely only on diff inspection?
• Did tests assert implementation instead of behavior?
• Did CI/CD signal false confidence?
• Was a critical path missing from the verification set?

This is how you improve the review system instead of blaming individual engineers.

A note on debugging: behavior-first evidence speeds root cause analysis too

This approach is not only about prevention. It improves debugging after failures.

When a regression report includes:

• exact workflow replay
• trace with timing
• screenshots at transitions
• network requests
• console logs
• backend event correlation

engineers can reason from observed behavior back to cause. That is much faster than staring at a diff and trying to imagine what happened in production.

The same artifacts that improve testing and review also improve debugging and developer productivity. They create a shared object of truth between author, reviewer, QA, support, and engineering leadership.

Without that, every incident becomes a debate over whether the code “should have worked.”

The cultural shift: stop treating QA as the first real user

Many teams still use code review as a filter for code quality and QA as the first serious pass at product behavior. That separation was always shaky. Under AI-assisted development, it breaks.

If generated pull requests are easy to produce and easy to approve, then user-flow validation has to move left into the PR itself.

That means:

• reviewers expect behavior evidence
• authors think in user tasks, not file changes
• CI/CD surfaces workflow results, not just unit test counts
• product teams define critical journeys explicitly
• post-merge QA becomes confirmation and exploration, not first-line regression detection

This is how mature teams maintain reliability when code generation gets cheaper.

Conclusion

AI-generated pull requests create a dangerous illusion: if the diff is neat, the tests are green, and the summary is coherent, the change feels safe.

But users do not run diffs. They run workflows.

The real failures now are not always obvious logic mistakes. They are hidden breaks in timing, selectors, permissions, state transitions, and cross-page continuity that no PR summary fully captures and no reviewer can reliably infer from code alone.

So the answer is not blind trust in CI/CD, more unit tests, or pushing the problem to post-merge QA. The answer is changing the unit of verification.

Treat user-flow regression proof as part of code review.

Attach replays. Publish traces. Show that key paths executed successfully. Require evidence for the workflows a change could affect.

Because in an AI-assisted world, the scarcest resource is no longer code generation. It is confidence that the software still works the way users need it to.

And that confidence cannot be read out of a diff.