The PR Looked Fine Until OAuth Broke: Why Mocked Auth Makes CI Lie

A pull request goes green. Unit tests pass. Integration tests pass. The preview environment loads. Someone clicks approve.

Then production users hit “Sign in with Google” and get bounced back to /callback?error=redirect_uri_mismatch. Or the login works once, then silently fails on refresh because the cookie was set with the wrong SameSite policy behind a different domain. Or an enterprise customer signs in successfully but lands on a blank page because your role mapping logic assumed a claim that only existed in staging. Or your app behaves perfectly in local dev and CI because the tests injected a token straight into local storage, neatly bypassing every brittle part of the authentication workflow.

This is one of the most common lies in modern software delivery: green CI checks that never validated the thing users actually do.

Authentication is the ideal place for false confidence. Teams mock the identity provider, stub token exchange, skip browser redirects, short-circuit session setup, and call it “covered.” The tests become fast, deterministic, and almost completely disconnected from reality. Then everyone acts surprised when sign-in breaks in production.

AI-generated code makes this worse, not better. It produces authentication integrations that look plausible, compile cleanly, and satisfy mocked tests. You get tidy middleware, token parsing, callback handlers, and refresh logic that appears correct under laboratory conditions. But OAuth failures are rarely about syntax. They are about coordination across browser behavior, cookies, redirect URIs, environment config, consent state, scopes, proxies, and provider-specific quirks. That entire surface area gets erased the moment CI pretends auth is just “set token, continue.”

If your product depends on signed-in workflows, then auth is not setup code. It is the front door. And a front door you only test by checking whether the lock object returns true is not tested at all.

The real problem is not OAuth complexity. It’s workflow blindness.

Most teams do not ship broken sign-in because they fail to understand OAuth diagrams. They ship broken sign-in because their validation strategy focuses on code paths rather than user workflows.

That distinction matters.

A code-path mindset asks:

• Did the callback handler return 200?
• Did the token parser decode claims?
• Did the middleware populate req.user?
• Did the auth client expose login() and refresh()?

A workflow mindset asks:

• Can a new user complete login from the browser with the real provider config?
• Does consent work for a first-time account?
• Does the app recover cleanly from an expired session?
• Does silent refresh actually happen in the deployed environment?
• Do role changes at the identity provider take effect correctly in the app?
• Do cookies survive redirects, domains, proxies, and browser policy differences?
• Can a returning user sign in, navigate, and perform protected actions without brittle edge-case failure?

The first set is what most CI pipelines validate. The second set is what production depends on.

OAuth and OIDC failures usually happen at the seams:

• your app and the browser
• your app and the identity provider
• frontend and backend session assumptions
• local and deployed environments
• first login and returning login
• happy-path tokens and real-world expired state

Mocks erase seams. Real users find them.

Why current approaches keep failing

Teams usually defend their auth testing in three ways:

1. “We have unit tests.”
2. “We have integration tests.”
3. “QA tested sign-in manually.”

All three can be true while production auth is still unreliable.

Unit tests validate logic, not trust boundaries

Unit tests are useful for auth code. You should test claim parsing, authorization helpers, session serializers, guard middleware, and token refresh decision logic.

But unit tests cannot tell you whether:

• the browser actually sends the cookie after redirect
• the callback URL matches what the provider expects
• your reverse proxy strips headers that auth middleware needs
• your PKCE verifier survives navigation state correctly
• your provider tenant is configured differently in staging and prod
• third-party bot checks or CAPTCHA gates interfere with login automation or real users

Here is a typical example of a unit or pseudo-integration test that passes while proving almost nothing meaningful about production auth:

js
import request from 'supertest';
import { app } from '../app';

jest.mock('../auth/verifyToken', () => ({
  verifyToken: () => ({
    sub: 'user_123',
    email: 'user@example.com',
    roles: ['admin']
  })
}));

describe('GET /api/projects', () => {
  it('returns projects for authenticated user', async () => {
    const res = await request(app)
      .get('/api/projects')
      .set('Authorization', 'Bearer fake-token');

    expect(res.status).toBe(200);
    expect(res.body).toEqual(expect.any(Array));
  });
});

This test validates your projects endpoint under an assumed authenticated state. Fine. Keep it.

But it does not validate login.

It does not validate callback handling.

It does not validate cookies.

It does not validate refresh.

It does not validate logout.

It does not validate the relationship between your browser app, backend, auth provider, and deployment environment.

Yet these are exactly the places sign-in breaks.

Integration tests often stub the most fragile part

A lot of teams say they run “integration tests” for auth. In reality, they test app integration with a fake or local auth contract.

For example:

js
beforeEach(async ({ page }) => {
  await page.goto('http://localhost:3000');
  await page.evaluate(() => {
    localStorage.setItem('access_token', 'fake-jwt');
    localStorage.setItem('user', JSON.stringify({
      email: 'user@example.com',
      roles: ['admin']
    }));
  });
});

test('admin can create a project', async ({ page }) => {
  await page.goto('http://localhost:3000/projects');
  await page.getByRole('button', { name: 'New Project' }).click();
  await page.getByLabel('Name').fill('Launch Plan');
  await page.getByRole('button', { name: 'Create' }).click();
  await expect(page.getByText('Launch Plan')).toBeVisible();
});

This is not an auth test. It is a privileged-state test. Sometimes that is useful, especially for downstream application workflows. But if this is your only browser-level validation, CI is lying to you about sign-in reliability.

The brittle path was skipped entirely.

Manual QA is too shallow and too late

Manual sign-in checks catch some issues, but they have structural limits:

• they are inconsistent
• they rarely cover multiple browsers and environments
• they do not reliably exercise expired sessions or token refresh timing
• they often use warmed accounts with existing consent state
• they happen after merge pressure is already high
• they do not scale with release frequency

The most common auth failures hide in transitions that manual QA does not reproduce consistently:

• first-time login vs returning login
• role changed externally after session issued
• callback URL slightly different in preview env
• cookies behaving differently under HTTPS and custom domain
• silent refresh blocked in one browser but not another
• SSO provider adding a new prompt or anti-bot challenge

Manual QA is valuable as exploratory coverage. It is not a credible primary defense for auth-dependent release quality.

AI-generated code makes the auth confidence gap wider

There is a specific reason AI-generated code increases authentication risk: it optimizes for plausibility.

That sounds useful until you remember that auth bugs are usually not about whether the code looks reasonable.

A generated OAuth handler often has all the expected ingredients:

• a redirect to the provider
• a callback route
• token exchange code
• cookie/session storage
• middleware to guard routes
• refresh token plumbing

Under mocked tests, all of this looks “done.”

But generated code frequently bakes in assumptions that only fail in real execution:

• redirect URIs hardcoded for localhost
• cookies missing Secure or wrong SameSite
• refresh logic that assumes a hidden iframe flow still works under current browser restrictions
• insufficient nonce/state validation
• role extraction from the wrong claim
• assumptions that access token expiry and app session expiry are aligned
• no distinction between browser session loss and API token invalidation
• provider-specific parameters omitted because they were not obvious from generic examples

A human reviewer often misses these because the code is clean and the tests pass. The generated implementation satisfies the shape of an auth integration without surviving the behavior of one.

That is the broader problem with AI in delivery: it can increase code volume faster than your workflow validation can keep up. When teams rely more on generated code, they need stronger end-to-end verification at system boundaries, not weaker.

Auth is one of the sharpest boundaries you have.

The failure modes CI usually misses

Let’s get concrete. These are the classes of auth issues that routinely escape PR validation.

Callback mismatch and environment drift

Your app redirects users to the provider with one callback in preview, another in staging, and a third in production. One of them is missing from the provider’s allowlist.

Locally, everything works because you used http://localhost:3000/callback. In CI, everything works because the tests never redirected at all. In production, customers get blocked.

This gets worse with:

• ephemeral preview URLs
• multiple subdomains
• load balancers rewriting host headers
• ingress/proxy config changing X-Forwarded-Proto
• frameworks inferring base URLs incorrectly

SameSite and cookie delivery bugs

Browser cookie behavior is one of the most under-tested parts of auth.

Common failures include:

• session cookie not sent after cross-site redirect
• cookie works on one domain but not another
• Secure not set in production HTTPS path
• SameSite=Lax or Strict breaking expected redirect/session behavior
• frontend and API on different subdomains causing cookie scope problems

All your backend tests can pass while the browser quietly drops the cookie that your app needs to be “logged in.”

Silent refresh and token expiry failures

A login that works for 30 seconds is still broken.

Teams often validate initial sign-in but not session continuity. Then production users hit failures like:

• access token expires and refresh never runs
• refresh runs but cookie/session no longer matches
• hidden iframe silent auth blocked by browser policy
• background refresh loses race conditions across tabs
• refresh token rotation implemented incorrectly
• app loops between 401, refresh, and redirect

These are not hypothetical edge cases. They are routine.

Role and permission drift

Authentication is not just identity. It is authorization context.

A test user in CI often has exactly the expected claims and roles because you fabricated them. Real providers are messier:

• claims differ by tenant
• enterprise SSO mappings change
• group membership sync lags
• custom claim namespaces vary by environment
• role names drift from what app code expects

That means protected workflows pass in CI and fail only for certain customer accounts or org configurations.

Bot checks, consent prompts, and provider UX variations

Modern login flows are not static.

Providers insert:

• consent prompts
• re-auth prompts
• anti-bot checks
• device verification
• MFA challenges
• account selection pages

Even if your app is technically correct, the end-to-end workflow can still break because your assumptions about the browser sequence are wrong.

You do not need to automate every MFA branch in every PR. But if your entire test strategy assumes login is a one-step token acquisition event, then you are validating a fantasy.

The core insight: auth must be tested as a user action in a browser, in CI

The only credible way to verify auth-dependent workflows is to test them as users execute them: through a browser, against a deployed environment, with real redirects, real cookies, real session establishment, and at least one real identity-provider-backed path.

Not every test needs to log in from scratch.

Not every PR needs a full matrix across every provider and every role.

But somewhere in your CI/CD system, before code reaches users, the system must prove:

• a browser can start unauthenticated
• complete the login action
• return through the real callback path
• establish the expected authenticated session
• access a protected workflow
• survive at least one meaningful auth state transition

That transition might be:

• page reload after sign-in
• access token expiration and refresh
• logout and re-login
• role-gated navigation
• expired session recovery

If you are not validating auth this way, then your green checks do not mean what stakeholders think they mean.

What this looks like in practice with Playwright

Playwright is not magic, but it is one of the more practical tools for browser-level auth validation because it can execute the workflow the browser actually sees.

There are two broad patterns:

1. Real login in CI for at least one controlled test account and environment
2. Captured authenticated state for downstream workflow tests after one trusted login path has been validated

The mistake is using only the second pattern.

A real browser login test

Here is a simplified Playwright example that validates an actual login path.

ts
import { test, expect } from '@playwright/test';

test('user can complete real OAuth login and reach dashboard', async ({ page }) => {
  await page.goto(process.env.APP_URL!);

  await page.getByRole('button', { name: /sign in with google/i }).click();

  // Provider-hosted page
  await page.getByLabel(/email/i).fill(process.env.E2E_AUTH_EMAIL!);
  await page.getByRole('button', { name: /next/i }).click();

  await page.getByLabel(/password/i).fill(process.env.E2E_AUTH_PASSWORD!);
  await page.getByRole('button', { name: /next/i }).click();

  // Back in the app after callback
  await page.waitForURL(/dashboard/);
  await expect(page.getByRole('heading', { name: /dashboard/i })).toBeVisible();

  // Protected API-backed UI proves session is usable
  await expect(page.getByText(/welcome/i)).toBeVisible();
});

In the real world you may need provider-specific setup, test accounts, or non-interactive enterprise login patterns. The exact mechanics vary. The important part is not “use this exact script.” It is that CI validates the browser redirect chain and resulting authenticated state.

Validate session persistence and refresh

Initial login is not enough. Add a check that proves the app can keep working after a token or session transition.

ts
import { test, expect } from '@playwright/test';

test('session survives reload and refresh path', async ({ page }) => {
  await page.goto(process.env.APP_URL!);
  await page.getByRole('button', { name: /sign in/i }).click();

  await page.waitForURL(/dashboard/);
  await page.reload();
  await expect(page.getByRole('heading', { name: /dashboard/i })).toBeVisible();

  // Wait long enough for your short-lived test token to require refresh,
  // or trigger a backend helper that invalidates the access token.
  await page.waitForTimeout(70_000);
  await page.goto(`${process.env.APP_URL}/account`);

  await expect(page.getByText(/account settings/i)).toBeVisible();
  await expect(page.getByText(/session expired/i)).not.toBeVisible();
});

For faster CI, teams often configure a dedicated test tenant with short token TTLs so refresh behavior can be exercised quickly.

Use saved storage state only after proving login works

Once one suite validates real login, downstream tests can reuse authenticated state for speed.

ts
import { test as setup, expect } from '@playwright/test';

setup('authenticate', async ({ page }) => {
  await page.goto(process.env.APP_URL!);
  await page.getByRole('button', { name: /sign in/i }).click();
  await page.waitForURL(/dashboard/);
  await expect(page.getByRole('heading', { name: /dashboard/i })).toBeVisible();
  await page.context().storageState({ path: 'playwright/.auth/user.json' });
});

Then in other tests:

ts
import { test, expect } from '@playwright/test';

test.use({ storageState: 'playwright/.auth/user.json' });

test('authenticated user can create a project', async ({ page }) => {
  await page.goto(`${process.env.APP_URL}/projects`);
  await page.getByRole('button', { name: 'New Project' }).click();
  await page.getByLabel('Name').fill('Revenue Model');
  await page.getByRole('button', { name: 'Create' }).click();
  await expect(page.getByText('Revenue Model')).toBeVisible();
});

That is a sensible compromise: one trusted real-login setup path, many fast authenticated workflow tests.

CI/CD wiring: make auth validation a release gate

If auth matters to production behavior, auth validation must be part of release gating, not an optional nightly curiosity.

A simple GitHub Actions setup might look like this:

yaml
name: e2e-auth

on:
  pull_request:
  push:
    branches: [main]

jobs:
  e2e-auth:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    env:
      APP_URL: ${{ secrets.E2E_APP_URL }}
      E2E_AUTH_EMAIL: ${{ secrets.E2E_AUTH_EMAIL }}
      E2E_AUTH_PASSWORD: ${{ secrets.E2E_AUTH_PASSWORD }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npm run test:e2e:auth
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/

A few important points:

• Run against a deployed preview or staging environment, not only localhost.
• Preserve traces, screenshots, and videos for debugging.
• Use dedicated test accounts and tenants.
• Keep this suite small and critical-path focused so it remains fast enough to gate merges.

For deeper environment-specific coverage, add a scheduled suite against staging and production-safe smoke tests.

Python example: backend checks still matter, just not alone

Browser tests should prove the workflow. Backend tests should still validate server behavior around auth boundaries.

For example, a Python service can verify that expired tokens are rejected and claim mapping behaves correctly.

python
from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

def test_protected_route_rejects_missing_token():
    response = client.get('/api/account')
    assert response.status_code == 401

def test_role_mapping_allows_admin(monkeypatch):
    def fake_user_from_token(token: str):
        return {
            'sub': 'user_123',
            'email': 'admin@example.com',
            'roles': ['admin']
        }

    monkeypatch.setattr('app.auth.user_from_token', fake_user_from_token)
    response = client.get(
        '/api/admin/stats',
        headers={'Authorization': 'Bearer fake-token'}
    )
    assert response.status_code == 200

These tests are good. Keep them. They validate important business logic quickly.

But do not mistake them for proof that a real user can sign in and use the app.

Tool comparison: what each layer is good for

The answer is not “replace everything with end-to-end tests.” The answer is to stop using lower-level tests as evidence for things they cannot prove.

Unit tests

Good for:

• claim parsing
• authorization rules
• session helpers
• callback state validation logic
• edge-case error handling

Bad for:

• browser redirects
• provider configuration
• cookie transport behavior
• silent refresh under real browser policies

Integration/API tests

Good for:

• protected endpoint behavior
• token-required access patterns
• service-to-service auth contracts
• role-based API responses

Bad for:

• real browser session establishment
• deployed callback correctness
• user-facing auth workflow continuity

Manual QA

Good for:

• exploratory auth edge cases
• weird provider UX changes
• visual or copy regressions in sign-in screens

Bad for:

• reliable release gating
• repeatability
• broad environment coverage
• refresh/expiry timing scenarios

Browser-level E2E with Playwright

Good for:

• real login flow validation
• redirect/callback verification
• cookie/session establishment
• protected workflow confirmation
• debugging production-like auth failures with traces

Bad for:

• replacing all lower-level tests
• broad combinatorial auth matrices in every PR if poorly scoped

The right model is layered testing with honest boundaries. Browser E2E is not the whole strategy. It is the missing truth layer.

Actionable practices that actually reduce auth incidents

If your team keeps shipping auth regressions despite green CI, here is what to change.

1. Add one real-login browser test as a required check

Start small. One test. One provider. One critical environment.

Success criteria:

• unauthenticated browser starts at app
• user initiates sign-in
• real redirect completes
• callback returns to app
• protected page loads successfully

That one test will catch more real auth breakage than a pile of mocked token tests.

2. Test one session transition, not just initial login

Pick at least one of these:

• reload after login
• token refresh after short TTL
• expired session recovery
• logout then re-login

A login that does not survive normal session behavior is not a passing auth system.

3. Use dedicated auth test tenants and accounts

Do not rely on a random employee account with sticky consent history and changing MFA state.

Create controlled test identities with:

• known roles
• stable profile attributes
• predictable consent setup
• explicit environment mapping
• short token lifetime where useful

This improves both reliability and debugging.

4. Validate role-based workflows with provider-issued claims

Do not fabricate admin claims in every test. At least one path should confirm that the provider-issued identity actually unlocks the expected app behavior.

This is where role/permission drift shows up.

5. Run auth E2E against deployed URLs, not just localhost

A lot of auth bugs only appear when domains, HTTPS, proxies, and callback URLs are real.

If preview environments are too dynamic for provider config, use a stable staging URL as a required gate before promotion.

6. Capture traces and network logs for debugging

Auth failures are painful when all you know is “login failed in CI.”

Use Playwright traces, screenshots, video, and network data. These turn flaky-seeming auth issues into diagnosable state transitions.

That directly improves debugging and developer productivity because engineers stop guessing where the redirect chain failed.

7. Separate “workflow authenticated” tests from “auth validation” tests

This is a subtle but important distinction.

• Auth validation tests prove the login system works.
• Workflow authenticated tests assume login already worked and exercise product behavior quickly.

You need both. Confusing them is how teams accidentally stop testing auth while believing they are testing it.

8. Treat AI-generated auth code as high-risk until workflow-proven

Generated code that touches:

• OAuth callbacks
• session storage
• token refresh
• cookie setup
• role mapping
• auth middleware

should trigger stricter review and browser-level verification.

Not because AI code is uniquely bad, but because it tends to look complete before it is behaviorally trustworthy.

9. Keep the critical auth suite intentionally narrow

Do not build a 90-minute auth matrix and then watch the team disable it.

Required PR auth coverage should be compact:

• one or two real login paths
• one protected workflow
• one session transition
• maybe one role-gated check

Then expand coverage in scheduled or pre-release suites.

10. Redefine what “green CI” means internally

This is partly cultural.

If leadership, product, and engineering all interpret green checks as “safe to ship,” then auth cannot live outside the checks that go green.

Be explicit:

• mocked tests prove logic
• browser auth tests prove sign-in works
• both are required for confidence

That is a healthier CI/CD contract with the business.

The bigger point: reliability comes from user workflows, not isolated correctness

Authentication is just the clearest example of a wider engineering problem.

Modern teams produce more code than they can realistically reason about line by line. AI accelerates that. CI pipelines then compensate by emphasizing fast, isolated tests that are easy to automate and easy to pass. Over time, “coverage” drifts further away from the system behavior users depend on.

That is why so many teams feel confused after incidents. The code looked fine. The tests passed. The PR was approved. But the workflow broke.

The lesson is not that testing is failing. The lesson is that teams often test the wrong abstraction.

Users do not care whether your auth middleware branch achieved 92 percent coverage. They care whether they can sign in, stay signed in, and use the product.

If CI never validated that workflow, then CI did not validate the product.

Conclusion

Mocked auth is seductive because it makes test suites fast, stable, and green. It also removes the exact parts of the system that fail most often in production.

That is why mocked auth makes CI lie.

The path from “click sign in” to “use a protected feature” crosses browser behavior, provider configuration, callbacks, cookies, consent, token lifecycles, and authorization context. Those are not implementation details. They are the product experience.

As AI-generated code increases the volume of plausible auth integrations, this gap will only get worse for teams that rely on token stubs and bypassed redirects as proof of correctness.

The fix is not to abandon unit tests or integration tests. The fix is to stop pretending they verify workflows they never execute.

If sign-in matters, then a browser in CI must prove it.

Not theoretically. Not through a mock. Not through a handcrafted fake JWT.

Through the real user action.

That is the difference between a green PR and a reliable release.