A pull request can be completely green and still ship a broken product.
That sounds obvious if you have spent enough time in production, but the failure mode is getting worse. Not because engineers suddenly forgot how to test, and not because CI/CD stopped being useful. The problem is that the shape of software delivery changed faster than the shape of our testing. AI-assisted development makes it easy to produce coherent-looking code across multiple surfaces at once: frontend auth screens, backend callback handlers, billing webhooks, email verification flows, feature flags, and deployment config. Each piece can look correct in isolation. Each piece can even pass its own tests. Then a real user clicks “Continue with Google,” gets redirected twice, loses session state, lands on a callback route that behaves differently in preview than production, and the entire workflow collapses.
The PR looked fine until OAuth redirected twice.
That sentence captures the modern reliability gap. Most PR validation tells you whether code paths behave under controlled assumptions. It does not tell you whether a multi-step product action survives a real browser, external timing, third-party redirects, cookies, storage, callback state, network jitter, and environment-specific config. And increasingly, that is what determines whether the software works.
The failure is in the handoff, not the function
Here is the pattern showing up across modern products.
An engineer or coding agent updates several connected systems in one branch:
- frontend login page changes
- auth provider config update
- backend callback handler refactor
- session middleware tweak
- billing portal redirect addition
- email verification template change
- analytics event instrumentation
- preview environment variable cleanup
Every change is reasonable. Tests exist. CI passes.
Unit tests confirm the callback parser reads the code parameter. Integration tests confirm the session service can create a valid session. Component tests confirm the login button renders. Contract tests confirm the billing webhook payload still parses. Maybe even an end-to-end happy path exists locally using mocked auth.
Then production says otherwise.
The browser drops a cookie because SameSite differs on the preview domain. The OAuth provider redirects to a callback URL that is registered for production but not ephemeral environments. The frontend assumes the session is available immediately after redirect, but the backend now writes it asynchronously through a queue-backed event. The callback handler does the right thing for a fresh login but not for a retried login after a back-button flow. A billing upgrade step appends return_url incorrectly, causing a second redirect to strip state. A magic-link email opens in a mobile in-app browser that shares less storage than desktop Chrome. None of these issues are really “inside” one function.
They happen at the handoff points.
That is the core quality problem in modern software. Not whether functions return correct values, but whether systems preserve intent across transitions.
Why this is getting worse in AI-assisted shipping
AI changes the distribution of mistakes.
A human engineer manually stitching together auth, billing, and notifications tends to feel some friction. They switch files, reconsider assumptions, and notice mismatched naming or missing state propagation because the work is slow enough to surface discomfort. An AI agent can generate all of it quickly and plausibly. It can produce callback handlers, frontend hooks, test fixtures, CI updates, and provider-specific config examples in minutes. That speed is useful. It is also dangerous.
The issue is not that AI writes uniquely bad code. The issue is that AI is very good at producing locally valid code. It tends to satisfy interfaces, imitate patterns, and cover obvious paths. That means it often strengthens the exact layer where teams already over-invest: code-level correctness.
Meanwhile the true source of breakage sits one level up:
- browser state persistence
- redirect ordering
- external provider timing
- callback race conditions
- environment configuration drift
- differences between mocked and real third-party behavior
- assumptions about eventual consistency that are not actually consistent enough
When more code is shipped faster, the number of cross-system edges grows. Traditional testing does not scale well to those edges because most of it is designed around deterministic local behavior.
That is why teams feel a strange disconnect today. Developer productivity is higher. CI/CD pipelines are faster. Test counts are up. Yet confidence is not improving proportionally.
Green pipelines are giving false confidence because they validate implementation artifacts, not critical user journeys.
Why current approaches fail
CI/CD validates build health, not product reality
CI/CD is essential. But many teams quietly expect it to answer a question it was never designed to answer.
A typical pipeline answers things like:
- did the code compile?
- did unit and integration suites pass?
- did linting and type checks pass?
- did the deploy artifact build?
- maybe: did a smoke test hit the homepage?
Those are table stakes. They are not user reliability.
A green pipeline says your repository produced a self-consistent artifact under lab conditions. It does not say a new user can sign up through Google on a preview deployment, verify email, upgrade a subscription, return to the app, and remain authenticated after a callback chain.
Most CI/CD setups also introduce blind spots by design:
- external services are mocked for speed and determinism
- secrets differ between CI and deployment environments
- browser tests use stable containers, not real end-user browsers
- callback URLs in test environments are simplified
- asynchronous operations are shortened or forced synchronous
- retries hide timing instability
These choices are rational. Without them, pipelines become slow and flaky. But teams often forget the tradeoff. The more you sanitize the environment, the less representative the workflow becomes.
That is how you end up “proving” that auth works while shipping a callback flow that loops only on Safari in a preview environment using HTTPS behind a platform-specific proxy.
Unit tests are too narrow for stateful workflows
Unit tests are excellent at protecting business logic and regression-prone transformations. They are poor tools for validating distributed user intent.
Take an OAuth login flow. A unit test can verify:
- the state parameter is generated
- the callback parser extracts
codeandstate - a token exchange function handles success and error responses
- a session object is serialized correctly
All useful.
But the actual workflow depends on more than that:
- was the original state stored in a browser-compatible way?
- did the cookie survive the provider redirect?
- did the callback hit the expected host?
- did middleware intercept and rewrite the request?
- was the session visible to the frontend before the next navigation?
- did the app redirect to onboarding before account creation finished?
Unit tests cannot capture those interactions without becoming fake integration environments, and when they try, they usually produce brittle simulations that still miss browser behavior.
Integration tests stop at the service boundary
Integration tests are often sold as the answer to unit test limitations. They are not enough either.
Most integration suites validate a service with real dependencies inside a controlled boundary: a database, a queue, a local HTTP service, maybe a test double for a provider. That helps catch schema drift and wiring mistakes. But many user-facing failures happen after the service boundary.
For example:
- your backend returns a valid 302, but the frontend immediately rewrites the route and loses query params
- the provider callback arrives correctly, but your CDN strips a header in preview only
- the session row is inserted, but the browser requests the next page before replication or cache invalidation completes
- the local integration test uses
localhost, but production uses subdomains that affect cookie scoping
These are system interaction failures. Not service failures.
Manual QA cannot keep up with combinatorial workflows
Manual QA still finds issues that automated suites miss, especially in auth and payments. But it does not scale to modern change velocity.
When AI-assisted shipping makes it easy to touch five systems in one PR, the number of meaningful workflow permutations explodes:
- first-time login vs returning login
- mobile vs desktop
- Chrome vs Safari
- preview vs staging vs production domain patterns
- verified vs unverified email
- billing active vs trial vs canceled
- provider timeout vs delayed webhook vs duplicate callback
No QA team can exhaustively explore that matrix before every merge. Even if they could, the result would still be transient because environment conditions and third-party behavior shift continuously.
Manual QA remains useful for exploratory work. It should not be your primary defense against handoff failures.
The core insight: test journeys, not just code paths
If the reliability problem lives in transitions, your testing strategy has to model transitions.
That means the fundamental artifact you validate before merge is not just a function or service. It is a user journey.
Examples:
- sign up with Google, complete onboarding, land in dashboard authenticated
- reset password from email link, set new password, resume intended destination
- upgrade plan through billing provider, return to app, see updated entitlements
- invite teammate by email, accept invite on another browser, join correct workspace
- start unauthenticated on a protected route, log in, preserve return path and state
These are not “end-to-end tests” in the simplistic sense of clicking around a UI. They are workflow assertions spanning browser state, external systems, and asynchronous completion.
The question is not: did the callback endpoint respond with 200?
The question is: after all redirects, retries, and side effects settle, is the user in the expected product state?
That is a different philosophy of testing.
A real OAuth failure, in code
Consider a common implementation pattern in JavaScript with Express.
jsapp.get('/auth/google/start', (req, res) => { const state = crypto.randomUUID(); res.cookie('oauth_state', state, { httpOnly: true, sameSite: 'lax', secure: true, maxAge: 10 * 60 * 1000, }); const redirectUri = `${process.env.APP_URL}/auth/google/callback`; const authUrl = new URL('https://accounts.google.com/o/oauth2/v2/auth'); authUrl.searchParams.set('client_id', process.env.GOOGLE_CLIENT_ID); authUrl.searchParams.set('redirect_uri', redirectUri); authUrl.searchParams.set('response_type', 'code'); authUrl.searchParams.set('scope', 'openid email profile'); authUrl.searchParams.set('state', state); res.redirect(authUrl.toString()); }); app.get('/auth/google/callback', async (req, res) => { if (req.query.state !== req.cookies.oauth_state) { return res.status(400).send('Invalid state'); } const tokens = await exchangeCodeForTokens(req.query.code); const user = await upsertUserFromGoogle(tokens); const session = await createSession(user.id); res.cookie('session', session.id, { httpOnly: true, sameSite: 'lax', secure: true, }); res.redirect('/dashboard'); });
Nothing obviously wrong there.
Now introduce a frontend middleware layer that protects /dashboard and redirects unauthenticated users to /login. It checks for session existence using an API call that hits a replica lagging behind the write path. On callback, the browser receives the session cookie and follows the redirect immediately. The middleware checks auth before the session is visible through that path, decides the user is not authenticated, and sends them back to /login. The login page detects an existing provider account and initiates another redirect. Congratulations: you built a redirect loop with fully correct unit-tested functions.
A more defensive callback flow might look like this:
jsapp.get('/auth/google/callback', async (req, res) => { if (!req.query.code || req.query.state !== req.cookies.oauth_state) { return res.status(400).send('Invalid OAuth callback'); } const tokens = await exchangeCodeForTokens(req.query.code); const user = await upsertUserFromGoogle(tokens); const session = await createSession(user.id); res.cookie('session', session.id, { httpOnly: true, sameSite: 'lax', secure: true, path: '/', }); res.clearCookie('oauth_state', { path: '/' }); // Land on a stabilization route that waits for session visibility. res.redirect(`/auth/complete?next=${encodeURIComponent('/dashboard')}`); }); app.get('/auth/complete', async (req, res) => { res.send(` <html> <body> <script> async function finalize() { for (let i = 0; i < 10; i++) { const r = await fetch('/api/session', { credentials: 'include' }); if (r.ok) { window.location.assign(${JSON.stringify('/dashboard')}); return; } await new Promise(resolve => setTimeout(resolve, 200)); } window.location.assign('/login?error=session_not_ready'); } finalize(); </script> </body> </html> `); });
This is not pretty, but it acknowledges reality: asynchronous systems need workflow-level stabilization points.
The point is not that every auth flow should poll. The point is that implementation correctness alone does not guarantee journey correctness.
The test that actually matters
The right place to catch the redirect-loop class of bug is a browser workflow test.
Using Playwright:
tsimport { test, expect } from '@playwright/test'; test('user can sign in with Google and reach dashboard once', async ({ page, context }) => { const redirects: string[] = []; page.on('framenavigated', frame => { if (frame === page.mainFrame()) { redirects.push(frame.url()); } }); await page.goto('/login'); await page.getByRole('button', { name: 'Continue with Google' }).click(); // In a real environment, this would use a test auth tenant or provider sandbox. await page.waitForURL('**/dashboard', { timeout: 30000 }); await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible(); await expect(async () => { const session = await context.cookies(); expect(session.some(c => c.name === 'session')).toBeTruthy(); }).toPass(); const dashboardHits = redirects.filter(url => url.includes('/dashboard')).length; const loginHits = redirects.filter(url => url.includes('/login')).length; expect(dashboardHits).toBeGreaterThanOrEqual(1); expect(loginHits).toBeLessThanOrEqual(1); });
That is still a simplified example. In production-grade workflow testing, you should also assert:
- no repeated callback hits
- state parameter is cleared or invalidated after use
- authenticated landing page loads expected personalized data
- session survives one refresh
- intended return path is preserved
- analytics or audit event marks completion once, not multiple times
If you only test that /auth/google/callback returns 302, you miss the failure. If you test the actual journey, you catch it before merge.
Python example for callback robustness
The same class of issue exists outside Node stacks. Here is a simplified FastAPI callback handler.
pythonfrom fastapi import FastAPI, Request, Response, HTTPException from fastapi.responses import RedirectResponse app = FastAPI() @app.get('/auth/callback') async def auth_callback(request: Request): code = request.query_params.get('code') state = request.query_params.get('state') stored_state = request.cookies.get('oauth_state') if not code or not state or state != stored_state: raise HTTPException(status_code=400, detail='Invalid callback') tokens = await exchange_code(code) user = await upsert_user(tokens) session_id = await create_session(user.id) response = RedirectResponse(url='/auth/complete?next=/dashboard', status_code=302) response.set_cookie( key='session', value=session_id, httponly=True, secure=True, samesite='lax', path='/' ) response.delete_cookie('oauth_state', path='/') return response
And a Playwright Python journey test:
pythonfrom playwright.sync_api import sync_playwright, expect def test_google_login_journey(live_server_url: str): with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() urls = [] page.on('framenavigated', lambda frame: urls.append(frame.url) if frame == page.main_frame else None) page.goto(f'{live_server_url}/login') page.get_by_role('button', name='Continue with Google').click() page.wait_for_url('**/dashboard', timeout=30000) expect(page.get_by_role('heading', name='Dashboard')).to_be_visible() login_hits = len([u for u in urls if '/login' in u]) callback_hits = len([u for u in urls if '/auth/callback' in u]) assert login_hits <= 1 assert callback_hits == 1 browser.close()
Again, the value here is not the exact assertion syntax. It is the model: test the workflow with browser state and real redirects in play.
Environment-specific failures are not edge cases anymore
One of the most expensive misconceptions in debugging modern delivery is calling these issues “edge cases.” They are not. They are environment-defined behavior.
Preview environments, staging systems, and production often differ in exactly the dimensions that break workflows:
- domain and subdomain shape
- HTTPS termination path
- cookie security settings
- OAuth callback registration
- CORS origin lists
- CDN and proxy behavior
- secret rotation timing
- webhook ingress and replay behavior
- provider sandbox vs live tenant differences
A test suite that runs only on localhost or in a hermetic CI container cannot tell you much about those differences.
This is why teams need workflow checks against deployed environments, not just source code. Not every test has to run against every preview deployment. But critical journeys absolutely should run where the real networking, cookies, and callback URLs exist.
A simple GitHub Actions job for post-deploy workflow verification might look like this:
yamlname: workflow-checks on: deployment_status: jobs: critical-journeys: if: github.event.deployment_status.state == 'success' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 - run: npm ci - run: npx playwright install --with-deps - name: Run auth and billing journeys env: BASE_URL: ${{ github.event.deployment_status.environment_url }} GOOGLE_TEST_USER: ${{ secrets.GOOGLE_TEST_USER }} GOOGLE_TEST_PASS: ${{ secrets.GOOGLE_TEST_PASS }} BILLING_SANDBOX_KEY: ${{ secrets.BILLING_SANDBOX_KEY }} run: npx playwright test tests/journeys/auth.spec.ts tests/journeys/billing.spec.ts
That pipeline pattern matters because it shifts validation from “did the code compile” to “does the deployed product complete critical actions.”
What to mock, and what not to mock
Teams usually get into trouble at two extremes.
One extreme mocks everything. The suite is fast and stable but blind to handoff failures.
The other extreme refuses to mock anything. The suite becomes slow, flaky, expensive, and hard to debug.
The practical approach is selective realism.
You should strongly consider using real or sandboxed versions of systems that define workflow boundaries:
- OAuth providers or dedicated test tenants
- billing providers in sandbox mode
- email capture systems for magic links and invitations
- deployed browser environment with real cookies and redirects
You can still mock lower-level or orthogonal dependencies:
- analytics delivery
- non-critical third-party enrichment APIs
- background jobs unrelated to the asserted outcome
- expensive recommendation systems
The principle is simple: do not mock the handoff you are trying to trust.
If your product promise depends on a redirect completing through a third-party provider and back into your app, that boundary should appear in at least one pre-merge or pre-release workflow check.
Tools comparison: what each layer catches
Here is the blunt version of the tool stack.
Unit tests
Good for:
- pure business logic
- serialization and parsing
- validation rules
- utility regressions
Bad at:
- browser state
- redirects
- provider timing
- environment config drift
Integration tests
Good for:
- service wiring
- database and queue behavior
- API contracts
- local dependency interactions
Bad at:
- cross-domain browser behavior
- callback URLs
- multi-system asynchronous workflows
- CDN or proxy effects
Traditional end-to-end tests with heavy mocking
Good for:
- stable UI regression checks
- app navigation under controlled conditions
- quick smoke coverage
Bad at:
- real auth and billing handoffs
- redirect and cookie realities
- third-party timing issues
Deployed workflow tests with selective realism
Good for:
- critical product journeys
- auth, billing, email, invitation, and callback correctness
- environment-specific debugging
- release confidence
Bad at:
- broad exhaustive coverage
- ultra-fast feedback loops
- low-maintenance operation if poorly designed
This is why the right strategy is layered, not ideological. Keep unit and integration tests. Keep CI/CD fast. But add a thin, high-value layer of workflow tests that exercise the user journeys where production failures actually happen.
Actionable practices that reduce this class of bugs
1. Define critical journeys explicitly
Most teams know their important workflows informally. That is not enough.
Write them down as merge-blocking product actions, for example:
- new user signs up with Google and lands in workspace
- user verifies email and is returned to original destination
- user upgrades plan and entitlements update within 30 seconds
- invited user accepts invitation and joins correct org
- logged-out user hitting protected route returns there after login
If a journey matters to revenue, activation, or access, it deserves first-class test status.
2. Add browser-level assertions around transitions
Do not stop at “page loaded.” Assert the transitions themselves.
Check things like:
- callback called exactly once
- no loop back to login
- session cookie present after redirect
- query params preserved or intentionally cleared
- final page reflects authenticated state
- refresh does not lose session
These assertions turn vague reliability concerns into deterministic checks.
3. Run workflow tests against real deployments
Local and CI-container execution is useful, but critical journeys should also run against deployed preview or staging environments where:
- real domains exist
- HTTPS is real
- proxy and CDN behavior is present
- provider callbacks are configured normally
This catches a category of environment-specific failures that code-only testing never sees.
4. Instrument handoff points for debugging
A major reason these failures are expensive is that logs often stop at service boundaries. Add tracing around workflow transitions.
For auth, log and correlate:
- auth start timestamp
- generated state ID or correlation ID
- callback received timestamp
- session created timestamp
- redirect target
- middleware auth decision on first protected route hit
For billing, log:
- checkout session creation
- provider redirect start
- return callback
- webhook receipt
- entitlement update completion
- UI fetch of entitlements
Good debugging starts with enough structure to reconstruct the journey.
5. Treat retries as a smell, not a solution
If your browser workflow test only passes with generous retries, that is information. Do not bury it.
Retries can be fine around obviously flaky external sandboxes, but when they hide your own race conditions, they reduce signal. A flaky auth journey is not “just test flake.” It is often a production flake with better marketing.
6. Add stabilization routes where asynchronous state needs to settle
Purists dislike explicit stabilization steps because they feel inelegant. Production does not care about elegance.
If there is a known delay between callback completion and session or entitlement visibility, create a clear route or state machine that handles it intentionally instead of hoping the next page load lines up correctly.
That can mean:
- a
/auth/completepage - a
/billing/processingpage - polling with timeout and user messaging
- idempotent callback handling with deduplication
The key is to design for eventual consistency instead of pretending everything is synchronous.
7. Verify behavior across at least one hostile browser
If you only test auth flows in Chromium on localhost, you are under-testing. Safari, mobile webviews, and privacy-restricted browsers expose cookie and storage assumptions quickly.
You do not need full matrix coverage for every PR, but you should maintain at least one regular workflow suite in a stricter browser environment.
8. Build idempotency into callbacks and return flows
OAuth callbacks, billing returns, and webhook-driven state transitions should tolerate duplicate delivery and partial completion.
Common failure pattern: first callback creates session but frontend aborts; second callback sees unexpected existing state and throws. Better pattern: callbacks can safely detect prior completion and route the user forward without duplication or error.
9. Separate merge confidence from release confidence
Not every workflow test has to block every PR. But teams should be explicit about which checks provide fast merge confidence and which provide release confidence.
A practical split might be:
- PR: fast unit, integration, and a few lightweight browser journey checks
- post-deploy preview: critical auth and onboarding workflow checks
- pre-release or continuous staging: billing, email, invitation, and recovery flows
That structure respects developer productivity without pretending all confidence can come from one phase.
10. Review PRs by journey impact, not file count
This matters more in AI-assisted development.
A PR with small diffs across auth, frontend routing, billing, and email may be riskier than a large refactor confined to one service. Reviewers should ask:
- which user journey crosses these changes?
- what state is handed off between systems?
- what callback or redirect assumptions changed?
- what deployed-environment test proves this still works?
That is a more realistic risk model than counting lines changed.
The new definition of quality
For a long time, engineering quality was approximated by code quality plus test coverage plus successful CI/CD. That approximation worked reasonably well when most failures lived inside the application boundary and changes were hand-authored at a pace that limited cross-system complexity.
That world is gone.
Modern products depend on browser behavior, third-party providers, asynchronous state propagation, and environment-specific infrastructure. AI-assisted development accelerates the creation of these cross-system workflows, which means teams can now produce integration surface area faster than traditional testing can meaningfully validate it.
So the quality bar has to move.
The new question is not, “Did the PR checks pass?”
The new question is, “Can the user complete the journey, in a real browser, through real handoffs, on a deployed environment, before we merge or release?”
That is a harder question. It is also the only one the user cares about.
When OAuth redirects twice, no one is impressed that your callback parser had 100% unit test coverage.
They just know the product broke.
If you want stronger reliability in this era, stop treating workflow testing as an optional top layer. It is the layer that tells you whether the software actually works. Everything below it is still necessary. None of it is sufficient.
That is the uncomfortable truth behind a lot of green pipelines right now: the PR looked fine because the tests were looking at the code. The failure was waiting in the journey.
