Your PR Passed Review. The Migration Still Broke Checkout.

A checkout outage rarely starts with a syntax error. It starts with a pull request that looked responsible.

The migration was reviewed. The types compiled. CI stayed green. Unit tests passed. Staging looked healthy. The deploy even worked for new customers created after the release. Then real users with older accounts started failing at the payment step because one service expected backfilled fields, another still read the legacy shape, and a long-running migration held locks just long enough to turn a routine traffic spike into a conversion drop.

This is the kind of failure modern teams keep rediscovering, especially now that more code is generated quickly and reviewed at a higher level. AI can produce a migration, backfill script, ORM update, API changes, and tests in minutes. It can also encode bad assumptions with remarkable efficiency. Reviewers see coherent code. CI validates isolated logic. Staging often runs against fresh fixtures or thin snapshots. None of that proves your system survives the state transition from yesterday’s production data to tomorrow’s schema.

That is the actual problem: we are still verifying code diffs when the real risk lives in stateful transitions.

If your application serves real users over time, migrations are not code-only events. They are system events. They change the shape of data, the timing of reads and writes, the behavior of indexes, the assumptions in downstream services, and the reliability of user workflows that depend on records created months or years ago. If your testing strategy does not exercise those transitions end to end, your green pipeline is mostly theater.

The migration failure pattern nobody wants to own

Most migration incidents follow a familiar path.

First, someone introduces a schema change that appears safe in isolation:

• split one column into two
• make a nullable field required
• move enum values around
• rename a status used by multiple services
• change a default value in the ORM
• add a unique index to “clean up” data integrity
• backfill derived fields for performance

Then the implementation gets spread across layers:

• ORM schema update
• application read/write logic
• background backfill job
• API contract changes
• analytics or event payload changes
• admin tooling adjustments
• integration updates in adjacent services

Every individual step can look reasonable. That is why review often misses it.

The failure usually appears only under one or more real-world conditions:

• old records don’t match new assumptions
• backfill runs slower than expected
• services deploy out of order
• one path dual-writes while another still single-reads
• lock contention delays writes during traffic
• a supposedly optional field is actually required in one workflow
• stale caches preserve pre-migration shapes
• rollback leaves mixed-state data behind

Checkout is a perfect victim because it crosses everything at once: user identity, cart state, pricing, inventory, payment methods, tax records, shipping preferences, fraud signals, and order creation. You can pass a hundred unit tests and still fail a single old customer trying to reorder with a saved card linked to a record shape that no longer exists.

That is why “the migration broke checkout” is not a weird edge case. It is the expected result of pretending stateless verification is enough for stateful systems.

Why PR review fails on migrations

Code review is good at spotting local mistakes. Migrations fail because the worst bugs are non-local.

A reviewer can read this and approve it:

ts
// before
model Order {
  id              String   @id @default(cuid())
  paymentStatus   String?
}

// after
model Order {
  id              String   @id @default(cuid())
  paymentState    String   @default("pending")
}

And maybe they also see the application logic updated:

ts
export function isPaid(order: Order) {
  return order.paymentState === "captured";
}

Looks fine. But what exactly happens to existing rows where paymentStatus was null, or authorized, or paid, or misspelled from some old bug, or set by a legacy admin workflow? What happens if one service maps paid -> captured while another maps paid -> settled? What happens if the backfill only covers 98% of rows before the app switch flips?

These are not style issues. They are transition issues. Review cannot prove them away because they depend on production history, deployment ordering, and runtime behavior under load.

AI-generated code makes this worse in a specific way: it often fills in the happy-path migration logic convincingly. It will produce the schema change, a backfill script, and a few tests that create clean fixtures matching the new world. That gives reviewers something polished to bless. But polished is not the same as verified.

The more productive your code generation becomes, the more your reliability strategy has to move downstream into realistic verification.

Why CI stays green while production breaks

CI/CD pipelines mostly validate one thing: can this version of the code build and pass the tests we thought to write?

That matters. It is just not enough.

In migration-heavy changes, green CI often hides four specific blind spots.

1. Fresh databases lie

Most test suites run against a newly created schema. That means they validate the end state, not the path to get there.

If your migration creates a non-null column with a default, a fresh test database will happily create rows in the new shape. It tells you nothing about millions of existing rows with legacy nulls, malformed values, or half-populated related records.

A dangerous anti-pattern looks like this:

yaml
# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx prisma migrate deploy
      - run: npm test

This validates that migrations can create the latest schema from scratch. Useful, but deeply incomplete.

You also need to test upgrading from an old schema with old data.

2. Unit tests isolate the bug out of existence

Unit tests usually mock repositories, construct idealized objects, or build fixtures that reflect current types. That strips away the legacy conditions where migration bugs actually live.

Example:

ts
import { expect, test } from 'vitest';
import { isPaid } from './payments';

test('captured orders are paid', () => {
  expect(isPaid({ paymentState: 'captured' } as any)).toBe(true);
});

This is not wrong. It is just almost irrelevant to migration risk.

The dangerous case is the real order row created 14 months ago with payment_status = 'paid' and no payment_state, being read by a service version deployed in region B while the backfill in region A is still running.

3. Staging almost never matches production history

Teams say “it worked in staging” as if that means anything. Usually it means:

• fewer records
• cleaner records
• newer records
• no malformed edge cases
• no realistic concurrency
• no long tail of customer states
• no stale caches or lagging replicas
• no actual load during migration windows

Staging helps validate infrastructure and smoke tests. It does not automatically validate upgrade realism.

4. CI/CD treats deployment as a single point in time

Migrations are not one moment. They are a time interval where multiple versions of code and data coexist.

That interval includes:

• before schema change
• after schema change but before backfill
• during backfill
• after partial backfill
• after one service deploys
• before another service deploys
• during rollback

If your pipeline only tests “old version” and “new version,” you are missing the most dangerous states in between.

Why manual QA doesn’t save you either

QA is valuable, but migrations routinely escape it because the trigger conditions are hidden in data lineage, not obvious UI behavior.

A tester can execute checkout with a new account and see success.

The broken path may require all of these at once:

• account created before March
• saved payment token from old provider
• tax exemption flag copied from CRM import
• cart containing subscription renewal plus physical item
• address record missing a normalized country code
• background backfill not yet processed customer profile

Nobody finds that manually unless they intentionally construct and preserve pre-migration states. Most teams do not.

This is the shift in mindset that matters: you are not only testing features. You are testing historical continuity.

The core insight: verify transitions, not just destinations

A migration is successful only if real user workflows still complete across the transition from old state to new state.

That means your verification strategy has to answer questions like:

• Can the old application version run against the new schema safely?
• Can the new application version tolerate old records before backfill completes?
• Do mixed-version services behave consistently during rollout?
• Can we replay representative production data through the migration?
• Do critical workflows succeed before, during, and after the backfill?
• If the migration is interrupted halfway, what still works?
• If we roll back application code but not schema, what breaks?

This is a very different standard from “tests passed.”

The right target is not line coverage or branch coverage. It is workflow continuity across upgrade paths.

For commerce, that means things like:

• returning customer can log in
• cart loads with old saved data
• shipping and tax calculate correctly
• payment authorization succeeds
• order persists in the new schema
• confirmation email and downstream fulfillment still trigger

For SaaS, it might mean:

• existing tenant can sign in
• legacy configuration loads
• billing portal still reflects entitlements
• audit events still serialize
• old API tokens still work

You want action-level checks that prove the business still functions while the system changes underneath it.

A concrete migration example: split a payment status field

Suppose you are moving from one overloaded field to two explicit ones.

Before:

sql
CREATE TABLE orders (
  id UUID PRIMARY KEY,
  payment_status TEXT,
  total_cents INTEGER NOT NULL
);

After:

sql
ALTER TABLE orders ADD COLUMN payment_state TEXT;
ALTER TABLE orders ADD COLUMN payment_provider_state TEXT;

Backfill plan:

sql
UPDATE orders
SET
  payment_state = CASE
    WHEN payment_status IN ('paid', 'captured') THEN 'captured'
    WHEN payment_status = 'authorized' THEN 'authorized'
    ELSE 'pending'
  END,
  payment_provider_state = payment_status
WHERE payment_state IS NULL;

Application code after migration:

ts
export type Order = {
  paymentState: 'pending' | 'authorized' | 'captured' | 'failed';
  paymentProviderState?: string | null;
};

export function canCheckoutComplete(order: Order) {
  return order.paymentState === 'authorized' || order.paymentState === 'captured';
}

Looks disciplined. But now consider the real failure cases:

• some rows contain 'PAID', 'settled', or empty string
• one service still writes payment_status
• another service starts reading only payment_state
• backfill takes hours and is not atomic
• old admin tools patch orders directly
• analytics pipeline still expects payment_status

Your test strategy has to force these states into existence.

Build migration tests around realistic old data

The first practical change is to stop testing migrations only from pristine fixtures. Create pre-migration datasets that reflect actual production history.

In Python, a seed script might intentionally generate dirty legacy states:

python
import psycopg2
from uuid import uuid4

legacy_statuses = [
    'paid',
    'captured',
    'authorized',
    'PAID',
    '',
    None,
    'settled'
]

conn = psycopg2.connect("dbname=app user=postgres password=postgres host=localhost")
cur = conn.cursor()

cur.execute("DROP TABLE IF EXISTS orders")
cur.execute("""
CREATE TABLE orders (
  id UUID PRIMARY KEY,
  payment_status TEXT,
  total_cents INTEGER NOT NULL
)
""")

for i, status in enumerate(legacy_statuses):
    cur.execute(
        "INSERT INTO orders (id, payment_status, total_cents) VALUES (%s, %s, %s)",
        (str(uuid4()), status, 1000 + i)
    )

conn.commit()
cur.close()
conn.close()

This kind of seed is more useful than a hundred clean factory objects because it captures the weirdness your system has accumulated.

Even better: derive sanitized snapshots from production patterns.

Not raw production data. Representative data.

That means preserving distributions and edge shapes such as:

• record ages
• null frequency
• duplicate patterns
• historical enum variants
• orphaned relationships
• large tenants versus small tenants
• accounts mid-workflow

If you only test with perfect rows, you are not testing migration risk. You are testing documentation.

Test upgrade paths in CI, not just final state

Your pipeline should explicitly simulate the sequence:

1. start from old schema
2. insert old data
3. run migration
4. run backfill, or partially run it
5. execute critical user workflows
6. validate downstream side effects

Here is a more realistic GitHub Actions structure:

yaml
name: migration-verification

on:
  pull_request:

jobs:
  upgrade-path-test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U postgres"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - run: npm ci
      - run: pip install psycopg2-binary

      - name: Prepare old schema and seed legacy data
        run: |
          psql postgresql://postgres:postgres@localhost:5432/postgres -f db/old_schema.sql
          python scripts/seed_legacy_orders.py

      - name: Run new migration
        run: psql postgresql://postgres:postgres@localhost:5432/postgres -f db/migrations/2026_01_payment_state.sql

      - name: Run partial backfill
        run: node scripts/backfill_payment_state.js --limit=3

      - name: Run workflow tests against mixed-state data
        run: npm run test:migration-workflows

      - name: Complete backfill
        run: node scripts/backfill_payment_state.js

      - name: Run workflow tests after full backfill
        run: npm run test:migration-workflows

Notice what changed:

• old schema is created explicitly
• old data is seeded intentionally
• partial backfill is treated as a first-class state
• workflow tests run before and after completion

That is what transition-aware CI/CD looks like.

Use Playwright to verify user workflows, not just APIs

If checkout is the thing that breaks, test checkout.

This is where browser automation becomes far more valuable than another layer of unit tests. You want to prove that the workflow visible to a user still completes while the underlying data model changes.

Example with Playwright:

ts
import { test, expect } from '@playwright/test';

test('legacy customer can complete checkout during partial payment backfill', async ({ page, request }) => {
  const seed = await request.post('http://localhost:3000/test/seed-legacy-checkout-state', {
    data: {
      customerAgeDays: 540,
      paymentStatus: 'paid',
      hasSavedCard: true,
      cartType: 'mixed'
    }
  });

  const { email, password } = await seed.json();

  await page.goto('http://localhost:3000/login');
  await page.getByLabel('Email').fill(email);
  await page.getByLabel('Password').fill(password);
  await page.getByRole('button', { name: 'Sign in' }).click();

  await page.goto('http://localhost:3000/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();

  await expect(page.getByText('Review your order')).toBeVisible();
  await page.getByRole('button', { name: 'Place order' }).click();

  await expect(page.getByText('Order confirmed')).toBeVisible();
});

This matters because it validates the path engineers actually care about: can a user complete the workflow?

Even better, pair UI checks with database assertions and trace collection.

ts
test('order persists with migrated payment fields', async ({ page, request }) => {
  // ... login and place order ...

  const response = await request.get('http://localhost:3000/test/last-order');
  const order = await response.json();

  expect(order.payment_state).toBeTruthy();
  expect(['authorized', 'captured']).toContain(order.payment_state);
  expect(order.legacy_payment_status).toBeNull();
});

Now your workflow testing is also doing targeted debugging when it fails.

Make mixed-version compatibility explicit

A lot of migration failures come from assuming all services switch at once. They do not.

If service A writes the new field while service B still reads the old one, or vice versa, you need compatibility windows.

A safe rollout usually follows this pattern:

1. add new schema in backward-compatible form
2. deploy code that dual-reads and dual-writes
3. backfill old data
4. switch reads to new field
5. remove old writes
6. remove old field later

Example in JavaScript:

ts
export function readPaymentState(row: any): string {
  return row.payment_state ?? mapLegacyStatus(row.payment_status);
}

export function writePaymentUpdate(input: { providerState: string }) {
  return {
    payment_state: normalizeProviderState(input.providerState),
    payment_status: input.providerState,
    payment_provider_state: input.providerState,
  };
}

function mapLegacyStatus(status?: string | null): string {
  switch ((status || '').toLowerCase()) {
    case 'paid':
    case 'captured':
    case 'settled':
      return 'captured';
    case 'authorized':
      return 'authorized';
    case 'failed':
      return 'failed';
    default:
      return 'pending';
  }
}

This code is not pretty. It is reliable.

Migration safety often looks uglier than clean-slate design because it acknowledges time.

Your tests should also exercise old/new combinations between services. If you have separate checkout, billing, and order services, run contract and workflow tests with staggered versions. Otherwise you are validating an imaginary deployment model.

Add debugging hooks for migration verification

Migration failures are hard to diagnose after the fact because the bad state is transient. By the time somebody investigates, the backfill may have progressed, caches may have expired, and the failing rows may be overwritten.

So build debugging into the verification process.

Good hooks include:

• logs tagged with migration version
• metrics for read fallbacks to legacy fields
• count of unmigrated rows over time
• lock wait timings during schema changes
• traces around critical workflows during migration windows
• sampled payload diffs between old and new representations

Example instrumentation:

ts
if (!row.payment_state && row.payment_status) {
  logger.warn('legacy_payment_fallback_used', {
    orderId: row.id,
    paymentStatus: row.payment_status,
    migration: '2026_01_payment_state'
  });
  metrics.increment('migration.payment_state.legacy_fallback');
}

In a healthy rollout, this metric should trend toward zero as the backfill completes. If it spikes after app deployment, your assumptions are wrong.

This is where debugging and testing meet. Good migration verification does not just fail fast; it fails informatively.

Tool comparison: what helps and what doesn’t

Different tools answer different parts of the problem.

Unit test frameworks: fast, precise, insufficient alone

Vitest, Jest, Pytest, JUnit—all useful for mapping rules, serializers, and compatibility functions.

Use them for:

• status mapping logic
• fallback behavior
• idempotency of migration helpers
• edge-case parsing

Do not trust them to validate:

• realistic old data
• mixed-service rollout states
• lock contention
• user workflow continuity

Database migration tools: necessary, not self-verifying

Prisma Migrate, Alembic, Flyway, Liquibase, Rails migrations—they manage change. They do not prove the change is safe for your production history.

Use them for:

• repeatable schema evolution
• version tracking
• execution ordering

Do not confuse successful execution with safe rollout.

Playwright and browser automation: best for business-critical workflows

Playwright is extremely effective when the risk is “does checkout still work?”

Use it for:

• end-to-end workflow verification
• realistic user action coverage
• trace capture for debugging
• validating UI plus API plus persistence together

Its weakness is setup complexity and speed. That is fine. You do not need 500 migration workflow tests. You need the 10 that protect the business.

Ephemeral environments: useful if they preserve old state

Preview environments and ephemeral stacks are great only if they can be booted from realistic pre-migration data states.

A fresh environment with fresh data is just staging with better branding.

Use ephemeral environments for:

• replaying upgrade paths
• testing service version skew
• collecting traces in isolation

Production canaries and shadow checks: excellent final safety net

The best teams also verify in production safely:

• canary deploy to a small percentage
• read-only shadow workflow checks
• compare old/new representation outputs
• monitor fallback metrics and key conversion funnels

This is not an excuse to skip CI verification. It is the final layer when pre-production realism still has gaps.

Actionable practices that actually reduce migration incidents

If you want a practical operating model, start here.

1. Classify migrations by workflow risk, not just schema size

A one-column change on a checkout table may be higher risk than a large analytics migration.

Tag migrations based on:

• touched workflows
• data age involved
• cross-service dependencies
• backfill duration
• rollback complexity

This helps allocate real verification effort where it matters.

2. Require a transition plan in every risky PR

For schema and data changes, the PR should include:

• old shape
• new shape
• compatibility strategy
• backfill strategy
• deployment order
• rollback plan
• workflow verification plan

If a change cannot explain how old data behaves during rollout, it is not ready.

3. Maintain representative legacy fixtures or snapshots

Keep versioned datasets that mimic important historical states.

Examples:

• customers from before major billing provider swap
• orders before tax logic rewrite
• accounts with partially migrated preferences
• tenants created before new identity model

These become core assets for testing, not ad hoc debugging artifacts.

4. Test partial backfills intentionally

Most teams test before and after. Real incidents happen during.

Run checks with:

• 0% backfilled
• 10% backfilled
• 50% backfilled
• interrupted job state
• rerun after partial failure

That exposes assumptions hidden in “eventual consistency” language.

5. Protect critical workflows with end-to-end migration checks

Pick the workflows that pay the bills:

• checkout
• sign-up
• invoice payment
• password reset
• provisioning
• API token auth

Then create dedicated migration-aware tests around them. This is where developer productivity gains are real: fewer blind deployments, faster debugging, and less emergency data surgery.

6. Verify rollback posture explicitly

Many rollbacks are fake. You can revert code, but the data shape has already moved.

Test questions like:

• can old code read new rows?
• can old workers process partially migrated records?
• what if schema additions remain but writes revert?

If rollback is impossible, say so and treat deploys accordingly.

7. Instrument migration-specific health signals

Do not watch only CPU and error rates. Watch state transition indicators:

• percent of records migrated
• fallback reads from legacy fields
• workflow completion rate by record age
• write latency on touched tables
• lock waits and deadlocks
• mismatch counts between old and new fields

These tell you whether the migration is safe in ways generic dashboards cannot.

8. Assume AI-generated migration code is plausible, not trustworthy

This is the practical stance teams need now.

AI can accelerate boilerplate, mappings, backfill scripts, and compatibility layers. Good. Use it.

But generated code tends to optimize for local completeness, not historical truth. It will often miss:

• undocumented legacy values
• race conditions during rollout
• operational constraints like lock duration
• mixed-version service behavior
• rollback realities

So the rule should be simple: generated migration code must be verified against realistic transition states before anyone trusts it.

What a better review standard looks like

For risky migrations, a strong review is not “LGTM, tests passed.”

A strong review asks:

• Where does old data come from in tests?
• What happens during partial backfill?
• How do old and new service versions coexist?
• Which workflows prove this is safe?
• What metrics will detect bad assumptions quickly?
• What is the rollback posture after schema change and after backfill?

That is a much better engineering standard than arguing over whether a migration function is elegant.

The hard truth is that many teams have better review discipline for naming than for state transitions. That is upside down.

Conclusion

Your PR can be well written, reviewed, and green in CI/CD and still break the business the moment it touches old data.

That is not because testing is useless. It is because most testing strategies are aimed at code correctness, while migration failures are usually failures of historical compatibility and system timing.

If AI is helping your team ship more code, this matters even more. More generated changes mean more plausible migrations, more review throughput, and more chances to ship assumptions nobody truly validated. The answer is not slower development. It is better verification.

Test the transition.

Replay realistic pre-migration data.

Run checks during partial backfills.

Exercise the workflows users actually depend on.

Instrument the rollout so debugging is possible while the state is changing.

That is what real reliability looks like in modern systems. Not green unit tests. Not a clean diff. Not staging on fresh data.

If the system changes underneath the user, your testing has to follow the user through the change.

Otherwise the next checkout outage is already sitting in an approved PR.