A team lets an AI agent clean up an aging billing schema. The pull request looks excellent. It renames columns, removes a join table nobody likes, backfills a few fields, updates the ORM models, and rewrites the reporting query. CI passes. Unit tests pass. Type checks pass. The migration script runs in a fresh test database. The PR gets merged.

Two hours later, support starts seeing tickets. Existing customers can still log in, but some cannot update payment methods. A subset of invoices render with blank totals. Finance exports are missing records created before last year. Checkout still works for new users, which makes the outage harder to spot. Engineering rolls back the application code, but the rollback does nothing for the already-migrated data. The dashboard is green. Production is not.

That is the new failure mode in AI-assisted delivery.

It is not that agents write uniquely bad migrations. Human engineers have always broken data. The difference is speed and confidence. AI can produce schema refactors, query rewrites, and migration plans faster than most teams can reason through their effects. It also generates the artifacts that make a pull request feel safe: updated tests, passing linters, plausible rollback notes, and a tidy summary of what changed. Traditional PR checks were already weak at validating stateful changes. With agent-written backend changes, they become dangerously reassuring.

The problem is simple: most CI proves the code compiles and tests pass against synthetic state. Very little of it proves that the product still works after real data has been transformed.

If your database migration changes how state is represented, then verifying the diff is not enough. You have to verify behavior against migrated state. Can a user sign up? Can an existing user check out? Can search still find old records? Can an admin edit a legacy account? Can a refund run on an order created before the migration? Those are not code-path questions. They are workflow questions.

The dangerous comfort of a green pipeline

Most engineering teams have been trained to trust green checks. A healthy CI/CD pipeline is supposed to reduce risk, increase developer productivity, and stop regressions before merge. In stateless code changes, that often works well enough. For stateful changes, especially migrations, green checks can be misleading.

A typical migration PR may pass all of the following:

application builds
unit tests
integration tests against seeded fixtures
migration runs against an empty or fresh database
static analysis
contract tests
smoke deploy in ephemeral environment

And still fail in production for reasons none of those checks model.

Why? Because state changes create a gap between what code expects and what data actually looks like. CI usually tests one side of the system at a time:

old code against old fixtures
new code against newly seeded fixtures
migration scripts against idealized data

Production is something else entirely:

new code against messy old data that has been transformed imperfectly
background jobs racing with online traffic during rollout
partial backfills
records with historical nulls and one-off fixes
foreign keys that were never truly enforced
strings that should have been enums
data written by previous versions, scripts, imports, and bugs

The migration “worked” in CI because CI verified a story where the database was obedient. Production fails because the database is archaeological.

That gap matters more now because AI agents are very good at making local changes coherent. They update model code, rewrite query builders, adjust serializers, regenerate tests, and leave the repository internally consistent. What they do not know, unless you force the issue, is whether that consistency survives contact with ten years of accumulated state.

Why fixture-based tests miss production-shaped data

Fixture-based testing is useful. It is also one of the main reasons teams overestimate migration safety.

Most test fixtures are too clean, too small, too recent, and too intentional. They represent the data engineers wish they had, not the data they actually operate.

A fixture might define three users, one canceled order, and a couple of invoice rows. Production has:

accounts created before validation rules existed
duplicate records from old import jobs
soft-deleted entities that still participate in joins
orphaned references hidden by application logic
JSON blobs with version drift
locale-specific formatting mistakes
timestamps from multiple time zones and historical DST bugs
records touched by support scripts and manual SQL
rows created by old services no longer in the codebase

A migration can be logically correct for fixture data and still break on real state.

Consider a simple AI-generated refactor. The agent replaces users.plan and users.billing_status with a normalized subscriptions table and updates application queries.

The unit tests may pass because fixture users all have valid plans and consistent status values. But production might contain users with:

plan = null but active invoices
billing_status = 'trialing' from an old experiment no longer represented
multiple overlapping subscriptions from previous bugs
grandfathered plans with names not in current enums
imported enterprise accounts that bypassed normal billing flow

The migration script may map 95% of records perfectly. The remaining 5% are where support tickets come from.

Here is a representative anti-pattern in test setup:

js
// tests/fixtures/users.js
export const users = [
  {
    id: 1,
    email: 'active@example.com',
    plan: 'pro',
    billing_status: 'active'
  },
  {
    id: 2,
    email: 'trial@example.com',
    plan: 'starter',
    billing_status: 'trial'
  }
]

This fixture only validates the happy path. It says nothing about the historical edge cases that make migrations risky.

Even if teams add more fixtures, they usually add the cases they already know. Real migration failures often come from unknown unknowns buried in production state distributions.

A better approach is to treat production-shaped data as a first-class test input. That does not require copying production recklessly into CI, but it does require building representative datasets from reality:

anonymized snapshots
sampled records from old cohorts
synthetic data generated from production profiles
edge-case corpora derived from observed anomalies
migration rehearsal environments seeded from recent state exports

In Python, you might at least codify state anomalies as explicit verification inputs:

python
legacy_accounts = [
    {
        "user_id": 101,
        "plan": None,
        "billing_status": "active",
        "has_open_invoice": True,
    },
    {
        "user_id": 102,
        "plan": "enterprise_2019",
        "billing_status": "grandfathered",
        "has_open_invoice": False,
    },
    {
        "user_id": 103,
        "plan": "pro",
        "billing_status": "trialing",
        "has_open_invoice": True,
    },
]

def test_migration_handles_legacy_billing_states(run_migration, fetch_subscription):
    run_migration(legacy_accounts)

    sub_101 = fetch_subscription(101)
    assert sub_101["status"] == "active"

    sub_102 = fetch_subscription(102)
    assert sub_102["plan_code"] == "enterprise_2019"

    sub_103 = fetch_subscription(103)
    assert sub_103["status"] in ["trial", "active"]

This is still not enough by itself, but it is directionally better than pretending migrations only need ideal fixture rows.

Why unit tests, integration tests, and QA still miss migration regressions

Teams usually respond to migration failures by asking for “more testing.” That is too vague to help. Different testing layers answer different questions, and most of them do not answer the one that matters here.

Unit tests

Unit tests are great for deterministic logic. They are weak at validating state transitions across schemas.

A unit test can prove a mapper converts old row shape to new row shape for a few examples. It cannot prove that checkout, refunds, search indexing, and admin edits still behave correctly when the whole system runs on transformed data.

Unit tests also tend to mirror the assumptions of the code author. When an AI agent writes both the migration and the unit tests, that problem gets worse. You are often validating the agent’s own interpretation of success.

Integration tests

Integration tests are better, but many still operate on tightly controlled seeds. They verify that application components can talk to each other. They often do not verify that workflows survive after historical data is migrated.

A common integration test pattern is:

bootstrap a fresh schema
seed current-style entities
run application flow
assert success

That is useful for feature correctness. It does not simulate the actual production event sequence:

years of state accumulate
schema changes
data migrates partially or fully
old and new code may coexist during rollout
user workflows hit transformed legacy records

Manual QA

Manual QA is often the final safety net, but it tends to skew toward new-account happy paths because those are easiest to execute in staging. QA checks signup, creates a test order, verifies page rendering, and moves on.

Meanwhile, the broken path is “admin edits a 2018 customer with a missing tax region whose subscription was migrated from three previous billing models.” That is not the sort of thing manual verification catches consistently.

CI/CD smoke tests

Smoke tests after deploy are better than nothing, but if they only hit basic health endpoints or create fresh data, they still miss the problem. A migration is risky because of existing state, not because a /healthz endpoint returns 200.

The core insight: verify workflows after the data changes

The right mental model is this: migrations are product changes, not just database changes.

If a migration alters user, order, catalog, billing, or permissions state, then the verification target is not merely schema validity. It is application behavior on migrated state.

That means CI/CD needs to shift from “did the code pass tests before merge?” to “did critical actions still work after realistic state was transformed?”

This sounds obvious when stated directly, but most teams do not implement it. They validate migration syntax, maybe some row counts, and stop there. That is not enough.

For important state changes, you want action-level checks like:

signup works for a new account after migration
login works for existing accounts created across different eras
checkout succeeds for migrated carts and stored payment profiles
search returns expected results for legacy and new records
admin can edit historical entities without losing fields
exports and reports include pre-migration records
webhooks and background jobs still process old objects correctly
rollback or roll-forward path preserves operability

Notice that these are not isolated SQL assertions. They are end-to-end checks of business behavior.

This is where browser automation and API-level workflow testing become more valuable than another layer of unit assertions. Tools like Playwright are useful not because browser testing is fashionable, but because they let you validate the product the way users and operators experience it.

What migration verification should look like in practice

A practical migration verification pipeline has three stages:

establish representative pre-migration state
execute migration and rollout sequence
verify critical workflows on migrated state

Let’s make that concrete.

Step 1: Create representative state

You need data that reflects reality. This can be done with:

anonymized database snapshot from a recent environment
generated records based on production distributions
curated legacy-state fixtures from known historical anomalies
cohort-specific seeds, such as accounts from old pricing models

A setup script in JavaScript might load representative records before migration:

js
import fs from 'fs'
import { Client } from 'pg'

const client = new Client({ connectionString: process.env.DATABASE_URL })

async function seedLegacyState() {
  await client.connect()

  const users = JSON.parse(fs.readFileSync('./datasets/legacy-users.json', 'utf-8'))

  for (const user of users) {
    await client.query(
      `INSERT INTO users (id, email, plan, billing_status, created_at)
       VALUES ($1, $2, $3, $4, $5)`,
      [user.id, user.email, user.plan, user.billing_status, user.created_at]
    )
  }

  await client.end()
}

seedLegacyState().catch(err => {
  console.error(err)
  process.exit(1)
})

This is still simplistic, but it forces the pipeline to deal with older shapes of data rather than pristine seeds.

Step 2: Run the migration as production would

Do not only test the migration in isolation. Test it in a sequence close to reality:

start from pre-migration app version if needed
load representative state
run migration scripts
run backfills
start new app version
run any indexing or materialization jobs
then verify workflows

That sequence matters. A migration can succeed but leave required derived state incomplete until a background process catches up. If your verification does not model that, your CI is lying.

Step 3: Verify user workflows and admin workflows

This is where Playwright or API-driven tests become the center of gravity.

A Playwright test for a migrated billing flow might look like this:

ts
import { test, expect } from '@playwright/test'

test('existing customer can update payment method after billing migration', async ({ page }) => {
  await page.goto('/login')
  await page.fill('[name=email]', 'legacy-customer@example.com')
  await page.fill('[name=password]', 'password123')
  await page.click('button[type=submit]')

  await expect(page).toHaveURL(/dashboard/)

  await page.goto('/settings/billing')
  await expect(page.locator('text=Current Plan')).toBeVisible()
  await expect(page.locator('text=Pro Annual')).toBeVisible()

  await page.click('text=Update payment method')
  await page.fill('[name=cardNumber]', '4242424242424242')
  await page.fill('[name=expiry]', '12/30')
  await page.fill('[name=cvc]', '123')
  await page.click('button:has-text("Save")')

  await expect(page.locator('text=Payment method updated')).toBeVisible()
})

That test is valuable only if legacy-customer@example.com exists in migrated state that reflects a real historical account shape.

An API-level workflow test can cover checkout faster and more deterministically:

python
import requests

def test_checkout_works_for_migrated_cart(base_url, legacy_user_token):
    cart_resp = requests.get(
        f"{base_url}/api/cart",
        headers={"Authorization": f"Bearer {legacy_user_token}"},
        timeout=10,
    )
    assert cart_resp.status_code == 200

    checkout_resp = requests.post(
        f"{base_url}/api/checkout",
        json={"payment_method": "pm_test_visa"},
        headers={"Authorization": f"Bearer {legacy_user_token}"},
        timeout=10,
    )
    assert checkout_resp.status_code == 200
    assert checkout_resp.json()["status"] == "confirmed"

For admin workflows, write tests people often forget:

ts
test('admin can edit a migrated legacy order without data loss', async ({ page }) => {
  await page.goto('/admin/login')
  await page.fill('[name=email]', 'admin@example.com')
  await page.fill('[name=password]', 'password123')
  await page.click('button[type=submit]')

  await page.goto('/admin/orders/ORD-2019-00421')
  await expect(page.locator('text=Legacy Imported Order')).toBeVisible()

  await page.fill('[name=customerNote]', 'verified after migration')
  await page.click('button:has-text("Save Changes")')

  await expect(page.locator('text=Changes saved')).toBeVisible()
  await expect(page.locator('[name=taxRegion]')).toHaveValue('EU-WEST')
})

This matters because admin tools often touch the oldest, strangest records in the system.

Rollback plans are often fiction

Migration PRs love a reassuring sentence: “Rollback is safe by reverting the app and migration.” In many cases, that is not a real rollback plan. It is a wish.

Once state changes are applied, especially destructive or lossy changes, reverting code may not restore operability. Examples:

columns dropped after backfill cannot be repopulated without preserved source data
enum normalization collapses multiple historical values into one
duplicated relationships are deduplicated irreversibly
background jobs mutate migrated records after deploy
old code cannot read new shape safely
external systems consume new identifiers during rollout

Teams need to stop pretending every migration is reversible. Many are only roll-forward safe. That changes how you should verify them.

If rollback is fictional, then pre-merge validation has to be stronger. You cannot rely on “we’ll just revert it” if migrated state has already crossed a one-way door.

A more honest PR template would ask:

Is this migration truly reversible?
If not, what is the roll-forward recovery path?
What data is preserved for reconstruction?
What workflows have been verified against migrated state?
What observability detects failure quickly after release?

Those questions are far more useful than generic rollback boilerplate.

What a better CI/CD job looks like

Here is a simplified GitHub Actions example that gets closer to meaningful migration verification:

yaml
name: migration-workflow-verification

on:
  pull_request:
    paths:
      - 'db/migrations/**'
      - 'backend/**'
      - 'frontend/**'

jobs:
  verify-migrated-workflows:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: app
          POSTGRES_PASSWORD: app
          POSTGRES_DB: app_test
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U app"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          npm ci
          pip install -r requirements.txt
          npx playwright install --with-deps

      - name: Start app at pre-migration version and seed legacy data
        run: |
          npm run db:reset
          node scripts/seed-legacy-state.js

      - name: Run migration and backfills
        run: |
          npm run db:migrate
          npm run backfill:billing-subscriptions

      - name: Start application
        run: |
          npm run start:test &
          npx wait-on http://127.0.0.1:3000/healthz

      - name: Verify API workflows on migrated state
        env:
          BASE_URL: http://127.0.0.1:3000
        run: |
          pytest tests/workflows/test_checkout_migrated_state.py
          pytest tests/workflows/test_search_migrated_state.py

      - name: Verify browser workflows on migrated state
        run: |
          npx playwright test tests/e2e/migrated-state

This is not perfect, but it is meaningfully better than “run migration on empty DB, then run unit tests.”

The big conceptual shift is that CI is now testing the product after the state change, not merely validating code around the change.

Tools comparison: what each layer is good for

No single tool solves migration safety. You need layered verification, but you also need clarity about what each layer can and cannot prove.

Unit tests

Best for:

mapping functions
invariants
transformation edge cases
query builders

Weak for:

realistic state diversity
cross-service behavior
end-to-end workflow confidence

Integration tests

Best for:

service boundaries
repository and database interactions
API contracts

Weak for:

historical production state unless deliberately modeled
UI/admin behavior
rollout sequencing

Database migration checks

Best for:

migration syntax
execution safety
row counts and referential integrity assertions
detecting obvious failures early

Weak for:

proving user-visible functionality
validating operational workflows

Playwright or browser E2E

Best for:

signup, checkout, admin edits, billing changes
validating real user flows against migrated state
catching UI plus backend mismatches

Weak for:

broad combinatorial coverage if overused
speed, if every test is full-browser and poorly scoped

API workflow tests

Best for:

fast action-level verification
critical path coverage without browser overhead
asserting business outcomes after migration

Weak for:

missing front-end regressions
not validating operator/admin UX directly

Manual QA

Best for:

exploratory investigation
weird edge-case probing after automated failures
release review for nuanced behavior

Weak for:

repeatability
broad historical state coverage
keeping up with AI-generated change velocity

The practical answer is usually:

use unit and integration tests for logic correctness
use migration assertions for structural correctness
use API and Playwright workflow tests for behavioral correctness on migrated state

That combination gives better debugging signals and better confidence than simply adding more low-level tests.

Actionable practices for teams shipping agent-written migrations

If your team is using AI to accelerate backend changes, these practices will improve reliability without turning every PR into a week-long ceremony.

1. Trigger workflow verification for stateful changes

Do not treat all PRs equally. Detect risky changes:

database migrations
ORM model changes
query rewrites
background job changes touching state
import/export format changes
permission model changes

Those PRs should trigger migrated-state verification jobs automatically.

2. Maintain a representative state corpus

Build a reusable dataset library for old account types, messy records, and historical edge cases. Update it from real incidents and production observations.

Think of it as a regression suite for your data model, not just your code.

3. Define critical workflows in product terms

Do not write only technical checks like “column populated” or “row count matches.” Define checks like:

existing user updates card
old order can be refunded
legacy customer appears in search
admin edits address on imported account
invoice export includes pre-2022 records

These are the workflows executives, support, and customers actually care about.

4. Test old data with new code

This sounds trivial, but many pipelines still mostly test new code with new data. For migration safety, old data transformed into the new shape is the key risk surface.

5. Be explicit about irreversible migrations

Label migrations as reversible, conditionally reversible, or roll-forward only. Make the recovery strategy part of the engineering review.

That honesty improves both testing and operational planning.

6. Add observability tied to workflows, not just infrastructure

After release, monitor:

checkout success rate
payment method update failures
admin save failures
search result drop-offs
export job record counts
webhook processing errors by object vintage

If you only watch CPU, latency, and 500 rates, you will miss business regressions that return 200 with wrong behavior.

7. Keep E2E scope narrow but meaningful

Do not respond by building a huge flaky browser suite. Pick a handful of core workflows that represent business-critical state transitions. Make them stable, deterministic, and tied to representative migrated accounts.

8. Make AI-generated PRs earn trust differently

When an agent opens a PR with migrations or data-shape changes, require stronger evidence than passing unit tests. For example:

migrated-state workflow test results
anomaly counts from rehearsal migration
before/after data diff summaries
explicit unsupported legacy cases
roll-forward plan

AI should reduce toil, not lower the bar for proof.

9. Rehearse migrations outside production with recent snapshots

For important changes, run the full migration and workflow verification against sanitized recent snapshots in a staging or rehearsal environment. This is slower than CI, but much cheaper than discovering data corruption through support tickets.

10. Turn incidents into state fixtures and workflow tests

Every migration incident should leave behind:

a representative dataset entry
a workflow test reproducing the failure
an operational checklist update if needed

That is how reliability compounds.

A note on debugging migrated-state failures

When these failures happen, debugging is painful because logs often point at symptoms, not causes. The request that fails after migration may be three layers removed from the bad transformation.

Good debugging practice includes:

correlation IDs linking migration batches to user-facing failures
migration metadata on transformed records
audit tables for before/after values on critical entities
cohort-based metrics by record age or source
ability to replay workflow tests against captured failing records

If a subset of 2019 accounts cannot check out after a subscription migration, you want to identify that cohort quickly, not sift through generic 500 logs for hours.

This is another reason workflow verification matters in CI/CD: the failure signal is closer to the business outcome, which shortens diagnosis.

The real standard for confidence

The old standard for confidence was: the code compiles, tests pass, and the migration executes.

That standard was always incomplete. In an era where AI agents can produce coherent code changes and polished PRs at high speed, it is now actively dangerous.

The new standard should be: after representative state is migrated, critical product workflows still work.

That is a higher bar, but it is also the only bar that maps to reality.

Because users do not care whether your schema migration was elegant. They care whether they can log in, search, check out, update billing, and complete work after you change the system underneath them.

And if your agent opened the PR, generated the migration, updated the tests, and wrote the rollout notes, then someone — or better, something automated and adversarial — still needs to verify that the migration actually worked.

Not in theory. Not on an empty database. Not on perfect fixtures.

On state that looks like production, with workflows that matter to the business.

That is what real testing looks like now. That is what reliable CI/CD should prove. And that is how teams keep AI-assisted delivery from becoming a faster way to ship stateful failures.