Build a Real File Upload Flow That Survives Broken PDFs, S3 Misconfigurations, and UI False Positives

File uploads look trivial right up until they matter.

A generated drag-and-drop component works. A progress bar animates. Your API returns 200. CI is green. Everyone says the feature is done.

Then production happens.

A customer uploads a PDF that opens fine in Preview but breaks your parser. Another user sees “Upload complete” while the object never became durable because the browser only finished sending bytes to a presigned URL, not the rest of your pipeline. An S3 CORS mistake only appears in real browsers, so your server-side tests never saw it. A large file triggers a timeout and retry loop, and the UI quietly reports success because it never checked downstream processing.

This is the kind of gap that keeps showing up in modern teams: we can generate upload code fast, but we still fail to verify the user workflow. And if you only test code paths in isolation, you get false confidence. The button rendered. The unit test passed. The user’s document still didn’t make it through the system.

In this walkthrough, we’ll build a production-style file upload flow in Next.js with:

• a drag-and-drop React UI
• presigned S3 uploads
• an application backend that tracks upload state
• background processing for uploaded PDFs
• durable status polling in the UI
• Playwright tests that upload real files and verify downstream outcomes
• CI wiring that catches failures beyond unit tests

The point is not to build a massive document platform. The point is to build the smallest realistic system that demonstrates where upload flows actually fail, and how to test them in a way that improves debugging, testing, CI/CD, and developer productivity.

What we’re building

Here’s the workflow:

1. User drops a PDF in the browser.
2. Frontend calls our app server for an upload session.
3. Server validates metadata and generates a presigned S3 PUT URL.
4. Browser uploads directly to S3.
5. Frontend tells the app server to finalize the upload.
6. Server marks the record as uploaded and enqueues processing.
7. A background worker fetches the file from S3 and parses the PDF.
8. Worker updates the record to processed or failed.
9. UI polls for status and only shows success when processing really finished.
10. Playwright verifies the whole flow using actual files.

That sequence matters because “bytes uploaded” is not the same thing as “user goal completed.”

The architecture decision that avoids fake success

A lot of upload implementations collapse multiple states into one. That’s the root cause of UI false positives.

Don’t model upload as a boolean. Model it as a state machine.

We’ll use these statuses:

• created: upload session exists, file not yet transferred
• uploading: optional client-side status only
• uploaded: S3 transfer completed and app was notified
• processing: background worker picked it up
• processed: parsing and persistence succeeded
• failed: durable failure with an error reason

If your UI shows “complete” at uploaded, users will think the file is ready when it may still fail parsing, virus scanning, OCR, indexing, or persistence. That’s how upload UIs lie.

Project setup

We’ll use:

• Next.js App Router
• TypeScript
• PostgreSQL with Prisma
• AWS S3
• BullMQ + Redis for background jobs
• pdf-parse for PDF parsing
• Playwright for end-to-end testing

You can adapt the backend pieces to Express if you want. The core testing ideas don’t change.

Install dependencies

bash
npm install next react react-dom @prisma/client prisma zod @aws-sdk/client-s3 @aws-sdk/s3-request-presigner bullmq ioredis pdf-parse
npm install -D typescript @types/node @types/react @types/react-dom playwright

Prisma schema

Create prisma/schema.prisma:

prisma
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model Upload {
  id            String   @id @default(cuid())
  fileName      String
  fileSize      Int
  mimeType      String
  storageKey    String   @unique
  bucket        String
  status        UploadStatus @default(CREATED)
  errorMessage  String?
  pageCount     Int?
  extractedText String?
  createdAt     DateTime @default(now())
  updatedAt     DateTime @updatedAt
}

enum UploadStatus {
  CREATED
  UPLOADED
  PROCESSING
  PROCESSED
  FAILED
}

Run:

bash
npx prisma generate
npx prisma migrate dev --name init_uploads

Environment variables

Create .env:

bash
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/uploads
REDIS_URL=redis://localhost:6379
AWS_REGION=us-east-1
S3_BUCKET=debuggai-upload-demo
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret
MAX_FILE_SIZE_BYTES=10485760
APP_URL=http://localhost:3000

S3 setup that won’t fail only in browsers

This is where teams lose hours debugging things unit tests never catch.

For direct browser uploads, you need correct CORS on the bucket. A misconfiguration often won’t show up in backend tests because the server can talk to S3 just fine. Browsers enforce CORS. Node test runners often don’t.

Use a bucket CORS config like:

json
[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["PUT", "GET", "HEAD"],
    "AllowedOrigins": ["http://localhost:3000"],
    "ExposeHeaders": ["ETag"]
  }
]

Common failure modes:

• missing PUT in AllowedMethods
• wrong frontend origin
• wildcard origin combined with credentials assumptions
• headers mismatch when sending Content-Type
• success in Postman but failure in browser

This is exactly why workflow-level testing matters. The browser is part of the system.

Shared server utilities

Create lib/prisma.ts:

ts
import { PrismaClient } from '@prisma/client'

const globalForPrisma = global as unknown as { prisma: PrismaClient }

export const prisma =
  globalForPrisma.prisma ||
  new PrismaClient({
    log: ['error', 'warn'],
  })

if (process.env.NODE_ENV !== 'production') globalForPrisma.prisma = prisma

Create lib/s3.ts:

ts
import { S3Client } from '@aws-sdk/client-s3'

export const s3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
})

Create lib/redis.ts:

ts
import IORedis from 'ioredis'

export const redis = new IORedis(process.env.REDIS_URL!)

Create lib/queue.ts:

ts
import { Queue } from 'bullmq'
import { redis } from './redis'

export const uploadQueue = new Queue('upload-processing', {
  connection: redis,
})

API: create upload session

This endpoint performs initial validation, creates a DB record, and returns a presigned URL.

Create app/api/uploads/create/route.ts:

ts
import { NextRequest, NextResponse } from 'next/server'
import { z } from 'zod'
import { prisma } from '@/lib/prisma'
import { s3 } from '@/lib/s3'
import { PutObjectCommand } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'

const schema = z.object({
  fileName: z.string().min(1),
  fileSize: z.number().int().positive(),
  mimeType: z.string().min(1),
})

export async function POST(req: NextRequest) {
  const body = await req.json()
  const parsed = schema.safeParse(body)

  if (!parsed.success) {
    return NextResponse.json({ error: 'Invalid request' }, { status: 400 })
  }

  const { fileName, fileSize, mimeType } = parsed.data
  const maxSize = Number(process.env.MAX_FILE_SIZE_BYTES || 10 * 1024 * 1024)

  if (mimeType !== 'application/pdf') {
    return NextResponse.json({ error: 'Only PDF uploads are supported' }, { status: 400 })
  }

  if (fileSize > maxSize) {
    return NextResponse.json({ error: 'File too large' }, { status: 400 })
  }

  const id = crypto.randomUUID()
  const storageKey = `uploads/${id}-${fileName.replace(/[^a-zA-Z0-9._-]/g, '_')}`
  const bucket = process.env.S3_BUCKET!

  await prisma.upload.create({
    data: {
      id,
      fileName,
      fileSize,
      mimeType,
      storageKey,
      bucket,
      status: 'CREATED',
    },
  })

  const command = new PutObjectCommand({
    Bucket: bucket,
    Key: storageKey,
    ContentType: mimeType,
  })

  const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 60 * 5 })

  return NextResponse.json({
    uploadId: id,
    uploadUrl,
    storageKey,
  })
}

A few important points:

• We validate type and size before generating the URL.
• We create the DB record first so every upload attempt has traceability.
• We don’t trust the browser’s future success just because we gave it a URL.

API: finalize upload

After the browser finishes PUTting to S3, it must notify the app server. This is where many flows cut corners.

Create app/api/uploads/finalize/route.ts:

ts
import { NextRequest, NextResponse } from 'next/server'
import { z } from 'zod'
import { prisma } from '@/lib/prisma'
import { uploadQueue } from '@/lib/queue'

const schema = z.object({
  uploadId: z.string().min(1),
})

export async function POST(req: NextRequest) {
  const body = await req.json()
  const parsed = schema.safeParse(body)

  if (!parsed.success) {
    return NextResponse.json({ error: 'Invalid request' }, { status: 400 })
  }

  const upload = await prisma.upload.findUnique({
    where: { id: parsed.data.uploadId },
  })

  if (!upload) {
    return NextResponse.json({ error: 'Upload not found' }, { status: 404 })
  }

  await prisma.upload.update({
    where: { id: upload.id },
    data: { status: 'UPLOADED' },
  })

  await uploadQueue.add('process-upload', { uploadId: upload.id })

  return NextResponse.json({ ok: true })
}

In a stricter implementation, you would verify the object exists in S3 with a HeadObject call before marking it uploaded. Do that in production if you want to reduce edge cases where the browser claims success but the object is missing or incomplete.

API: fetch upload status

Create app/api/uploads/[id]/route.ts:

ts
import { NextResponse } from 'next/server'
import { prisma } from '@/lib/prisma'

export async function GET(
  _req: Request,
  { params }: { params: { id: string } }
) {
  const upload = await prisma.upload.findUnique({
    where: { id: params.id },
  })

  if (!upload) {
    return NextResponse.json({ error: 'Not found' }, { status: 404 })
  }

  return NextResponse.json(upload)
}

Background worker: parse the PDF

Now for the part most demos skip: what happens after upload.

Create worker.ts:

ts
import { Worker } from 'bullmq'
import { redis } from './lib/redis'
import { prisma } from './lib/prisma'
import { s3 } from './lib/s3'
import { GetObjectCommand } from '@aws-sdk/client-s3'
import pdfParse from 'pdf-parse'

function streamToBuffer(stream: any): Promise<Buffer> {
  return new Promise((resolve, reject) => {
    const chunks: Buffer[] = []
    stream.on('data', (chunk: Buffer) => chunks.push(chunk))
    stream.on('error', reject)
    stream.on('end', () => resolve(Buffer.concat(chunks)))
  })
}

new Worker(
  'upload-processing',
  async job => {
    const { uploadId } = job.data as { uploadId: string }

    const upload = await prisma.upload.findUnique({ where: { id: uploadId } })
    if (!upload) throw new Error(`Upload ${uploadId} not found`)

    await prisma.upload.update({
      where: { id: uploadId },
      data: { status: 'PROCESSING', errorMessage: null },
    })

    try {
      const object = await s3.send(
        new GetObjectCommand({
          Bucket: upload.bucket,
          Key: upload.storageKey,
        })
      )

      if (!object.Body) {
        throw new Error('Uploaded object has no body')
      }

      const buffer = await streamToBuffer(object.Body)
      const parsed = await pdfParse(buffer)

      if (!parsed.text || !parsed.text.trim()) {
        throw new Error('PDF parsed but no text was extracted')
      }

      await prisma.upload.update({
        where: { id: uploadId },
        data: {
          status: 'PROCESSED',
          extractedText: parsed.text.slice(0, 5000),
          pageCount: parsed.numpages,
          errorMessage: null,
        },
      })
    } catch (err: any) {
      await prisma.upload.update({
        where: { id: uploadId },
        data: {
          status: 'FAILED',
          errorMessage: err.message || 'Processing failed',
        },
      })

      throw err
    }
  },
  { connection: redis }
)

This worker is intentionally simple, but it exposes real-world truths:

• a file can upload correctly and still fail business processing
• MIME type checks in the browser are weak
• malformed PDFs may pass initial checks and fail later
• durable success needs background result verification

React drag-and-drop UI

Create app/page.tsx:

tsx
'use client'

import { useCallback, useMemo, useState } from 'react'

type UploadRecord = {
  id: string
  status: 'CREATED' | 'UPLOADED' | 'PROCESSING' | 'PROCESSED' | 'FAILED'
  errorMessage?: string | null
  extractedText?: string | null
  pageCount?: number | null
}

export default function HomePage() {
  const [dragging, setDragging] = useState(false)
  const [progress, setProgress] = useState(0)
  const [message, setMessage] = useState('')
  const [upload, setUpload] = useState<UploadRecord | null>(null)
  const [busy, setBusy] = useState(false)

  const statusLabel = useMemo(() => {
    if (!upload) return 'No upload yet'
    switch (upload.status) {
      case 'CREATED': return 'Preparing upload'
      case 'UPLOADED': return 'Uploaded, waiting for processing'
      case 'PROCESSING': return 'Processing document'
      case 'PROCESSED': return 'Document processed successfully'
      case 'FAILED': return 'Document processing failed'
    }
  }, [upload])

  const pollStatus = useCallback(async (uploadId: string) => {
    let attempts = 0
    while (attempts < 60) {
      attempts++
      const res = await fetch(`/api/uploads/${uploadId}`)
      const data = await res.json()
      setUpload(data)

      if (data.status === 'PROCESSED' || data.status === 'FAILED') {
        return data
      }

      await new Promise(r => setTimeout(r, 1000))
    }

    throw new Error('Timed out waiting for processing')
  }, [])

  const uploadFile = useCallback(async (file: File) => {
    setBusy(true)
    setMessage('')
    setProgress(0)
    setUpload(null)

    try {
      if (file.type !== 'application/pdf') {
        throw new Error('Only PDF files are allowed')
      }

      const createRes = await fetch('/api/uploads/create', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          fileName: file.name,
          fileSize: file.size,
          mimeType: file.type,
        }),
      })

      if (!createRes.ok) {
        const err = await createRes.json()
        throw new Error(err.error || 'Failed to create upload')
      }

      const { uploadId, uploadUrl } = await createRes.json()

      await new Promise<void>((resolve, reject) => {
        const xhr = new XMLHttpRequest()
        xhr.open('PUT', uploadUrl)
        xhr.setRequestHeader('Content-Type', file.type)

        xhr.upload.onprogress = evt => {
          if (evt.lengthComputable) {
            const percent = Math.round((evt.loaded / evt.total) * 100)
            setProgress(percent)
          }
        }

        xhr.onload = () => {
          if (xhr.status >= 200 && xhr.status < 300) resolve()
          else reject(new Error(`S3 upload failed with ${xhr.status}`))
        }

        xhr.onerror = () => reject(new Error('Network error during upload'))
        xhr.send(file)
      })

      const finalizeRes = await fetch('/api/uploads/finalize', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ uploadId }),
      })

      if (!finalizeRes.ok) {
        const err = await finalizeRes.json()
        throw new Error(err.error || 'Failed to finalize upload')
      }

      const finalState = await pollStatus(uploadId)

      if (finalState.status === 'PROCESSED') {
        setMessage('Upload and processing complete')
      } else {
        throw new Error(finalState.errorMessage || 'Processing failed')
      }
    } catch (err: any) {
      setMessage(err.message || 'Upload failed')
    } finally {
      setBusy(false)
    }
  }, [pollStatus])

  const onDrop = useCallback((e: React.DragEvent<HTMLDivElement>) => {
    e.preventDefault()
    setDragging(false)
    const file = e.dataTransfer.files?.[0]
    if (file) uploadFile(file)
  }, [uploadFile])

  return (
    <main style={{ maxWidth: 700, margin: '40px auto', fontFamily: 'sans-serif' }}>
      <h1>Upload a PDF</h1>
      <div
        onDragOver={e => { e.preventDefault(); setDragging(true) }}
        onDragLeave={() => setDragging(false)}
        onDrop={onDrop}
        style={{
          border: '2px dashed #999',
          borderColor: dragging ? '#11AB5A' : '#999',
          padding: 40,
          borderRadius: 12,
          textAlign: 'center',
          background: dragging ? '#f0fff5' : '#fff',
        }}
        data-testid="dropzone"
      >
        Drag and drop a PDF here
      </div>

      <div style={{ marginTop: 20 }}>
        <progress value={progress} max={100} style={{ width: '100%' }} />
        <p data-testid="progress">Transfer progress: {progress}%</p>
        <p data-testid="status">{statusLabel}</p>
        {message && <p data-testid="message">{message}</p>}
      </div>

      {upload?.status === 'PROCESSED' && (
        <section data-testid="result" style={{ marginTop: 20 }}>
          <p>Pages: {upload.pageCount}</p>
          <pre style={{ whiteSpace: 'pre-wrap' }}>{upload.extractedText}</pre>
        </section>
      )}

      {upload?.status === 'FAILED' && (
        <section data-testid="error" style={{ marginTop: 20, color: 'crimson' }}>
          {upload.errorMessage}
        </section>
      )}
    </main>
  )
}

Two important UX choices here:

1. The progress bar reflects transfer progress only.
2. Success is shown only after backend processing reaches PROCESSED.

That separation is the difference between a truthful UI and a deceptive one.

Why client-side checks are not enough

The UI checks file.type === 'application/pdf'. That’s useful, but weak.

Problems it won’t catch:

• renamed files like not-a-real-pdf.pdf
• malformed PDFs that still claim the correct MIME type
• encrypted or corrupted PDFs your parser can’t handle
• giant files that compress strangely or trigger infrastructure limits
• files uploaded successfully but inaccessible due to bucket policy issues

This is why you need downstream verification. Not because validation is bad, but because validation at one layer cannot prove the whole workflow.

A better definition of done for uploads

An upload feature is not done when:

• the dropzone renders
• the POST endpoint returns success
• the object exists in storage
• the progress bar reaches 100%

It’s done when a realistic file goes through the exact path a user takes and you can assert the intended business outcome.

In our case: upload a real PDF, process it, and show extracted results or a real failure message.

Playwright tests that catch the failures people actually ship

Most upload bugs live in the seams between systems. That’s where browser-based end-to-end testing earns its keep.

Install Playwright browsers:

bash
npx playwright install

Create test fixtures:

• tests/fixtures/valid.pdf
• tests/fixtures/broken.pdf

Use an actual small PDF for valid.pdf. For broken.pdf, you can create a fake file with a PDF extension or a malformed PDF byte sequence.

Playwright config

Create playwright.config.ts:

ts
import { defineConfig } from '@playwright/test'

export default defineConfig({
  testDir: './tests',
  use: {
    baseURL: 'http://127.0.0.1:3000',
    trace: 'retain-on-failure',
  },
  webServer: {
    command: 'npm run dev',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
  },
})

Test: happy path with downstream assertion

Create tests/upload.spec.ts:

ts
import { test, expect } from '@playwright/test'
import path from 'path'

test('uploads and processes a valid PDF', async ({ page }) => {
  await page.goto('/')

  const fileChooserPromise = page.waitForEvent('filechooser')

  await page.setInputFiles('input[type="file"]', path.join(__dirname, 'fixtures/valid.pdf')).catch(() => {})

  await page.evaluate(async () => {
    const response = await fetch('/api/uploads/create', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        fileName: 'valid.pdf',
        fileSize: 1024,
        mimeType: 'application/pdf',
      }),
    })
    if (!response.ok) throw new Error('setup failed')
  }).catch(() => {})

  const dataTransfer = await page.evaluateHandle(() => new DataTransfer())
  const input = await page.locator('[data-testid="dropzone"]')

  await page.locator('body').setInputFiles(path.join(__dirname, 'fixtures/valid.pdf'))

  await expect(page.getByTestId('progress')).toContainText('100%', { timeout: 30000 })
  await expect(page.getByTestId('status')).toContainText('Document processed successfully', { timeout: 30000 })
  await expect(page.getByTestId('result')).toContainText('Pages:')
})

The exact drag-and-drop mechanics can vary depending on your component implementation. In practice, many teams add a hidden file input for easier automation and accessibility. That’s what I recommend.

Let’s improve the UI slightly to support that.

Update app/page.tsx with a hidden input:

tsx
<input
  data-testid="file-input"
  type="file"
  accept="application/pdf"
  style={{ display: 'none' }}
  onChange={e => {
    const file = e.target.files?.[0]
    if (file) uploadFile(file)
  }}
/>

Make the dropzone clickable:

tsx
<label htmlFor="file-input" style={{ display: 'block', cursor: 'pointer' }}>
  <div ... data-testid="dropzone">
    Drag and drop a PDF here, or click to choose a file
  </div>
</label>

Then the test becomes much simpler and more reliable.

ts
import { test, expect } from '@playwright/test'
import path from 'path'

test('uploads and processes a valid PDF', async ({ page }) => {
  await page.goto('/')
  await page.getByTestId('file-input').setInputFiles(path.join(__dirname, 'fixtures/valid.pdf'))

  await expect(page.getByTestId('progress')).toContainText('100%', { timeout: 30000 })
  await expect(page.getByTestId('status')).toContainText('Document processed successfully', { timeout: 30000 })
  await expect(page.getByTestId('result')).toContainText('Pages:')
  await expect(page.getByTestId('message')).toContainText('Upload and processing complete')
})

Test: malformed PDF should fail after upload, not before

This is a critical workflow test because it proves your app handles the common “looks uploaded, fails later” path correctly.

ts
test('shows a processing failure for a malformed PDF', async ({ page }) => {
  await page.goto('/')
  await page.getByTestId('file-input').setInputFiles(path.join(__dirname, 'fixtures/broken.pdf'))

  await expect(page.getByTestId('progress')).toContainText('100%', { timeout: 30000 })
  await expect(page.getByTestId('status')).toContainText('Document processing failed', { timeout: 30000 })
  await expect(page.getByTestId('error')).toBeVisible()
})

That one test catches a category of bug that unit tests around your React component and API route won’t.

Test: oversized file rejected early

ts
test('rejects oversized files before upload begins', async ({ page }) => {
  await page.goto('/')

  await page.route('/api/uploads/create', async route => {
    await route.continue()
  })

  const bigBuffer = Buffer.alloc(11 * 1024 * 1024, 'a')
  await page.getByTestId('file-input').setInputFiles({
    name: 'huge.pdf',
    mimeType: 'application/pdf',
    buffer: bigBuffer,
  })

  await expect(page.getByTestId('message')).toContainText(/File too large|Upload failed/)
})

Test: browser-visible S3 failure

You can’t reliably unit test this class of problem. Simulate an S3 upload failure by intercepting the presigned PUT destination or by pointing tests at a bucket with intentionally broken CORS in a dedicated environment.

A pragmatic approach is to mock the PUT request failure in Playwright:

ts
test('surfaces S3 upload failures to the user', async ({ page, context }) => {
  await page.route('https://*.amazonaws.com/**', async route => {
    await route.fulfill({
      status: 403,
      contentType: 'application/xml',
      body: '<Error><Code>AccessDenied</Code></Error>',
    })
  })

  await page.goto('/')
  await page.getByTestId('file-input').setInputFiles(path.join(__dirname, 'fixtures/valid.pdf'))

  await expect(page.getByTestId('message')).toContainText('S3 upload failed', { timeout: 30000 })
})

This won’t perfectly emulate a browser CORS preflight failure, but it still verifies your UI doesn’t silently claim success when storage rejects the upload. For true CORS validation, run at least one environment-backed browser test against real S3 config.

What unit tests and mocked integration tests miss

This is the gap most teams underestimate.

Unit tests can prove:

• your create upload route validates size and MIME type
• your UI updates progress state correctly
• your worker marks a record as failed when parsing throws

Those are useful. Keep them.

But they do not prove:

• the browser can actually PUT to the signed URL
• S3 CORS is correct
• the frontend finalizes only after storage transfer
• the background worker sees the exact object uploaded
• the user sees processed output, not just transfer completion
• your status transitions behave under real timing

That’s why CI/CD pipelines built only around unit tests produce false confidence. They verify components, not workflows.

CI wiring that tests the system instead of admiring the code

You want CI to fail when the upload flow is broken, not just when TypeScript complains.

Here’s a GitHub Actions example.

Create .github/workflows/ci.yml:

yaml
name: ci

on:
  push:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: uploads
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U postgres"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5

      redis:
        image: redis:7
        ports:
          - 6379:6379

    env:
      DATABASE_URL: postgresql://postgres:postgres@localhost:5432/uploads
      REDIS_URL: redis://localhost:6379
      AWS_REGION: us-east-1
      S3_BUCKET: debuggai-upload-demo-ci
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      MAX_FILE_SIZE_BYTES: 10485760
      APP_URL: http://127.0.0.1:3000

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - run: npm ci
      - run: npx prisma migrate deploy
      - run: npx playwright install --with-deps

      - name: Start worker
        run: node --import tsx worker.ts &

      - name: Run unit tests
        run: npm test

      - name: Run Playwright tests
        run: npx playwright test

A couple of notes:

• The worker must run during end-to-end tests, or your UI will wait forever for processing.
• If you use real S3 in CI, keep a dedicated bucket with known-good CORS config.
• You can also use LocalStack, but run at least some browser tests against real cloud config before trusting it.

Silent breakages you catch only with full-flow verification

Here are the exact classes of bugs this setup catches better than traditional testing alone.

1. Progress bars that lie

If your UI marks success when XMLHttpRequest finishes, the app can show “done” even when processing fails one second later.

Workflow test catches it because it waits for PROCESSED, not just 100%.

2. Browser-only storage failures

Signed URL works. Backend can access S3. Unit tests pass.

But browser upload fails because CORS headers are wrong.

Only a browser-based end-to-end test sees the truth.

3. Broken PDFs that pass client checks

The file extension is .pdf. MIME type says PDF. The upload succeeds.

The parser throws. If your UI never reflects worker failure, users are stuck.

A full-flow test proves failure handling actually reaches the interface.

4. Queue or worker outages

Your upload endpoint works, but Redis is down or the worker process isn’t running.

Without a workflow-level test, you may ship a UI that reports uploads but never completes processing.

5. Finalization gaps

The browser uploads bytes to S3 but never successfully calls /finalize due to a transient network issue.

Now you have orphaned objects in storage and no processing job.

A mature system adds reconciliation for this. At minimum, your tests should cover the finalize step explicitly.

Practical hardening you should add in production

The demo is intentionally lean. In production, add these upgrades.

Verify uploaded object before enqueueing

In /api/uploads/finalize, call HeadObject and compare content length if you have it. Don’t rely purely on the client saying “the upload worked.”

Add idempotency

Users retry. Browsers retry. Agents generate duplicate calls.

Make finalization safe to call multiple times. If status is already UPLOADED or beyond, return success without duplicating jobs.

Record upload attempt metadata

Store:

• client-reported MIME type
• final storage ETag if available
• processing duration
• failure code/category
• retry count

This makes debugging much faster when production incidents happen.

Add dead-letter handling

If processing repeatedly fails, route jobs to a dead-letter queue or mark them with retry exhaustion metadata. Don’t let them disappear into logs.

Consider malware scanning and content validation

For user-generated files, parsing is not enough. You may need antivirus scanning, format verification, and safe document handling rules.

Add cleanup for orphaned objects

If create succeeds but finalize never happens, clean up old objects and stale DB rows with a scheduled job.

Debugging upload systems without losing a day

When uploads break, debug from the user workflow outward.

Use this sequence:

1. Did the browser successfully get a presigned URL?
2. Did the browser PUT to storage without CORS or auth errors?
3. Did finalize get called after transfer?
4. Did the DB status move from CREATED to UPLOADED?
5. Was a queue job enqueued?
6. Did the worker fetch the exact object key?
7. Did parsing succeed?
8. Did the UI poll and render the final state?

This sounds obvious, but most teams debug uploads from logs in one service while the failure is in another layer entirely.

A state machine plus end-to-end tests compresses that search space dramatically. That’s a real developer productivity gain: faster debugging, fewer ambiguous incidents, less time guessing whether the problem is frontend, storage, worker, or CI/CD setup.

Tooling comparison: what each layer is good for

You do not need to choose one type of testing. You need to stop expecting one type to prove everything.

Unit tests

Best for:

• validation logic
• state transition helpers
• parser wrapper behavior
• UI rendering for each upload state

Weak at:

• real browser upload behavior
• CORS
• timing between services
• proving user outcomes

Integration tests

Best for:

• API + DB interactions
• worker + storage parsing logic
• queue semantics in a controlled environment

Weak at:

• drag-and-drop/browser realities
• full signed-upload behavior from the client side

Playwright end-to-end tests

Best for:

• proving the workflow a user actually experiences
• catching browser-visible failures
• validating async status transitions end-to-end

Weak at:

• being your only test layer
• ultra-fast feedback for all code paths

The practical strategy is simple:

• use unit tests for fast local confidence
• use integration tests for service boundaries
• use Playwright for critical workflows like file upload

That combination gives you much better CI/CD signal than green unit tests alone.

The bigger lesson: stop testing implementation milestones

The reason upload features are such a good example is that they expose a broader testing mistake.

Teams often test milestones that are easy to automate but not meaningful to users:

• did the button render?
• did the API return 200?
• did the object store receive bytes?

Users do not care about those milestones. They care whether their document made it through the full system and is usable.

As AI-generated code speeds up implementation, this matters more, not less. It’s now easier than ever to produce code that appears complete while missing key workflow guarantees. Generated code tends to handle the obvious path first. Reliability comes from the verification discipline around it.

That’s the shift technical teams need to make: measure correctness at the workflow level.

Wrap-up

A robust file upload feature is not a drag-and-drop component plus a presigned URL. It’s a multi-step workflow across browser, backend, storage, queue, worker, and UI state.

If you only test pieces in isolation, you’ll miss the failures that matter most:

• malformed files that upload but fail processing
• S3/CORS problems visible only in browsers
• progress indicators that hit 100% before the user goal is complete
• queue or finalize gaps that strand uploaded files

The fix is not more ceremony. It’s better definitions of done:

• model upload as explicit states
• separate transfer success from business success
• verify downstream processing
• run browser-based end-to-end tests with real files
• wire CI/CD to exercise the full workflow, not just unit tests

That’s how you build an upload flow that survives real usage.

And it’s how you avoid the trap a lot of modern teams fall into: shipping something that looked finished in code review, passed CI, and still failed the first user who actually needed it.