Debugging Problems That Only Appear in Production

The Problem #

Occasionally, two status updates for the same order would arrive out of order and leave it in a stale state — visible only in production, never once in local testing, no matter how deliberately I tried to reproduce it by clicking through the same flow.

Context #

Locally, I'm one user making one request at a time. Production has multiple drivers and dispatchers acting concurrently, sometimes on the same order within the same second.

What I Tried #

Added logging around the status-update handler and tried to reproduce by scripting rapid sequential requests against my local API.

What Went Wrong #

Sequential requests, even fast ones, aren't the same as genuinely concurrent ones — my local repro attempts always resolved in the order I sent them, because there was no real race for them to lose. The bug needed two requests actually overlapping in-flight, which a single-user local session structurally can't produce on its own.

The Solution #

Reasoned about the concurrency instead of trying to brute-force reproduce it: two status updates for the same order, both read-then-write against the database without any coordination, could interleave so the earlier update's write lands after the later one's, overwriting a newer status with a stale one.

// Vulnerable: read, compute, write — another request can interleave here
const order = await Order.findById(id);
await Order.updateOne({ _id: id }, { status: newStatus });
 
// Fixed: atomic conditional update, no read-then-write gap for a race to land in
await Order.updateOne(
  { _id: id, statusVersion: expectedVersion },
  { status: newStatus, statusVersion: expectedVersion + 1 }
);

Why It Works #

An atomic, conditional update closes the exact window a race needs — there's no separate read step for a concurrent write to slip in behind. Reasoning about the concurrency model directly was necessary because the bug's precondition (real concurrent requests) doesn't exist in a local, single-user testing setup no matter how it's scripted.

Lessons Learned #

"I can't reproduce it locally" isn't evidence a bug is rare or unlikely — it can mean the bug's actual precondition is structurally absent from how local testing works, and the fix is to reason about the failure mode directly rather than keep trying to force a repro.

What I Would Do Differently #

I'd default to atomic, conditional writes for any state that multiple users can update concurrently from the start, instead of treating read-then-write as safe until a production-only race proved otherwise.

Related Concepts #

Race conditions in concurrent request handling, optimistic concurrency control, atomic database operations.