The bug Redis didn't cause

My upgrade endpoint had a race. Two requests for the same account could both read idle, both write processing, and both start an upgrade.

I did what most people do. I blamed Redis.

Redis was innocent. The bug was mine, and it was not really a concurrency bug. It was a missing business rule and a missing owner. Getting to that answer took me through Durable Objects, the actor model, sagas, and workflow engines. The fix was one command. The thinking was the useful part.

Two requests, one key

Here is the code, minus the noise.

const key = `upgrade:${accountId}`;
const state = await redis.get(key);
 
if (state === "idle") {
  await redis.set(key, "processing");
  await startUpgrade(accountId);
}

It reads like a single decision. It is not. It is three steps.

Request A: GET  -> idle
Request B: GET  -> idle
Request A: SET  -> processing
Request B: SET  -> processing
Request A: startUpgrade
Request B: startUpgrade

Redis ran every command in order. There was no corruption. The order was simply GET A, GET B, SET A, SET B. Between my read and my write, another request did its own read.

The rule I wanted was "only one upgrade per account". I wrote it in TypeScript, between two Redis calls. Nothing enforced it.

A single Redis server does not make a multi-step operation atomic. It makes each command atomic.

One Redis is not one owner

This is the distinction I kept missing. Redis serializes commands. It does not serialize my operation, because my operation only exists in my application code.

Ownership is a design decision: who is allowed to decide that this account is upgrading right now? In my code, nobody owned that decision. Every request could read the value and write a new one.

Choosing a database does not answer that question. Postgres with READ COMMITTED would have let the same interleaving through. A mutex in one process would have failed the moment I ran two instances. One Redis did not help because the gap was above Redis, in the gap between two commands.

The fix was one command

I used SET NX, which writes only if the key does not exist.

const claimed = await redis.set(key, "processing", { NX: true });
 
if (claimed === null) {
  return c.json({ error: "upgrade already running" }, 409);
}
 
await startUpgrade(accountId);

Redis now makes the check and the write as one command. The second request gets null and stops. A Lua script or WATCH/MULTI would also work. The point is not the command. The point is that the decision moved into the place that owns the state.

That is why the actor pitch felt fake to me at first. If one command fixed it, why would I need a new architecture? The answer is scope. One command fixes one key. An actor fixes every operation on one entity. My bug needed the first. A system with six writers needs the second.

What a Durable Object actually is

A Durable Object is one instance per ID, running in one location at a time, with a mailbox and durable storage. Requests to the same object go through one event loop, like Node.

That gives you a natural home for the upgrade rule. The object for account:123 can check its state and reject a second upgrade without a distributed lock.

But single-threaded does not mean "no interleaving". This is the part I got wrong.

// Wrong
if (this.state === "idle") {
  await chargeCard(this.accountId);
  this.state = "processing";
}

While chargeCard is awaiting the network, another request can enter. Cloudflare's input gates hold new events while a storage operation is in flight. They do not hold every event across every await. So the fix is to write the state before you do slow work.

// Right
if (this.state !== "idle") throw new Error("upgrade already running");
 
this.state = "processing";
await this.ctx.storage.put("state", this.state);
 
await chargeCard(this.accountId);

Now the second request sees processing. The cost is honest: the object lives in one place, so users far from it pay the latency. One hot object also has a throughput ceiling. Durable Objects give you coordination, not free global speed.

Rate limiting on Cloudflare, without the fog

Rate limiting has the same shape. First decide what you are protecting, then pick the layer.

• WAF rate limiting rules catch broad abuse before the Worker runs. Use them for public endpoints, login attempts, and scraping.
• The Workers Rate Limiting API is for product limits keyed by user, API key, or route.
• A Durable Object is for a strict limit when being wrong costs money.
• A queue is for expensive work. Let the queue decide how fast you call the model.

The Workers limiter is easy to reach for, but read the accuracy note. Cloudflare says limits are approximate and apply separately in each location. Counts update in the background, so requests can exceed the configured limit.

const { success } = await env.RATE_LIMITER.limit({ key: userId });
 
if (!success) {
  return new Response("Too many requests", { status: 429 });
}

If your rule is "about 100 requests a minute", that is fine. If your rule is "bill the customer for exactly 100", it is not. Put that counter in one Durable Object per API key, or in a database transaction, and accept the extra latency.

"Aren't actors just queues?"

The mailbox is a queue, yes. But the queue is an implementation detail. The idea is ownership: one entity, one mailbox, one place where its state changes.

A lock, a mutex, a transaction, and an actor can all enforce ownership. They differ in where the ownership lives and what happens when the owner fails. Calling actors "just queues" is like calling a database "just a file". It is true at one level and useless at the level where you are designing.

"I can build the same thing with locks"

You can. I thought I could too, until I tried to move a user between teams.

Feature one locks the user, then the team. Feature two locks the team, then the user. Two requests arrive, each holds one lock, and both wait forever. Nobody made a mistake. The lock order was different, and that was enough.

The actor version has the same coordination problem. It just gives the coordination one address. Both features send a command to the user's object. The object runs them in order.

That is the real trade. You stop coordinating over shared memory and start coordinating over messages. It does not remove coordination. It makes the owner visible in the code.

Actors fail too, just differently

Actors do not delete failure. They move it.

Two actors can deadlock by waiting on each other's replies. Two actors can ping-pong messages forever. A request can time out after the actor already changed state. Half a credit transfer can succeed, and that is not a locking problem at all.

That last one matters. Shared-state bugs become message-flow bugs. You trade a class of bugs you can reproduce with two threads for a class you reproduce with a trace. Whether that is an upgrade depends on which one your team can debug at 2 a.m.

The actor owns state. The workflow owns the process.

For an upgrade, the account object should answer one question: what is true right now? idle, upgrade_pending, or active.

The workflow answers a different question: what happens next? Reserve the seat, charge the card, provision the workspace, activate the account. If step three fails, the workflow runs the compensations for steps one and two.

This is why I added upgrade_pending. The state transition is short. The workflow is long. A second request can read the account while the workflow is halfway through, and that is normal. The actor should not pretend the process has finished just because it accepted the command.

A coordinator is not a god object if it only sends commands and tracks the process instance. It becomes a god object when it also owns the account, the billing record, and the email preferences.

Saga, compensation, idempotency

These words get used as synonyms. They are not.

• Compensation is one undo. Refund the card, release the seat.
• A saga is a sequence of steps, each with a compensation.
• Idempotency is the rule that makes retries safe.
• An operation ID is the mechanism that lets a service ignore a stale or duplicate request.

Sagas exist because you cannot run one transaction across five services. Each service commits its own change. The saga remembers what to undo when a later step fails.

await chargeCard({
  accountId,
  operationId: `upgrade:${accountId}:${attemptId}`,
});

The payment provider sees the same operationId on a retry and returns the first result instead of charging twice. That is idempotency. It is more important than the saga, because most real failures are retries, not clean step failures.

What I got wrong

Two things.

First, I treated a business rule as a database problem. The rule was "a second upgrade request should be rejected while one is running". I never wrote that sentence down. I wrote if (state === "idle") and hoped the database would make it true.

Second, I assumed ownership had to come from infrastructure. It can come from a single atomic command, a transaction, or an actor. The right amount is the smallest one that enforces the rule.

An actor would have serialized my read and write. It would not have answered "what happens if a cancel arrives during an upgrade?" That is still a business decision. I would have had the same missing rule, with more moving parts.

Actors earn their keep when many services touch the same entity. At one service and one Redis, they are ceremony.

Three questions before picking a tool

Before you reach for Redis, a lock, or a Durable Object, answer these.

1. Who owns this state?
2. What should happen when two commands conflict?
3. What happens if step 3 of 5 fails?

The tool follows the answer. Not the other way around.

I still use Redis. I still reach for the small fix first. The difference is that I now write the rule before I write the query. Redis was never going to enforce a rule I had not decided.

Appendix: claims checked against the docs

• The Workers Rate Limiting API is approximate and applies separately in each Cloudflare location. Counts update in the background, so requests can exceed the limit. For a strict limit, Cloudflare points to Durable Objects (docs).
• A Durable Object is a single-threaded, globally addressable instance with private, strongly consistent storage. Input gates stop new events while a storage operation runs, but a network await still lets other events in (docs, gates explained).
• KV is eventually consistent, so a read-modify-write counter in KV has the same gap I hit with Redis (docs).