What happens when a job worker crashes

A background worker pulls a job off the queue, starts processing it, and halfway through is killed — an out-of-memory kill, a deploy that didn’t drain cleanly, a host that simply vanished. What happens to that job? If the answer is “it disappears,” you have a system that silently drops work, and you find out from a customer rather than a dashboard. But notice where the difficulty actually is. Re-running a job is easy once you know it needs re-running; the hard part is knowing. A healthy worker can tell you it succeeded or failed, but a worker that was killed tells you nothing at all — it just stops. So the real problem is crash detection: how does the system notice that a worker died holding a job, when the dying worker can’t say so itself?

The answer depends almost entirely on how liveness is represented, and that varies by architecture. I want to walk through two common ones. The first is in-process workers backed by BullMQ and Redis, where the queue and the detection machinery are the same Redis instance the workers talk to. The second is a message broker fronting a SQL database, where the durable record of the job and the delivery mechanism are separate systems. This post covers the first; the message-broker-and-SQL version is its own story, and it’s the next thing I’ll write.

Case study one: in-process workers with BullMQ and Redis

BullMQ and Redis turn a crash into something detectable using three cooperating pieces. The first is a per-job lock with a TTL: when a worker claims a job it creates a lock that means “this job is mine.” The lock is a lease — it lets the worker prove it’s alive and still working — and without it a worker that died mid-job would leave its job sitting in active forever, claimed but never finished. The second is lock renewal: the worker re-sets the lock on a heartbeat, comfortably inside the TTL, so a healthy worker’s lock keeps self-extending and never lapses. The third is the stalled-job sweeper: a periodic scan that finds active jobs whose lock has expired — the ones with no live owner left — and moves them back to wait to be retried, or fails them if they’ve stalled too many times. The rest of this section is those three in detail.

The thing that makes crash detection possible here is the ability to set a job lock with a TTL. When a worker picks up a job, it moves the job into an active state and takes out a lock on it in Redis — a key with a time-to-live and an owner that says “this particular worker is responsible for this particular job, and the claim is good until the TTL runs out.” The default lockDuration is thirty seconds, which is far shorter than many jobs take, and that’s deliberate. The worker doesn’t set the lock and forget it; while it’s processing, it renews the lock on a heartbeat, every lockDuration / 2 — fifteen seconds by default. As long as the worker is alive and its event loop isn’t blocked, the lock keeps getting pushed forward and never expires.

Stripped to its Redis primitives, the lock is just that — a key with a TTL and an owner token:

# Worker claims the job: set the lock only if it's unclaimed (NX),
# with a 30s TTL (PX). The value is a token identifying this worker.
SET lock:job:42 worker-7f3a NX PX 30000

# Heartbeat, every 15s while processing: push the TTL forward.
PEXPIRE lock:job:42 30000

The NX is what makes claiming a job atomic — only one worker can win the key — and the token in the value is what makes the lock owned, so a worker can only renew or release a lock it actually holds. BullMQ wraps these operations in Lua scripts so the check-and-set happens atomically, but underneath it’s this: acquire with a TTL, then keep extending the TTL as your proof of life.

That reframing is the whole trick: the lock isn’t really a duration, it’s a liveness signal. A worker proves it’s alive by continuing to renew, and a TTL turns “stopped renewing” into something Redis can observe on its own, without the worker’s cooperation. This is exactly what you need, because a crash is the one case where the worker can’t cooperate. A crashed worker stops renewing, nothing else changes — the job is still sitting in active, still locked — but within at most lockDuration the TTL elapses and the lock expires. An expired lock on an active job is the detectable fact: a job someone claimed responsibility for, whose owner is no longer proving it’s alive.

It’s worth being clear about why detection has to work this indirectly, by watching for an absence, rather than by the worker reporting its own failure. BullMQ’s completed and failed events only fire on graceful outcomes — the handler returned, or it threw and the worker caught it. A SIGKILL or a vanished host fires neither. There is no callback for “the process no longer exists.” The expired lock is the only evidence a killed worker leaves behind, which is why crash detection is built on noticing that a heartbeat stopped rather than on receiving a signal.

Turning that expired lock into action is the easy part, and it’s easy precisely because the job never really lived in the worker. The durable record — “this job exists and needs doing” — sat in Redis the whole time; the worker was only borrowing it. So a stalled-job check periodically scans the active jobs, finds the ones whose lock has expired, flags them as stalled, and moves them back to wait so a healthy worker can claim them through the normal queue. In older versions this sweep lived in a separate QueueScheduler process; in BullMQ v5 it’s built into the Worker itself. The one guardrail worth knowing is maxStalledCount, default one: a job that stalls and gets recovered too many times is suspected of being a “job of death” that crashes every worker it touches, so instead of walking it around the fleet, BullMQ marks it failed.

How fast is a crash detected

You can read the worst-case detection time straight off two timeouts. After a crash, the lock takes up to lockDuration to expire, and then the abandoned job waits up to stalledInterval — the interval on which the sweep runs, also thirty seconds by default — for the next scan to find it:

worker picks up job ──► holds lock (lockDuration = 30s), renews every 15s
        │
   worker crashes ✗   (renewals stop)
        │
   ≤ 30s later: the lock's TTL expires in Redis
        │
   next stalledInterval tick (≤ 30s): the sweep finds the expired lock
        │
        ▼
   job marked stalled ──► moved back to `wait` ──► picked up by another worker

So with the defaults, a crash can go unnoticed for up to about sixty seconds before the job is requeued. You can shrink that by lowering the timeouts, but this is where the tension lives, and it’s a detection problem, not a recovery one. Set lockDuration too low and a job that’s merely slow — or whose worker had a briefly blocked event loop and missed a single renewal — gets falsely detected as dead and run a second time, with whatever duplicate side effects that implies. Set it too high and genuine crashes take longer to notice. The right value tracks how long your handlers actually take: a worker that makes slow outbound calls, like webhooks that can run for tens of seconds, wants enough headroom that a slow-but-healthy job is never mistaken for a corpse.

Concretely, both knobs live on the Worker, not the Queue. The queue is just a handle to the jobs sitting in Redis; the detection machinery — the lock and the sweep — belongs to the thing that actually processes them:

import { Queue, Worker } from 'bullmq';

const connection = { host: '127.0.0.1', port: 6379 };

// The queue: a handle to the jobs in Redis. Producers add work here.
const queue = new Queue('emails', { connection });
await queue.add('send', { to: 'user@example.com' });

// The worker: pulls jobs and processes them. The crash-detection
// settings live here, because the worker is what holds the lock.
const worker = new Worker(
  'emails',
  async (job) => {
    await sendEmail(job.data);
  },
  {
    connection,
    concurrency: 10,
    lockDuration: 30_000,    // lock TTL before a silent job is presumed dead (default 30s)
    stalledInterval: 30_000, // how often this worker sweeps for expired locks (default 30s)
    maxStalledCount: 1,      // fail a job after it has stalled this many times (default 1)
  },
);

Stalled Job Sweeper

Each bullMQ worker will run the stalled job sweeper every stalledInterval seconds. But if have more than one worker (such as each server will have a worker), each runs its own worker against the same Redis, and so each runs its own stalled-job check loop. The natural worry is that they all redundantly sweep for expired locks every interval. They do all wake up and try, but the work is deduplicated through Redis. Before a worker runs the scan for a queue, it tries to acquire a per-queue lock key with a TTL roughly equal to the check interval; only the worker that wins it runs the scan that tick, and the others find the lock taken and skip. So across N servers you get about one detection sweep per interval, not N.

This is a textbook race condition. Every worker’s timer fires on the same stalledInterval, so they all reach for the scan at roughly the same moment. The dangerous pattern would be the check-then-act: a worker reads “has the check run recently?”, sees no, and then runs it — because between the read and the act, every other worker can read the same “no” and they all pile in. The fix is to make the check and the claim a single indivisible step, and BullMQ does that with a guard key, the stalledCheckKey, at the very top of the moveStalledJobsToWait Lua script:

-- already swept within this window? do nothing, return empty results
if rcall("EXISTS", stalledCheckKey) == 1 then return {{}, {}} end

-- otherwise claim the window: set the key with a TTL of maxCheckTime
rcall("SET", stalledCheckKey, timestamp, "PX", maxCheckTime)

-- ... only now does the actual scan for expired locks run ...

What makes this safe isn’t the two lines themselves — written as two separate round trips from the client they’d have exactly the race we’re worried about. It’s that they run inside one Lua script, and Redis executes scripts atomically on its single thread. No other command, including another worker’s copy of this same script, can interleave between the EXISTS and the SET. So in any given window, exactly one worker’s script runs first, finds the key absent, and claims it by setting stalledCheckKey with a PX TTL of maxCheckTime; every other worker’s script runs afterward, sees EXISTS == 1, and returns {{}, {}} — empty stalled and failed lists — without scanning anything. The TTL is what makes it periodic rather than one-shot: once it expires, roughly a stalledInterval later, the next window opens and one worker claims it again.

The mental model is that Redis is the single point where liveness is tracked, and the workers are symmetric peers — any of them can run the check, and Redis’s atomic scripts guarantee that for any given window and any given job, exactly one worker does the work. Adding servers buys throughput and availability without multiplying the detection load; the only cost is a little extra Redis chatter, since every worker still wakes each interval and runs the guard script even when it ends up doing nothing.

Graceful shutdown is the case that needs no detection

It’s worth saying explicitly that none of this machinery is involved in a normal deploy, because it’s easy to assume every restart leans on it. BullMQ workers expose a close() that waits for in-flight jobs to finish and releases their locks cleanly. A well-behaved shutdown — a rolling deploy that sends the signal and waits — drains the worker, so jobs either complete or are released deliberately, and nothing ever has to be detected as stalled. Detection is the safety net for the ungraceful case: the kill, the crash, the disappearance. If you find the stall checks firing during ordinary deploys, that’s usually a sign the shutdown isn’t actually draining.

What detection can and can’t promise

Detecting a dead worker lets you recover its job, but it forces an at-least-once model, and the gap is the part to design around. Picture a worker that crashes after its side effect has already happened — the webhook POST went out — but before the job was marked complete. From Redis’s point of view the job was never finished, its lock simply expired, so it’s indistinguishable from a job that died before doing anything, and it gets re-run. Detection can tell you a worker stopped proving it was alive; it cannot tell you how far the work had gotten. So handlers whose effects matter need to be idempotent, whether through a dedup key, a conditional write, or an upstream that tolerates repeats.

The other thing to keep in view is that Redis is the single source of truth for the whole scheme — the job data, the locks, the liveness state all live there. Detection is only as durable as Redis is. If Redis loses data, the locks and the jobs go with it, and there’s nothing left to notice a crash against. The mechanism assumes Redis durability, and that assumption is worth making explicit rather than discovering during an incident.

Case study two: a message broker (RabbitMQ) and a SQL job table

A message broker like RabbitMQ doesn’t have stalled-job detection of the kind we just built, and the reason is its delivery model. A broker thinks in acknowledgements: it hands a message to a consumer and waits for an ack. If the consumer acks, the broker considers the message delivered and drops it; if the channel closes before the ack, the broker redelivers. That sounds like crash recovery, but it only covers the window before the ack. The broker knows about delivery, not about completion — it has no concept of “this message was acked, the worker is partway through the real work, and now that worker has died.” Once a message is acked it’s gone from the broker’s world, and a crash after that point is invisible to it.

So to get the kind of crash detection BullMQ gets for free, you pair the broker with a database: the broker handles delivery, and a SQL table holds the durable job state plus the lock that proves a worker is still alive. This is how Firecrawl does it, and the mechanism is the same lease idea as before, just expressed in table columns instead of Redis keys. The lock lets a worker prove “this job is mine, I’m still alive and working on it,” and without it a worker that died mid-job would leave its row sitting in active forever. Two columns on the job row carry it:

-- columns on the job table
lock        uuid,
locked_at   timestamptz,

When a worker acquires a job, it stamps a fresh random UUID into lock and the current time into locked_at, and flips the row to active:

UPDATE job
SET status = 'active', lock = gen_random_uuid(), locked_at = now()
WHERE id = $1;

The worker keeps that UUID in memory, and nobody else has the value — that’s what makes the lock owned. Renewal is the heartbeat: every fifteen seconds the worker pushes locked_at forward, but only for the row it actually holds.

UPDATE job
SET locked_at = now()
WHERE id = $1 AND lock = $2 AND status = 'active';

The AND lock = $2 clause is the whole point — only the worker carrying the matching UUID can move locked_at. Finishing or failing a job uses the same guard, so if the worker’s lock was revoked while it was busy, the final write matches zero rows and becomes a no-op; the worker notices it updated nothing and learns the job was taken from it.

Detection is a separate process that plays the role the stalled-job sweeper played in Redis. Firecrawl runs it as a pg_cron job every fifteen seconds:

UPDATE job
SET status = 'queued', lock = null, locked_at = null, stalls = stalls + 1
WHERE locked_at <= now() - interval '1 minute'
  AND status = 'active'
  AND stalls < 9;
-- after 9 stalls, mark it failed instead of requeuing

Any active job whose locked_at is older than a minute is presumed orphaned — the worker crashed, was OOM-killed, or lost the network — and gets returned to the queue with its lock cleared and its stall count bumped. After nine stalls the job is declared a “job of death” and marked failed rather than bounced around forever, exactly the role maxStalledCount played in the BullMQ version.

This is where locking in a table genuinely differs from locking in Redis, and it’s worth being precise about. A Redis lock is a key with a TTL, so when the worker stops renewing, the key deletes itself — expiry is automatic, and detection is just noticing the key is gone. A SQL row has no TTL. The lock and locked_at columns don’t disappear on their own; nothing expires, so there’s no absence to notice. The expiry has to be computed by hand: the reaper reads locked_at directly and compares it against the clock — locked_at <= now() - interval '1 minute' — to decide whether the lease has lapsed. Where Redis hands you expiry as a built-in property of the key, the table approach makes liveness an explicit timestamp that some process has to keep checking.

The renew loop exists for the same reason it did in Redis: real jobs outlast the timeout. A scrape can easily run longer than a minute, and without renewal the reaper would yank a healthy job out from under its worker. Renewing every fifteen seconds against a sixty-second timeout gives four times the headroom — a worker has to miss three consecutive renewals before it’s reaped — which is the same tension between false reaps and slow detection that lockDuration carried earlier.

The choice of a UUID rather than a plain boolean “locked” flag is what makes this safe under races. If two workers ever believe they hold the same job — the reaper requeues it, then the original worker returns from a network partition and tries to finish — the UUID guarantees only the worker with the matching value can mutate the row. The other’s UPDATE ... WHERE lock = $2 matches zero rows and quietly does nothing, so there’s no double-completion and no worker overwriting another’s result. It’s the table-shaped version of Redis’s owner token.

Put end to end, the lifecycle reads almost identically to the Redis one, with the reaper standing in for the sweeper:

acquire           →  lock = UUID_A, locked_at = now, status = active
[every 15s]
  renew(UUID_A)   →  locked_at = now        (proof of life)
[on success]
  finish(UUID_A)  →  status = completed, lock = null
                     OR if the reaper already revoked it:
                     zero rows updated → worker sees it lost the job
[on crash]
  no renewal for 60s
  reaper          →  status = queued  (or failed after 9 stalls)

What we learned

We’ve now looked closely at two different scenarios for handling a worker crash. The two setups look different on the surface — one is a Redis key, the other a row in a SQL table — but they converge on the same idea, because the problem forces it. A killed worker can’t announce its own death, so liveness has to be something a worker continuously proves rather than something it reports once. Both reach for a lease the worker must keep renewing, and both pair it with a separate process that watches for renewals to stop. The only real divergence is who notices the silence: Redis gets expiry for free from a key’s TTL, while the SQL version has to compute it by reading a timestamp against the clock. Everything else — the heartbeat, the requeue, the cap on repeated stalls — is the same pattern wearing different clothes. And in both, detection buys you at-least-once and nothing stronger, so the last piece of the puzzle isn’t in the queue at all: it’s writing job handlers that can safely run twice.