Imagine a publishing tool that generates a preview image whenever an editor saves a page. The image worker stops for an hour. The queue remains available, so every save is accepted. When the worker returns, it starts rendering all the intermediate versions the editor no longer wants.

No message was lost. Yet the latest preview is still waiting behind obsolete work. If the worker publishes results without checking their version, a slow old render can even replace a newer one.

This is an illustrative service, not an incident report. Its problem is a missing definition of usefulness. Keeping work durably and deciding whether to execute it are different responsibilities.

Decide what expires before choosing a limit

For the preview service, an editor usually wants an image for the current page revision. A request for revision 12 becomes obsolete when revision 13 replaces it, even if it has waited only a second. Age is one reason to stop; supersession is another.

Other work has different rules. An audit event should not disappear merely because it is late. A scheduled reminder may lose its purpose after the event. A requested historical export may remain useful even after the underlying data changes. Define the disposition for each class instead of applying one deletion policy to every queue.

For our preview jobs, use a compact contract like this. The deadline is a product decision to be measured and agreed, not a universal queue setting.

Identity
One job for one page revision and rendering specification.
Useful until
The revision is superseded, the page is removed, or the agreed start deadline passes.
Before rendering
Confirm that the job is still wanted and has time left to start.
Before publication
Publish only if the page still selects this revision.
If obsolete
Record a skipped outcome; do not silently describe it as a successful render.
If overdue
Record expiry and expose a deliberate way to request fresh work.

Measure the wait, not just the pile

A queue length of 500 says little without the cost of each job and the time people have already waited. Keep the original acceptance time and observe the delay to first execution. Track retries separately so a repeatedly failing job does not look like newly admitted work.

The Amazon Builders’ Library article Avoiding insurmountable queue backlogs describes message-age measurements and distinguishes first-attempt latency from retry latency. Those measurements reveal delays that a fast response to the producer cannot show.

In the preview service, also measure how long the current page revision has lacked its requested preview. That is closer to the editor’s experience than the time spent rendering superseded versions. An old job being skipped is not necessarily a service failure; a current revision receiving no useful output is.

Do not rely only on timings emitted by workers when they finish. If every worker stops, those samples stop too. Pair them with pending-work age and worker-progress observations, and distinguish “no pending work” from “no observations received.”

Check usefulness at the worker boundary

A broker’s retention mechanism can help limit waiting, but its scope matters. RabbitMQ’s TTL documentation defines message expiry in the queue and notes a delivery race: a message can expire after being written to the socket but before reaching the consumer. Message expiry in the broker is not a general deadline for an operation already running.

Carry the application’s original deadline with the job. The preview worker checks it before starting expensive rendering. Preserve that deadline across retries; giving each attempt a fresh lifetime can keep an unwanted job alive indefinitely. Across hosts, document the clock assumptions and allowed skew rather than treating timestamps as perfectly synchronized.

Check the revision again before exposing the result. A check at the start cannot prevent the editor from saving a new revision during rendering. Store generated images under revision-specific names and make selecting the current image conditional on the page still having that revision. A separate check followed by an unconditional overwrite leaves a race.

“Too late to start” and “no longer allowed to publish” are separate decisions. Give each a check at the point where it matters.

Budget the recovery before opening the floodgates

Consider an invented, simplified recovery calculation. There are 6,000 useful jobs waiting. New work arrives at 20 jobs per second, and the workers can safely complete 30 per second. Only 10 jobs per second are available to reduce the backlog, so clearing it takes about 600 seconds. This assumes uniform job cost, steady arrivals, and no retries.

If completion capacity merely matches new arrivals, the backlog does not shrink under those assumptions. Restoring the old steady-state rate is therefore different from restoring normal waiting time. AWS’s backlog discussion also cautions that downstream systems may not tolerate a surge during recovery.

For previews, discard superseded jobs according to the contract before spending rendering capacity on them. Limit concurrent renders to what the image store and page database can sustain. If fresh jobs still cannot start within the promised window, show that delay or refuse additional work explicitly. An immediate “accepted” response should not imply an immediate preview.

Make replay a new decision, not a reflex

A failed-job list is useful only if someone can decide what to do with it. Keep the job identity, original deadline, page revision, failure reason, and attempt history. Avoid retaining complete private page content when a protected reference is sufficient.

Before replaying a preview job, ask whether the referenced page still exists and whether that revision is still wanted. A deliberate request for a fresh preview should have its own recorded intent. Recovery of the original request should retain its identity and duplicate protection. Changing the identifier solely to evade a retry limit hides the history without resolving the failure.

Expose useful final states: rendered, superseded, expired, canceled, or failed with review required. They describe different outcomes for the editor. A queue reaching zero because every job expired is not the same as every requested preview becoming available.

Test the delay as part of the feature

A fast happy-path test never asks whether old work is safe. Use a controllable clock and a paused worker in an isolated test environment to exercise the contract without waiting through a real outage.

  • Before the deadline: a current revision can start and produce the expected preview.
  • After the deadline: an overdue job is recorded as expired without starting a render.
  • During rendering: a newer page revision prevents the old result from becoming current.
  • After a retry: the original identity and deadline remain intact.
  • With stopped workers: pending age grows visibly even though completion samples disappear.
  • During recovery: fresh useful work progresses without exceeding downstream limits.

Keep these results tied to their tested scope. A passing deadline test does not establish real recovery throughput; a controlled load test is a separate claim. The queue’s job is to preserve a chance to do useful work later. The application still has to decide when that chance has passed.