The queue drained. Nothing was consuming it.

Understand it one step at a time

The short runs these in order in about 32 seconds. Here they are written out — pick any step to jump the short straight to it.

1. 1

   09:12 — the deploy goes out

   Three instances of orders-worker, health checks green, consumer connected. An ordinary Tuesday.

   14 March · deploy 4f21a9 · all green

2. 2

   09:40 — the consumer stops

   A serialisation bug in the new build. The handler throws on start-up and never receives anything.

   09:40 · 0 consumers · depth climbing

3. 3

   10:41 — the queue starts draining

   Depth begins falling. Nothing is consuming. No worker has come back. Read that twice.

   10:41 · depth falling · 0 consumers

4. 4

   The messages are just gone

   No dead-letter entry. No exception. No log line. The graph looks like recovery — it is deletion.

   DLQ 0 · errors 0 · orders missing

5. 5

   The setting I skipped

   Somebody set a one-hour TTL so stale orders would not pile up. That is the trap, and it is armed by a default.

   TTL 01:00:00 · dead-lettering on expiration: false

6. 6

   With it on, they survive

   Same bug, same six hours, same TTL. The expired orders land in the dead-letter queue instead of nowhere.

   DLQ 4,318 · TTLExpiredException · replayable

7. 7

   We found out from customers

   Not from an alert. Support tickets at 16:20, four hours after the graph said everything was fine.

   16:20 · 61 tickets · “where is my order?”

8. 8

   Three things failed

   The bug was one. The one-hour TTL was two. The default nobody read was three — and only three was silent.

   postmortem · 3 contributing causes

9. 9

   The four-line fix

   Three properties on the queue and one alert rule. Ten minutes of work, and none of it is application code.

   TTL · dead-letter on expiry · retries · alert

10. 10

    Defaults are decisions

    Somebody chose that default, and it was not you. Read the properties on anything holding your data.

    Azure Service Bus · read the defaults