The outage lasted 18 minutes. Your retries lasted 4 seconds.

Understand it one step at a time

The short runs these in order in about 50 seconds. Here they are written out — pick any step to jump the short straight to it.

1. 1

   40 orders a minute, all fine

   A queue, a worker, and a downstream API. Every setting on this queue is the Azure default.

   orders queue · 0 deep · inventory-api 200 OK

2. 2

   11:03 — the inventory API starts failing

   Not down. Worse: up, fast, and returning 503 to everything. It will do this for eighteen minutes.

   inventory-api · 503 · 180ms · outage 00:00

3. 3

   One order. Ten attempts. Four seconds

   Order 41883 arrives, fails ten times, and is dead-lettered before you could refresh a dashboard.

   order 41883 · DeliveryCount 10 · dead-lettered

4. 4

   Nothing waited between the attempts

   An abandoned message is redelivered at once. Service Bus has no backoff, and never claimed to.

   abandon() → immediate redelivery → DeliveryCount + 1

5. 5

   The outage had 17 more minutes to run

   The message needed to survive eighteen minutes. It was built to survive four seconds.

   survived 4.1s · outage 18:00 · gap 17:56

6. 6

   720 orders in the dead-letter queue

   Forty a minute for eighteen minutes. The queue did not back up — it emptied itself into the DLQ.

   orders/$deadletterqueue · 720 · outage 18:00

7. 7

   Not one of them was poison

   Every message in there is valid. Replay them now, with the API back, and all 720 succeed.

   dlq 720 · valid 720 · poison 0

8. 8

   MaxDeliveryCount is not a retry policy

   It caps how many times one bad message can hurt you. It says nothing about how long you try.

   attempts: configurable · interval: not configurable

9. 9

   Retry on a schedule, not in a loop

   Complete the message and re-enqueue a copy for later. Now your attempts span the outage instead of the first second of it.

   ScheduledEnqueueTimeUtc · 5s · 30s · 2m · 8m · 30m

10. 10

    Raising it to 100 changes nothing

    A hundred attempts with no gap is forty-one seconds. You bought 37 more seconds of an 18-minute outage.

    MaxDeliveryCount 100 · 41 seconds · still dead

11. 11

    Dead-letter the real poison on attempt one

    A message that can never succeed should not get ten tries. Your handler knows the difference; the broker does not.

    DeadLetterAsync(reason, description) · immediately

12. 12

    Ten tries is not ten chances

    The count is yours to set. The interval was never on offer — you have to build it.

    azure service bus · retry window · by design