The gateway timed out. Where did the payment go?

Understand it one step at a time

The short runs these in order in about 32 seconds. Here they are written out — pick any step to jump the short straight to it.

1. 1

   11:42 — a customer pays

   The order is on the queue. A worker picks it up and calls the payment gateway.

   payments-queue · 3 waiting

2. 2

   The gateway times out

   No response in 30 seconds. Did the card get charged, or not? The worker cannot tell.

   payment gateway · 504 · no response

3. 3

   Don't complete what you can't confirm

   Complete() deletes the message for good. On an unconfirmed charge, that is how a payment disappears.

   worker · abandoning · lock released

4. 4

   It comes back automatically

   You wrote no retry loop for this. The broker redelivers, and the count goes up on its own.

   redelivered · delivery 2 / 10

5. 5

   Attempt 3 — the gateway is back

   Transient failures are the common case. Nobody was paged and nothing was lost.

   200 OK · charge ch_8812 · completed

6. 6

   But some messages never succeed

   Order #90219 is missing its currency field. The handler throws on it every single time.

   payments-queue · #90219 · handler threw

7. 7

   Ten attempts. Same failure.

   Every redelivery costs a worker slot, and the result is identical. MaxDeliveryCount is the stop.

   delivery 10 / 10 · MaxDeliveryCount reached

8. 8

   Dead-letter queue catches it

   Azure moves it to payments/$DeadLetterQueue with the reason attached. Off the queue, still stored.

   $DeadLetterQueue · MaxDeliveryCountExceeded

9. 9

   Skip it, and you lose the customer

   The DLQ is on by default — Azure fills it for you. The mistake nobody admits is never reading it.

   DLQ depth 47 · oldest 6 days

10. 10

    Failures don't vanish

    Retry absorbs the blip. The dead-letter queue holds everything else — until somebody looks.

    Azure Service Bus · retry + dead-letter