Money Debited, Order Failed: Payment Webhooks and Reconciliation on Azure

“Payment succeeded but the order wasn’t created” is one of the worst tickets a food delivery app can get. The money left the customer’s account, the app said the order failed, and no restaurant ever saw it. The fix isn’t a faster server. It’s a payment webhook that saves the event before doing anything else, a Service Bus queue that finishes the work, and a reconciliation job on Azure Functions that asks the gateway every few minutes what really happened.

It’s 13:15 on Sunday, the lunch rush, and you’re the engineer on call. The first ticket reads:

“₹414 gone from my account. Your app says payment failed. Where is my biryani?”

Ten minutes later there are forty of them. Every customer has a UPI debit message from their bank. Every order in the database says PaymentFailed. And the dashboards say the payment gateway is fine.

Sunday lunch, and two fixes that made it worse

The app had recently added UPI. Unlike a card charge, a UPI payment doesn’t finish inside your API call. The app opens the customer’s UPI app, the customer approves the payment there, and the result reaches you later, by webhook.

Neither hotfix touched the cause. Scaling out doesn’t make a 14-second handler fast. And a longer expiry only moves the moment the order is wrongly marked failed. It still gets marked failed.

The query that found the missing orders

At 14:20 someone stops reading dashboards and asks the only question that matters: which captured payments have no placed order? The gateway lets you export its captured payments, loaded here into dbo.GatewaySettlement:

SELECT o.Status                AS OrderStatus,
       COUNT(*)                AS payments,
       SUM(s.Amount)           AS amount
FROM dbo.GatewaySettlement AS s
LEFT JOIN dbo.Orders AS o
       ON o.Reference = s.OrderReference
WHERE s.Status = 'captured'
  AND s.CapturedAt >= '2026-10-04T07:00:00'   -- 12:30 IST, stored in UTC
  AND (o.Id IS NULL OR o.PaidAt IS NULL)
GROUP BY o.Status;

131 customers. ₹54,234 taken, no food on the way. The 13 still PendingPayment would have been marked failed too, within minutes.

The webhook’s own telemetry in Application Insights shows why:

requests
| where timestamp between (datetime(2026-10-04 07:00) .. datetime(2026-10-04 08:30))
| where name == "POST /api/payments/webhook"
| summarize deliveries = count(),
            p95_ms     = percentile(duration, 95),
            over_5s    = countif(duration > 5000),
            failed     = countif(success == false)
    by bin(timestamp, 15m)
| order by timestamp asc

Every delivery over 5 seconds was a delivery the gateway had already given up on. Some of those handlers finished anyway, and those orders were fine. The ones on the two restarted instances never finished at all.

Root cause: the webhook was doing the whole job

The webhook handler did everything in one request: verify the signature, load the order, mark it placed, call the restaurant’s tablet API, send the customer a push, add loyalty points, commit. When the restaurant API slowed down, that request took 14 seconds, inside a database transaction.

Three things then failed together.

The gateway doesn’t wait 14 seconds. It waits 5, counts the delivery as failed and retries later, on its own schedule: 5 minutes, 30 minutes, then hours. The two instance restarts killed handlers before they committed, so those orders stayed PendingPayment. And the expiry job, written to clean up abandoned checkouts, marked every order still PendingPayment after 10 minutes as PaymentFailed. When the gateway’s retry finally arrived, the handler saw an order that was no longer pending and ignored it, returning 200. The gateway marked the event delivered. Case closed, on both sides, with the money in the wrong place.

The app was no backup. It was supposed to report success when the customer came back from the UPI app, but Android had killed it in the background on most low-memory phones, so that call never happened.

If I had to put it on a sticky note: the only record that the customer had paid lived inside a slow HTTP request, and nothing ever asked the gateway again.

The same problem every async payment has

This isn’t a UPI problem. Any payment that completes outside your API call has it: UPI, wallets, 3-D Secure card checks, bank redirects, buy-now-pay-later. The gateway knows the truth first, and it tells you by webhook. Gateways retry webhooks, which helps, but they all say the same thing in their docs: deliveries can be late, out of order, or repeated, and you shouldn’t depend on them alone.

So the design needs three properties:

1. A webhook is never lost once it reaches you. Save it before doing anything with it.
2. Processing it twice is harmless. A retried webhook, or one that arrives twice, changes nothing the second time.
3. A webhook that never reaches you is found anyway. Something on your side asks the gateway on a schedule.

Let’s build each one on Azure. Here’s the whole idea first, without any code:

The new flow: order first, then money, then the event

The order already exists before payment, as PendingPayment, with an idempotency key from the app. That part came from the double tap that charged twice. The one addition is that the API passes the order’s reference, ord-1042, to the gateway when it creates the payment. Every payment the gateway takes can now be traced back to exactly one order, by a value you chose.

The webhook: verify, save, enqueue, 200

The new webhook is an HTTP-triggered Azure Function, and it does four things only. It runs on the .NET isolated worker with the ASP.NET Core integration, so it gets HttpRequest and IActionResult:

public class PaymentWebhook(
    PaymentEventStore events,
    [FromKeyedServices("payment-events")] ServiceBusSender queue,
    IOptions<GatewayOptions> gateway)
{
    [Function("PaymentWebhook")]
    public async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "payments/webhook")] HttpRequest req,
        CancellationToken ct)
    {
        // 1. The signature covers the exact bytes the gateway sent. Read them raw;
        //    JSON that has been parsed and re-serialized will not match.
        using var buffer = new MemoryStream();
        await req.Body.CopyToAsync(buffer, ct);
        byte[] body = buffer.ToArray();

        if (!IsSigned(body, req.Headers["X-Gateway-Signature"]))
            return new UnauthorizedResult();

        var evt = JsonSerializer.Deserialize<GatewayEvent>(body)!;

        // 2. Save first. A redelivered event hits the unique index, and that is fine.
        await events.TryInsertAsync(evt, Encoding.UTF8.GetString(body), ct);

        // 3. Hand the work to the queue. If this throws, the 500 makes the gateway
        //    retry, and the reconciliation timer re-queues unprocessed events anyway.
        await queue.SendMessageAsync(new ServiceBusMessage(body)
        {
            MessageId = evt.Id,
            Subject = evt.Type
        }, ct);

        // 4. Done. No order update, no restaurant call, no push.
        return new OkResult();
    }

    private bool IsSigned(byte[] body, string? signature)
    {
        if (string.IsNullOrEmpty(signature)) return false;

        byte[] expected = HMACSHA256.HashData(gateway.Value.WebhookSecret, body);
        byte[] given;
        try { given = Convert.FromHexString(signature); }
        catch (FormatException) { return false; }

        return CryptographicOperations.FixedTimeEquals(expected, given);
    }
}

The header name and encoding are this scenario’s gateway; yours will document its own, and many also send a timestamp you should check to refuse replayed events. Two details matter whatever the gateway. Compare with FixedTimeEquals, never ==, so the comparison time leaks nothing about the secret. And keep the webhook secret in Key Vault, read through a Key Vault reference in the app settings, not in code.

TryInsertAsync is a plain insert into a table whose only job is to remember every event the gateway ever sent:

CREATE TABLE dbo.PaymentEvents (
    Id               bigint IDENTITY  PRIMARY KEY,
    GatewayEventId   varchar(64)      NOT NULL,
    EventType        varchar(64)      NOT NULL,
    OrderReference   varchar(32)      NOT NULL,
    GatewayPaymentId varchar(64)      NOT NULL,
    Amount           decimal(10, 2)   NOT NULL,
    Payload          nvarchar(max)    NOT NULL,
    ReceivedAt       datetime2        NOT NULL DEFAULT SYSUTCDATETIME(),
    ProcessedAt      datetime2        NULL
);

CREATE UNIQUE INDEX UX_PaymentEvents_GatewayEventId
    ON dbo.PaymentEvents (GatewayEventId);

CREATE INDEX IX_PaymentEvents_Unprocessed
    ON dbo.PaymentEvents (ReceivedAt)
    WHERE ProcessedAt IS NULL;

A second delivery of evt_9f2 fails with error 2601, duplicate key. The code catches it, treats it as “already saved”, and still sends to the queue, because the first attempt may have saved the row and then died before the send. Sending twice is harmless; you’ll see why in the processor.

The answer goes back in about 120 milliseconds, whatever the restaurant API is doing. The gateway’s 5-second timeout stops mattering.

The processor: one event, one conditional update

The Service Bus queue payment-events has duplicate detection switched on, which needs the Standard tier or above, and a max delivery count of 10. A Service Bus-triggered Function takes each event:

[Function("ProcessPaymentEvent")]
public async Task Run(
    [ServiceBusTrigger("payment-events", Connection = "ServiceBus")] ServiceBusReceivedMessage message,
    CancellationToken ct)
{
    var evt = message.Body.ToObjectFromJson<GatewayEvent>();

    if (evt.Type == "payment.captured")
    {
        var result = await orders.MarkPaidAsync(evt.OrderReference, evt.PaymentId, evt.Amount, ct);

        switch (result)
        {
            case MarkPaid.Placed:        // the normal case
            case MarkPaid.AlreadyPaid:   // a redelivery: nothing to do
                break;

            case MarkPaid.OrderClosed:   // money arrived for an order that was cancelled or expired
            case MarkPaid.SecondPayment: // the customer paid twice for one order
                await payments.RefundAsync(evt.PaymentId, evt.Amount,
                                           idempotencyKey: $"refund-{evt.PaymentId}", ct);
                await notify.RefundedAsync(evt.OrderReference, result, ct);
                break;

            case MarkPaid.AmountMismatch:
                await orders.FlagAsync(evt.OrderReference, "NeedsAttention", ct);   // a person decides
                break;
        }
    }

    await events.MarkProcessedAsync(evt.Id, ct);
}

If anything throws, the Function fails, Service Bus makes the message visible again and retries it. After 10 attempts it goes to the dead-letter queue, where an alert waits for it. Nothing is ever silently dropped.

MarkPaidAsync is the whole correctness of the design in one statement:

BEGIN TRANSACTION;

UPDATE dbo.Orders
SET    Status           = 'Placed',
       GatewayPaymentId = @paymentId,
       PaidAt           = SYSUTCDATETIME()
WHERE  Reference = @orderRef
  AND  Status    = 'PendingPayment'      -- 0 rows if it was already placed, or closed
  AND  Total     = @amount;              -- 0 rows if the amount is not what we asked for

IF @@ROWCOUNT = 1
    INSERT INTO dbo.Outbox (MessageId, QueueName, Body, CreatedAt)
    VALUES (@orderRef, 'order-placed', @body, SYSUTCDATETIME());

COMMIT;

When the update changes one row, the order is placed, and in the same transaction an outbox row asks for the restaurant notification and the customer push. That’s the transactional outbox from the hotel booking saga, and it’s why the slow restaurant API can no longer hold a payment hostage: it runs later, from its own queue, with its own retries.

When the update changes zero rows, the code reads the order once to learn why, and returns AlreadyPaid (same payment ID), SecondPayment (a different one), OrderClosed or AmountMismatch. A redelivered event, a duplicate message that slipped past duplicate detection, and a reconciliation run that raced the webhook all end up in AlreadyPaid. Duplicate detection saves work; the conditional update is the guarantee.

Reconciliation every five minutes, with a timer Function

The webhook path now survives slowness and restarts. It still can’t survive a webhook that never arrives: a gateway incident, a misconfigured URL after a deploy, a firewall rule. Reconciliation covers that, by asking the gateway directly.

[Function("ReconcilePayments")]
public async Task Run([TimerTrigger("0 */5 * * * *")] TimerInfo timer, CancellationToken ct)
{
    foreach (var order in await orders.PendingPaymentOlderThanAsync(TimeSpan.FromMinutes(3), ct))
    {
        var payment = await gateway.GetByOrderReferenceAsync(order.Reference, ct);

        switch (payment.State)
        {
            case GatewayPaymentState.Captured:
                // The webhook never landed. Feed the same queue; the processor decides.
                await queue.SendMessageAsync(new ServiceBusMessage(
                    BinaryData.FromObjectAsJson(GatewayEvent.Captured(payment)))
                {
                    MessageId = $"recon-{payment.Id}"
                }, ct);
                break;

            case GatewayPaymentState.Failed:
                await orders.MoveAsync(order.Reference, "PendingPayment", "PaymentFailed", ct);
                break;

            case GatewayPaymentState.Pending when order.Age >= TimeSpan.FromMinutes(15):
                // Close it at the gateway first, so a late approval cannot take money.
                if (await gateway.CancelAsync(order.Reference, ct))
                    await orders.MoveAsync(order.Reference, "PendingPayment", "Expired", ct);
                break;
        }
    }

    // Events saved by the webhook whose queue send never happened.
    await events.RequeueUnprocessedAsync(olderThan: TimeSpan.FromMinutes(2), queue, ct);
}

It looks like a small loop, but four things about it matter.

The old expiry job is gone. An order is no longer marked failed because a clock ran out. It’s marked failed when the gateway says the payment failed, or expired after the gateway confirms the payment can no longer be completed. Time alone never decides.

Reconciliation doesn’t place orders itself. It puts a synthetic event on the same queue, so there’s exactly one piece of code that turns money into an order. Two paths into one processor, and the conditional update makes them safe to race.

A timer trigger runs on one instance at a time, even when the Function app scales out, because the runtime takes a blob lease for it. Two runs can’t ask the gateway the same question at once.

And the load is tiny. On a normal day the query returns a handful of orders, the ones whose customers are still deciding in the UPI app, so it’s a few gateway calls every five minutes, well inside any gateway’s rate limit.

What the app shows while it waits

The app changes too, in one small way: it never says “failed” on its own. In CheckoutBloc, when the customer comes back from the UPI app, or opens the app after Android killed it, the order is read from the API, and a pending order gets its own state:

// lib/features/checkout/bloc/checkout_bloc.dart, after returning from the UPI app
final Result<Order> result = await _getOrder(state.placedOrder!.id);
result.fold(
  (Failure failure) => emit(state.copyWith(status: CheckoutStatus.confirmingPayment)),
  (Order order) => emit(switch (order.paymentStatus) {
    PaymentStatus.paid    => state.copyWith(status: CheckoutStatus.placed, placedOrder: order),
    PaymentStatus.failed  => state.copyWith(status: CheckoutStatus.paymentFailed),
    PaymentStatus.pending => state.copyWith(status: CheckoutStatus.confirmingPayment),
  }),
);

“Confirming your payment” is honest. The push from the outbox arrives the moment the processor places the order, so the customer rarely sees that screen for more than a few seconds, and it’s never wrong.

How efficient is the new design?

Each layer catches a different failure, and the timer catches everything the others miss. Here is where Sunday’s 131 customers would have ended up:

Side by side, for two Sunday lunches:

What it comes down to: the longest a paid customer can wait without an order is one reconciliation interval, whatever the webhook does. Webhook health now decides how fast an order appears, not whether it does. And the gateway calls per run equal the orders still pending after three minutes, which grows with abandoned checkouts, not with traffic.

When to use this design, and when it’s too much

Even in the right-hand column, keep one part: never do slow work inside a webhook. Verify, save, answer. That costs nothing and removes the most common failure.

Why the obvious fixes were wrong

• “Scale out the App Service plan.” Six instances running a 14-second handler are six instances holding locks for 14 seconds. The handler had to get shorter, not wider.
• “Raise the expiry to 30 minutes.” It moves the wrong decision later. Customers abandon, and a webhook delayed 31 minutes is still lost.
• “Ask the gateway to wait longer.” Most gateways don’t let you, and the restart would still kill handlers.
• “Let the app confirm the payment instead.” The app is the least reliable party in the flow. Android kills it, networks drop, customers close it.
• “Ignore events for orders that aren’t pending.” That’s exactly how the money disappeared.
• “Retry the webhook handler with Polly.” A retry inside a request the gateway has already abandoned makes the request longer.

The guardrail: one query and two alerts

Nothing should stay unresolved for long. This query lists every order the design has failed to settle:

SELECT Reference, Status, Total, CreatedAt,
       DATEDIFF(minute, CreatedAt, SYSUTCDATETIME()) AS minutes_open
FROM dbo.Orders
WHERE (Status = 'PendingPayment' AND CreatedAt < DATEADD(minute, -20, SYSUTCDATETIME()))
   OR  Status = 'NeedsAttention'
ORDER BY CreatedAt;

An order pending for 20 minutes means the reconciliation timer itself isn’t running, or the gateway’s status API is down. Run it as an Azure Monitor log alert, or from the timer Function itself, and page on any row.

The second alert is on the dead-letter queue of payment-events: an Azure Monitor metric alert on DeadletteredMessages greater than zero. A dead-lettered payment event is money nobody has decided about yet.

Once a day, the 14:20 query from the top of this article runs against the gateway’s settlement export: captured payments with no placed order. It should always return nothing. On the day it returns a row, you hear about it before the customer does.

System design interview questions from this design

1. The payment succeeded, but the order shows failed. Walk through what happened and design the fix. Look for: the webhook as the source of truth, save-first, queue processing and reconciliation.
2. How do you make a webhook handler idempotent? A unique index on the gateway’s event ID, and a conditional update on the order’s state.
3. Why answer 200 before the order is updated? The gateway only needs to know the event is safe. Doing the work inside the request ties the payment to the slowest dependency.
4. The gateway is down for an hour and sends no webhooks. What happens? Orders wait as pending; the app shows “confirming”. When the status API returns, reconciliation places every captured payment within one run.
5. A payment arrives for an order the customer cancelled. What do you do? Decide explicitly: refund automatically with an idempotency key, and tell the customer. Never ignore it.
6. Why does reconciliation enqueue an event instead of placing the order directly? One code path turns money into an order, so the two sources can race safely.

What to do on day one, not after the first refund

• Create the order before the payment, and give the gateway your order reference.
• Make the webhook verify, save, enqueue and answer. Nothing else.
• Process payment events from a queue with retries and a dead-letter queue.
• Never mark an order failed because a clock ran out. Ask the gateway.
• Run reconciliation from the first release, and the settlement query every day.

Key takeaways

• A payment webhook is an event to save, not a request to finish. Save first, answer 200 fast.
• Process events from Service Bus with a conditional update, so duplicates and retries are harmless.
• An event that doesn’t fit the order’s state is money in the wrong place. Decide what happens; don’t ignore it.
• A reconciliation timer that asks the gateway bounds every failure to one interval.
• Only the gateway can say a payment failed. The clock cannot.

The next Sunday

A week later the restaurant API has a bad lunch again. Its p95 touches 11 seconds at 13:00.

The webhook doesn’t notice. It answers in 120 milliseconds all afternoon. The order-placed queue backs up for eight minutes and restaurants get their orders a little late, but every order is placed the moment its payment lands.

At 13:40 an App Service deployment points the webhook URL at a slot that’s still warming up, and 22 webhooks fail. Nobody notices that either. The reconciliation timer finds 22 captured payments at 13:45 and places them. The worst wait, for one customer, is four minutes and fifty seconds.

The settlement query on Monday morning returns no rows.

Test yourself: answer in the comments