At-least-once delivery is easy; exactly-once is a product decision

Every backend engineer reaches a point in their career where someone in product management asks for "guaranteed exactly-once message delivery." It sounds completely reasonable on a slide deck: an order is submitted, so the kitchen display should ring exactly once; a card is swiped, so the merchant ledger should debit the balance exactly once.

The reality of physical distributed networks is that exactly-once message delivery over an unreliable network is impossible. Packets drop, Wi-Fi radios switch access points, TCP timeouts trigger before an acknowledgment returns, and mobile devices brown out mid-flight.

1. The Illusion of Exactly-Once

When you dispatch an event over Server-Sent Events (SSE) or a WebSocket, you can guarantee that the event was pushed into the TCP buffer. What you cannot guarantee is whether the application process on the receiver executed the handler before crashing, or whether the ACK packet sent by the receiver made it back across the cell tower.

If you don't receive an acknowledgment, you have two choices:

• At-most-once: Never retry. If the packet dropped, the ticket is lost forever. In a busy restaurant or payment gateway, this is catastrophic.
• At-least-once: Retry until confirmed. If the packet arrived but the ACK dropped, the receiver gets the message twice.

Therefore, production engineering begins by embracing at-least-once delivery as a mechanical default, and pushing the responsibility of deduplication into a contract between the data model and the client.

2. The Idempotency Key Contract

The bedrock of safe retries is a deterministic idempotency key. In our restaurant kitchen delivery architecture (AIO), every ticket dispatch payload carries a cryptographically unique UUID generated at the moment of order inception, combined with a monotonic mutation sequence:

interface OrderDispatchJob {
  idempotencyKey: string;     // e.g. "order_9f81c4e2_seq_003"
  orderId: string;
  terminalId: string;
  attempt: number;
  payload: {
    table: string;
    items: OrderLineItem[];
    timestamp: number;
  };
}

When the Kitchen Display System (KDS) receives an event, it executes a local atomic lookup against its in-memory and local disk cache:

1. Has idempotencyKey already been acknowledged and rendered? If yes, immediately reply with an HTTP 200 Device ACK and discard the duplicate event payload.
2. If no, render the ticket to the screen, trigger the audible buzzer, commit the key to local storage, and then transmit the Device ACK.

3. Explicit Device ACK vs. Transport Ping

A common architectural defect is relying on the WebSocket/SSE ping-pong frame as proof of delivery. A ping only proves that the socket connection is open; it tells you nothing about whether the UI thread was frozen by garbage collection, whether the printer was out of paper, or whether the device crashed before rendering the order.

In our pipeline, the BullMQ worker retains the job in an active state. Only when an explicit cryptographic Device ACK is received via the upstream API does the queue worker call job.moveToCompleted().

4. Retry Budgets & Dead-Letter Isolation

Naive retries are a distributed denial-of-service attack on your own infrastructure. If a hundred kitchen devices disconnect due to a network provider outage, retrying immediately every 500 milliseconds creates an avalanche that saturates CPU and database connection pools when the network flickers.

We mitigate this with three bounded rules:

• Exponential backoff with full jitter: Retry delays scale from 2 seconds up to a maximum cap of 15 minutes, with randomized jitter to break synchronization waves.
• Fixed retry budget: A maximum ceiling of 10 retry attempts spanning a 6-hour disaster window.
• Dead-Letter Queues (DLQ): If an event exhausts its budget, it is moved to a DLQ alongside complete telemetry metadata (last error, timestamps, hardware IP), raising an urgent P1 observability alert on Grafana.