Designing for the six-hour outage

In a cloud-native whiteboard diagram, high availability is drawn with multiple availability zones and auto-scaling groups. But when your backend controls physical hardware — such as 500+ kitchen display screens and POS registers in restaurants — the failure modes are grounded in concrete reality:

• A backhoe cuts a municipal fiber trunk down the road from the restaurant.
• A transformer blows, plunging a kitchen into darkness for four hours.
• A rogue DHCP lease reclaims IP addresses on kitchen monitors midway through Friday night rush.

When the hardware reboots or reconnects six hours later, your backend must not choke, drop orders placed via mobile apps in the interim, or corrupt inventory counts.

1. The Single Heartbeat Pipeline

The first casualty in an outage is accurate operational awareness. Prior to our architectural overhaul at AIO, device presence was fractured: the POS wrote to one database table, the kitchen screens wrote to another, and mobile terminals maintained their own heartbeat thresholds.

We unified all hardware heartbeats into an event-driven telemetry pipeline:

[Hardware Device]
       │ (MQTT Keep-Alive, 15s interval)
       ▼
[AWS IoT Core]
       │ (Rule Action)
       ▼
[AWS SQS Buffer]
       │ (Batch Consumer Worker)
       ▼
[Redis Hash: presence:device_id]
       │ (Async Hourly Persistence)
       ▼
[DynamoDB DR Snapshot with TTL]

By decoupling device heartbeat ingestion via AWS IoT Core and SQS, even if our primary application instances were under heavy query load, device status ingestion never suffered starvation. If Redis restarted, liveness state was rebuilt from DynamoDB snapshots within seconds.

2. The Exponential Backoff Curve with a 6-Hour Ceiling

When an edge terminal drops off the network, pushing orders to it will immediately fail. If you discard the order, the kitchen misses food prep. If you retry naively every 500ms, your queue workers waste cycles attempting doomed TCP handshakes millions of times.

We established a strict 10-attempt exponential backoff schedule with full jitter designed to span an operational 6-hour disaster window:

Attempt 1:   ~2 seconds
Attempt 2:   ~8 seconds
Attempt 3:   ~30 seconds
Attempt 4:   ~2 minutes
Attempt 5:   ~8 minutes
Attempt 6:   ~20 minutes
Attempt 7:   ~45 minutes
Attempt 8:   ~1.5 hours
Attempt 9:   ~3.5 hours
Attempt 10:  ~6.0 hours (Final attempt before DLQ escalation)

When power is restored at hour five, the next scheduled retry hits the device, receives the cryptographic ACK, and smoothly flushes pending orders without overwhelming the kitchen display with a thundering herd.

3. Queue Backpressure and Chunked Flushes

A major risk when 50 devices come back online simultaneously after an extended outage is queue backpressure. Tens of thousands of delayed events attempt to flush to PostgreSQL and Redis at once.

We hardened this path by enforcing:

• Chunked Bulk Operations: Batching database writes into discrete transactions of 50 records with explicit savepoints.
• Transaction Reversibility: If item 48 fails validation, items 1–47 are preserved and recorded, rather than aborting the entire batch.
• Rate Limiting per Terminal: Edge display socket handlers throttle inbound backfill bursts to 20 tickets/second, preventing UI thread freezing on low-cost Android hardware.