AIO — Real-Time Order Delivery & Data Integrity
"At-least-once real-time dispatch with zero financial drift under continuous hardware failure."
// 01: THE CONTEXT
Physical restaurants under peak dinner rush
In a commercial kitchen running on an All-in-One (AIO) platform, dozens of physical screens (Kitchen Display Systems, expo monitors, customer pickup displays, mobile server tablets) coordinate continuously. A dropped ticket means a cold meal, delayed orders, and furious guests.
The existing infrastructure struggled with three high-stakes engineering challenges:
- Delivery Gaps: Orders dispatched over ephemeral real-time connections vanished when tablets moved out of Wi-Fi range or experienced brownouts.
- Fragmented Liveness Telemetry: Device online/offline state was split across four disparate database tables with competing heartbeat timeouts, leaving support blind to actual hardware connectivity.
- Financial Ledger Inconsistencies: Edge-case payment bugs (such as zero-tip credit transactions and cash drawer reconciliation) accumulated discrepancies at end-of-day register closing.
// 02: ARCHITECTURE & EVENT FLOW
At-Least-Once Delivery with Explicit Device ACK & Disaster Recovery
When an order is created, it is persisted to PostgreSQL and immediately registered as a BullMQ job. The job is announced across server instances via Redis Pub/Sub, routing to the connected Server-Sent Events (SSE) socket tied to that kitchen terminal.
The job does not terminate upon network send. It remains pending until the hardware client fires an explicit cryptographic acknowledgment (Device ACK). If no ACK arrives within the timeout window, BullMQ retains the job and schedules an exponential-backoff retry (up to 10 attempts spanning a 6-hour disaster window). When the hardware terminal reboots or regains internet access, the missed orders flood through deterministically, preventing order drops.
// 03: KEY SUBSYSTEMS
Critical infrastructure shipped
Consolidated four competing status stores into a single source of truth. Heartbeat pings from kitchen terminals, barcode scanners, and handhelds stream through AWS IoT Core into an SQS ingestion queue, which populates in-memory Redis presence hashes and takes asynchronous DynamoDB disaster-recovery snapshots.
Architected and replaced a broken legacy gift card module in approximately 10 days. Designed a normalized Entity-Relationship Diagram (ERD) supporting sell, reload, transfer, and multi-tender redemption at the POS, establishing clean SOLID domain boundaries that unified three siloed engineering teams.
Resolved edge-case financial accuracy bugs across register shifts. Fixed zero-tip credit card payment ledger calculation bugs and cash-drawer closing-balance reconciliations. Reduced payment rounding tolerance thresholds from 3 cents down to 1 cent, eliminating discrepancies in managerial audit logs.
Hardened asynchronous event pipelines by implementing bulk chunking operations, transactional rollback reversibility, TTL-based cleanup routines, and aggressive elimination of N+1 database queries. Stabilized master-slave write consistency under peak weekend dinner traffic.