Designing a Chat Application

Analogy categoryAnalogies

A chat application (like WhatsApp, Slack, or iMessage) must deliver messages between users in near real time, preserve ordering within a conversation, work across multiple devices per user, tolerate flaky mobile networks, and scale to millions of concurrent connections. Unlike a typical request/response web service, chat is fundamentally about maintaining long-lived, bidirectional connections and reliably routing events to whichever server currently holds a recipient's connection — this connection-routing problem, along with message delivery guarantees, is what makes chat system design distinct from most CRUD services.

🏏

Cricket analogy: A chat app is like live ball-by-ball commentary that must reach every fan's radio in near real time and in the exact order balls were bowled, work whether they're listening on a phone or car radio, and keep broadcasting even through patchy stadium wifi — fundamentally different from a static post-match scorecard page.

Connection management and the routing problem

Clients maintain a persistent WebSocket (or similar, e.g. MQTT for mobile-optimized cases) connection to one of many stateful connection servers behind a load balancer. Because a user's connection lives on exactly one specific server at a time, sending them a message requires knowing which server that is — a stateless HTTP-style 'any server can handle any request' model doesn't directly apply. Systems solve this with a connection registry (commonly Redis) mapping user_id -> connection_server_id, updated on connect/disconnect. When server A needs to deliver a message to a user connected on server B, it looks up the mapping and forwards the message to server B via an internal pub/sub channel or RPC, and server B pushes it down the live socket.

🏏

Cricket analogy: Each broadcaster maintains a dedicated live feed to one specific production truck (not just any truck), so routing a director's cue to a specific commentator requires knowing which truck they're plugged into — a central assignment sheet maps commentator to truck, and one truck relays cues to another via an internal radio channel.

Delivery guarantees, ordering, and offline users

Messages are always persisted to durable storage (a message store, often partitioned by conversation ID) before being considered 'sent,' so delivery can be retried if the recipient is offline — the write to storage and the real-time push to a connected recipient happen independently. Ordering within a single conversation is typically preserved using a per-conversation monotonically increasing sequence number, since wall-clock timestamps across distributed servers aren't reliably ordered. For offline users, the system queues undelivered messages and pushes them (or notifies via a mobile push notification service like APNs/FCM) once the user reconnects. Read receipts and delivery receipts are usually separate, asynchronous status updates layered on top of the base message flow, not blocking parts of it.

🏏

Cricket analogy: A run is only recorded 'official' once written to the master scorebook, before the crowd sees it flash on the big screen, and deliveries are numbered sequentially per over rather than trusting stadium clocks; a fan who stepped out gets the missed overs queued up and pushed via a recap notification, with 'watched' status tracked separately from the raw commentary feed.

text

Message send flow (1:1 or small group):

1. Client A sends message over its WebSocket to Connection Server 1
2. Server 1 assigns per-conversation sequence number, writes message to durable Message Store
3. Server 1 publishes event to internal pub/sub: "deliver to user B"
4. Registry lookup: user B's active connection is on Connection Server 2 (or offline)
   -> if online: Server 2 receives event, pushes message down B's live socket
   -> if offline: message stays queued in Message Store; push notification sent via APNs/FCM
5. Client A receives a "sent" ack once Message Store write succeeds
6. Client B, on receipt, sends a delivery receipt back through the same path (async, non-blocking)
7. When B opens the conversation, client fetches any missed messages by sequence number range

WhatsApp's original architecture (built on Erlang/OTP) is a well-known real-world example of this pattern: extremely lightweight per-connection processes on connection servers, a message queue/store for durability, and reliance on sequence numbers rather than timestamps for ordering — a design choice that let a small engineering team support hundreds of millions of concurrent connections.

A common mistake is relying on wall-clock timestamps to order messages across devices and servers. Clock skew between distributed servers (and especially client devices, which users can manually change) means two messages sent moments apart can appear out of order or even identical in timestamp. A per-conversation logical sequence number, assigned by a single authoritative point (the server handling the write), avoids this entirely.

Chat systems require long-lived, stateful connections, unlike typical stateless request/response services.
A connection registry (e.g., Redis mapping user_id to server_id) is essential for routing messages to the server holding a recipient's live connection.
Messages are persisted to durable storage before being marked sent, decoupling durability from real-time delivery.
Per-conversation sequence numbers, not wall-clock timestamps, are the reliable way to preserve message ordering.
Offline delivery relies on message queuing plus mobile push notifications (APNs/FCM) to wake the client.
Read/delivery receipts are layered asynchronously on top of the core message flow and should not block message delivery.