Designing for failure: real-time software that assumes the network will drop

The important question when a connection drops isn't "will we reconnect" but "once we reconnect, what has to be rebuilt and what has to stay exactly as it was." Two different layers are involved here, and they need to behave differently.

The media layer (audio and video) tries, on its own real-time transport engine, to re-establish the path without restarting the call from scratch. But rebuilding the path itself isn't enough: some settings (received quality, audio output route) can silently revert to the engine's defaults during that rebuild, rather than to whatever the user had actually chosen. The lesson we learned in our own product was not to assume "reconnected" means "everything is back to how it was" — we explicitly reassert those settings after every recovery, because the underlying engine has forgotten them.

The signaling layer (the WebSocket connection carrying session events) follows a different rule: it never panics on disconnect, reconnects with exponential backoff — starting from a short delay, doubling on each failed attempt, capped at a bound — and silently discards malformed frames rather than breaking the whole connection over one bad message. One subtlety matters here: the backoff counter should not be reset just because a socket opened; it should wait for the first real message to arrive, because a socket that opens and immediately drops again, if it resets the counter too early, effectively creates an unbounded rapid-retry loop. This is exactly the bug we once hit in production and fixed.

A smaller but equally real point: state recovery isn't only an infrastructure concern, it's a UI concern too. A user whose video freezes for a few seconds mid-session needs to understand that the app is "recovering," not conclude that it has crashed and close and reopen it themselves — which creates a real, entirely unnecessary disconnection. A UI that completely hides automatic background recovery looks cleaner at first glance, but in practice it pushes the user into making exactly the wrong call.

More important than either of these is session identity. When a connection re-establishes, the user rejoins the same room with the same participant identity and the same valid token — this is not a fresh join, it is a recovery of the same membership. The server side carries the same assumption: a participant is only marked as having "left" once their connection is genuinely and conclusively closed, not at the first sign of trouble — because a mobile network produces plenty of false signals that look like disconnection but aren't.

When the network gets worse, the right question isn't "keep the call or drop it" but "what do we sacrifice first." We use a degradation ladder: video quality steps down first, then incoming video is dropped entirely, and in the worst case the outgoing camera is turned off too, leaving only voice. Voice is the floor, not something to be traded away — because a call without video is still a call, but a call without audio isn't a call at all.

What actually makes this ladder useful is its time asymmetry: the decision to step down happens fast (because an immediate bad experience is the worst outcome), while the decision to step back up happens slowly and cautiously. If both directions moved at the same speed, an unstable — not bad, just unstable — network would bounce quality up and down every few seconds, which is worse for the user than a steady degraded state. Stability, even at a lower level, matters more than oscillation.

Ordinary systems observe HTTP around the request: one log line per request, one status code, one response time. A real-time session stays alive for hours, and "healthy" cannot be read off a single log line. What actually needs to be recorded is: when each session connected and disconnected, which attempt number a given reconnection was and how many attempts remain before giving up, and a coarse, periodic signal indicating whether real media is actually flowing — without inspecting its content.

The other layer that needs watching is backpressure. Each client's outbound queue must be bounded; when a client falls behind and can't consume events as fast as they're produced, the system should drop the oldest events and log that drop — rather than letting the queue grow without bound until memory runs out, and rather than letting the whole system wait on one slow client.

Pulling this together into one definition: a session isn't healthy just because its socket is open. It's healthy if media is flowing continuously, if its most recent reconnection succeeded and its settings were reasserted, if its outbound queue isn't growing unboundedly, and if — even while degraded — that degradation was predictable and stepwise rather than sudden and total. This definition is deliberately more complicated than "returned 200" — because the underlying problem is more complicated than a single HTTP request.

A team that builds a real-time system and borrows its dashboard from HTTP metrics will miss exactly what its users experience: not a call that went bad, but a call that never properly connected, or one whose quality quietly degraded with no alarm ever firing. Designing for network loss means accepting from the start that "connected" is a spectrum, not a boolean.

Engineers new to real-time systems are often tempted to apply the same simple retry pattern that works for an HTTP request to a real-time session: if it fails, try again. That pattern is correct for a stateless request, because each attempt is independent of the last. A real-time session, however, carries state — who's in the room, what language they're speaking, what quality settings were chosen — and a naive retry that ignores this state effectively ejects the user from a room and drops them into a "fresh" one with the same name. The difference may be invisible from the outside, but from the inside it means losing everything that had accumulated in that session. Correct design treats state recovery as part of the contract from the start, not a feature bolted onto a simple retry afterward.