Long-lived connections look free until they aren't. Most teams discover the real cost the same way: a Friday-night traffic spike where the socket layer falls over not because of throughput, but because of inventory. The connections that were quietly idle suddenly all become work at once.
We treat connection inventory as a first-class capacity metric. Throughput tells you what your fleet did last second. Inventory tells you what it has to be ready to do.
Where the cost hides
Per-connection memory is the headline cost, but it is rarely the binding one. On a modern Linux kernel, an idle TCP socket plus its userspace bookkeeping costs somewhere between 8 and 20 kilobytes depending on how diligent you are. A million of those is comfortably in the gigabytes, not the terabytes.
File descriptors and ephemeral ports bite harder. Default limits are set for workstations, not socket servers. We raise nofile and somaxconn before we raise instance size, and we monitor port-range exhaustion on the loopback as carefully as we monitor egress.
Keepalive frequency is the dial nobody touches. Too aggressive and you spend CPU on heartbeats. Too lax and you collect ghost connections for the load balancer to garbage-collect during your next deploy. We tune keepalive interval per system: tighter on mobile cellular, looser on home wifi.
What to build, what to buy
The most expensive mistake we see is teams that build their own socket termination on day one because they read a post about Discord doing it. Vendor termination at the edge is excellent until you have a reason it isn't. Have a reason, then build.
When you do build, build for graceful degradation. The socket layer that survives a regional outage is the one whose clients reconnect without thundering herd, fall back to long-polling without ceremony, and rejoin presence without asking the orchestrator a thousand times in the same minute.