How TCP/IP Works Under the Hood

Your Node.js server calls res.send("hello") and somewhere, 1400 bytes cross the planet in 80 milliseconds and appear correctly in a browser. That's wild. But what actually happens? TCP/IP is so well-abstracted that most developers never need to think about it — until they hit a mysterious latency spike, a connection limit, or a TIME_WAIT storm in production. Then it helps a lot to know what the kernel is doing.

Let's go from send() all the way down.

The Mental Model: Four Layers, One Job

TCP/IP is really two separate protocols layered together. IP handles addressing and routing — getting a packet from point A to point B. TCP sits on top and adds reliability — ensuring all packets arrive, in order, exactly once.

Each layer adds its own header going down, strips it going up. The data itself is never copied between layers — the kernel threads a pointer chain through the sk_buff structure, which is Linux's core packet abstraction. This is why adding a TCP header costs nanoseconds, not microseconds.

IP: The Routing Layer

IP is connectionless and has zero memory. Every packet is independent. The IP header is 20 bytes minimum:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |    DSCP   |ECN|         Total Length          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Identification        |Flags|      Fragment Offset    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Time to Live |    Protocol   |         Header Checksum       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Source Address                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination Address                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The TTL field starts at 64 (Linux) or 128 (Windows) and decrements at each router hop. When it hits zero the packet is dropped and an ICMP "time exceeded" message is sent back — that's how traceroute works.

The Protocol field (6 = TCP, 17 = UDP, 1 = ICMP) tells the receiving kernel which upper-layer handler to dispatch to.

Fragmentation happens when a packet exceeds the link's MTU (1500 bytes for Ethernet). The IP layer splits it into fragments with matching Identification values, and the receiving side reassembles them. In practice, you want to avoid fragmentation — the Path MTU Discovery mechanism sends packets with the DF (Don't Fragment) bit set and adjusts based on ICMP "fragmentation needed" responses. This is why setting jumbo frames requires consistency across your entire network path.

TCP: The Reliability Layer

TCP's job is to make an unreliable IP network look like a reliable byte stream. It does this with three mechanisms working simultaneously: sequencing, flow control, and congestion control.

The TCP Header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Acknowledgment Number                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Data |           |U|A|P|R|S|F|                               |
| Offset| Reserved  |R|C|S|S|Y|I|            Window             |
|       |           |G|K|H|T|N|N|                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Checksum            |         Urgent Pointer        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The Sequence Number and Acknowledgment Number are the core of TCP's reliability guarantee. Each byte in the stream has a sequence number. The receiver acknowledges by sending back the next byte it expects. If segments arrive out of order, TCP holds them in a receive buffer and delivers them to the application in order.

The Initial Sequence Number (ISN) is intentionally random — chosen using a cryptographic algorithm in modern kernels — to prevent sequence number injection attacks from the 1990s era of the internet.

Connection Lifecycle: What Actually Happens in a Handshake

# On the server side, this is what the kernel does before your accept() call
[SYN received] → added to SYN queue (incomplete connections)
[ACK received] → moved to accept queue (established, waiting for accept())
[accept() called] → dequeued, fd returned to application

Two kernel queues are involved here. The SYN queue (controlled by tcp_max_syn_backlog) holds half-open connections — SYN received, waiting for the final ACK. The accept queue (controlled by min(backlog, somaxconn)) holds fully established connections waiting for the application to call accept(). When your server is slow to call accept() under load, the accept queue fills up and the kernel starts dropping or refusing new connections. This is why you want your accept() loop fast and your worker threads plentiful.

Connection States

The TCP state machine has 11 states. The ones you actually care about:

• ESTABLISHED — normal data transfer
• TIME_WAIT — connection closed by this side, lingering so late-arriving segments from the old connection can't be mistaken for the new one
• CLOSE_WAIT — remote side closed, local side hasn't closed yet (common source of connection leaks)
• SYN_RCVD — in the SYN queue, waiting for final ACK

TIME_WAIT is worth getting exactly right, because the number people quote is neither the spec's nor Linux's. RFC 9293 says a connection "MUST linger in the TIME-WAIT state for a time 2xMSL", and defines MSL in its glossary as "the time a TCP segment can exist in the internetwork system. Arbitrarily defined to be 2 minutes." Two times two minutes is four minutes. Linux doesn't do that; it hardcodes 60 seconds in include/net/tcp.h:

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
				  * state, about 60 seconds	*/

A compile-time constant, note — there is no sysctl to change it, which is why every "tune your TIME_WAIT timeout" guide reaches for tcp_tw_reuse instead. So: 240 seconds per the RFC, 60 on Linux, and 120 is a number that appears in a lot of blog posts and no source tree.

# Check connection states on a Linux server
ss -s
# Or per-connection breakdown:
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

Flow Control: Sliding Window

The receiver advertises how much buffer space it has via the Window field. The sender must not have more unacknowledged data in flight than the receiver's window. This prevents a fast sender from overwhelming a slow receiver.

The effective window can grow beyond 65535 bytes (the 16-bit limit) via the Window Scale TCP option, which extends it to 30 bits — allowing windows up to 1GB. This matters a lot for high-bandwidth, high-latency links. The Bandwidth-Delay Product (BDP) tells you the optimal buffer size:

BDP = bandwidth × RTT
# Example: 1 Gbps link, 100ms RTT
BDP = 1,000,000,000 / 8 × 0.1 = 12.5 MB

If your TCP window is smaller than the BDP, you're leaving throughput on the table even when the link is idle.

Congestion Control: How TCP Shares the Network

This is where it gets clever. TCP has no explicit signal from the network about congestion — it infers it from packet loss and delay. The congestion window (cwnd) is an additional constraint on how much data can be in flight, independent of the receiver's window. The actual send limit is min(cwnd, receive_window).

Slow Start — deceptively named, it's actually exponential growth. cwnd starts at ~10 MSS (roughly 14KB), and doubles every RTT until either a loss occurs or cwnd reaches the slow start threshold (ssthresh).

Congestion Avoidance — after reaching ssthresh, growth becomes linear: +1 MSS per RTT. This is the steady-state for a long-lived connection.

Fast Retransmit + Fast Recovery — on receiving 3 duplicate ACKs (meaning a packet was lost but later packets arrived), the sender retransmits immediately without waiting for a timeout. Then cuts ssthresh in half and sets cwnd = ssthresh + 3.

CUBIC (Linux default since 2.6.19) replaces linear growth with a cubic function of time since the last congestion event. Growth is aggressive far from the last loss point and flattens as it approaches it, which recovers a large window in far fewer RTTs than +1 MSS per round trip would. That's the whole reason it displaced Reno on high-BDP paths: at 10 Gbps and 100 ms RTT, linear growth needs minutes to refill a window.

BBR (Bottleneck Bandwidth and Round-trip propagation time), developed by Google in 2016, is a fundamentally different approach. Instead of reacting to packet loss, BBR continuously estimates the available bandwidth and minimum RTT, probes to keep those estimates fresh, and paces to the model rather than to loss signals. Loss-based control has a structural problem on fast, long paths: to fill the pipe it needs the loss rate below roughly the inverse square of the BDP, which for a 10 Gbps / 100 ms path means fewer than about one loss in 30 million packets.

The gap that opens up when that assumption fails is genuinely enormous, and there is a specific published measurement rather than a vibe. From Google Cloud's 2017 announcement, verbatim:

Consider a typical server-class computer with a 10 Gigabit Ethernet link, sending over a path with a 100 ms round-trip time (say, Chicago to Berlin) with a packet loss rate of 1%. In such a case, BBR's throughput is 2700x higher than today's best loss-based congestion control, CUBIC (CUBIC gets about 3.3 Mbps, while BBR gets over 9,100 Mbps).

Two things to hold onto before you repeat that figure. It is an emulated single-flow microbenchmark at one specific operating point, and Google says so in the same post — these benchmarks "illustrate the nature (though not necessarily the typical magnitude) of the advantages." Their production result is much more modest and much more useful: YouTube throughput improved "by 4 percent on average globally — and by more than 14 percent in some countries." Quote the 4% if you want to set an expectation; quote the 2700x only with the conditions attached.

# Check current congestion control algorithm
sysctl net.ipv4.tcp_congestion_control
# Switch to BBR
sysctl -w net.ipv4.tcp_congestion_control=bbr

What Happens Inside the Kernel on send()

When your application calls send(), the kernel doesn't immediately put bytes on the wire:

// Simplified path for a TCP send in the Linux kernel
sock_sendmsg()
  → tcp_sendmsg()
    → tcp_sendmsg_locked()  // appends to sk_write_queue
      → tcp_push()          // flushes pending segments
        → __tcp_push_pending_frames()
          → tcp_write_xmit()    // respects cwnd and rwnd
            → tcp_transmit_skb() // builds TCP header
              → ip_queue_xmit()  // routes, builds IP header
                → dev_queue_xmit() // hands to NIC driver

The NIC driver and hardware maintain a ring buffer. The driver deposits sk_buff pointers there, and the NIC uses DMA to read data directly from memory without CPU involvement. On the receive side, the NIC interrupts the CPU (or uses NAPI polling under load), and the kernel drains the ring buffer.

Practical Implications

TIME_WAIT is not a bug. Thousands of TIME_WAIT sockets is the protocol working correctly — ensuring stragglers from a dead connection can't be accepted into a new one on the same four-tuple. On Linux each one occupies its local port for 60 seconds, so a service making many short-lived outbound connections runs out of ephemeral ports long before it runs out of anything else. The fixes, in order of how much they help: reuse connections (pooling, HTTP keep-alive), then net.ipv4.tcp_tw_reuse=1, then widen net.ipv4.ip_local_port_range. SO_REUSEADDR is not on that list — it lets a listener bind a port that has TIME_WAIT sockets on it, which is a different problem from an outbound connector exhausting ports. And don't reach for tcp_tw_recycle: it broke NAT'd clients and was removed from the kernel in 4.12.

Buffer sizing matters at scale. Linux auto-tunes TCP buffers, but the defaults cap out around 4MB. On high-BDP paths:

# Check and tune TCP buffer sizes
sysctl net.core.rmem_max        # max receive buffer
sysctl net.core.wmem_max        # max send buffer
sysctl net.ipv4.tcp_rmem        # min default max for receive
sysctl net.ipv4.tcp_wmem        # min default max for send

The accept queue is where your app interacts with TCP. If your application is slow — maybe it's doing blocking I/O in the accept path, or your worker pool is exhausted — the accept queue fills and the kernel drops connections. You'll see this in your TCP stats as ListenOverflows in netstat -s or /proc/net/netstat. This is usually a better signal than CPU or memory when diagnosing connection refusals.

SACK makes recovery better. TCP Selective Acknowledgment lets the receiver acknowledge non-contiguous ranges. Without it, a single lost packet forces retransmission of everything sent after it. SACK is negotiated in the handshake and is on by default in modern kernels. If you're debugging performance on a lossy link and SACK is disabled, turn it on first.

RTT is your real limit. Light in fiber does about 200,000 km/s. New York to London is roughly 5,500 km, so 27.5 ms one way — and 55 ms round trip, which is the number that matters, and which people routinely halve by accident. Real measured RTT on that path is closer to 70 ms once you add routing that isn't a great circle and switching at every hop. TCP needs one RTT for the handshake before a byte of payload moves, then one RTT per congestion window during slow start. A new connection to a distant server is slow for several round trips no matter how much bandwidth you bought, which is why connection reuse — keep-alive, HTTP/2 multiplexing, pooling — beats raw bandwidth for latency-sensitive workloads every time.

The next time your service has connection issues, don't reach for logs first. Run ss -s and netstat -s — the TCP state machine usually tells you exactly what's wrong faster than any application-level trace.