106. TCP and UDP from the inside: handshake, windows, retransmissions, NAT

IP delivers packets as best it can: it loses them, reorders them, duplicates them. UDP adds only ports and a checksum on top, TCP adds a lossless byte stream: sequence numbers, acknowledgments, a window and retransmissions. This section takes both apart byte by byte, and the experiment goes further than in section 105: the section's code itself, with raw frames, performs a TCP handshake, a data exchange and a close with the Linux kernel in a neighbouring container — with no network stack on its own side.

Frames and IP are in section 105, sockets in section 78.

Terms used in the documentation

Term Meaning

segment a TCP unit of data

datagram a UDP unit of data

pseudo-header addresses, protocol and length included in the TCP and UDP checksum

three-way handshake the three-segment connection setup

initial sequence number (ISN) the first sequence number of a connection

acknowledgment number the next byte expected

maximum segment size (MSS) the largest data in one segment

receive window how much the receiver can accept

window scaling a shift applied to the window field

retransmission timeout (RTO) the timeout before resending

exponential backoff doubling the timeout on each retransmission

reset (RST) an abrupt connection reset

TIME_WAIT the waiting state after a close

network address translation (NAT) translation of addresses and ports

keepalive probes that keep a connection alive

Term	Meaning
segment	a TCP unit of data
datagram	a UDP unit of data
pseudo-header	addresses, protocol and length included in the TCP and UDP checksum
three-way handshake	the three-segment connection setup
initial sequence number (ISN)	the first sequence number of a connection
acknowledgment number	the next byte expected
maximum segment size (MSS)	the largest data in one segment
receive window	how much the receiver can accept
window scaling	a shift applied to the window field
retransmission timeout (RTO)	the timeout before resending
exponential backoff	doubling the timeout on each retransmission
reset (RST)	an abrupt connection reset
TIME_WAIT	the waiting state after a close
network address translation (NAT)	translation of addresses and ports
keepalive	probes that keep a connection alive

106.1 UDP: ports and checksum

UDP is 8 bytes: source port, destination port, length and checksum. The checksum is computed over a pseudo-header — the addresses, the protocol and the length — and over the datagram itself:

uint32_t l4_sum(uint32_t src, uint32_t dst, unsigned proto, const uint8_t *seg, unsigned n)
{
    uint32_t s = (src >> 16) + (src & 0xFFFFu) + (dst >> 16) + (dst & 0xFFFFu) + proto + n;
    for (unsigned i = 0; i + 1 < n; i += 2)
        s += rd16(seg + i);
    if (n & 1u)
        s += (uint32_t)seg[n - 1] << 8;
    while (s >> 16)
        s = (s & 0xFFFFu) + (s >> 16);
    return ~s & 0xFFFFu;                /* по сегменту с верной суммой — ноль */
}

The datagram "STM32" from 10.106.106.30:40001 to port 9106 is 9c 41 23 92 00 0d 84 72 53 54 4d 33 32: length 13, checksum 0x8472. The addresses are included in the checksum so that a datagram delivered to the wrong host fails the check. Zero in the UDP checksum field means "no checksum", so a computed zero is sent as 0xFFFF:

    uint32_t c = l4_sum(src, dst, 17u, u, 8u + n);
    wr16(u + 6, c ? c : 0xFFFFu);       /* ноль в поле значит «суммы нет» */

UDP promises nothing more: no delivery, no ordering, no protection against duplicates.

106.2 The TCP header and options

Bytes	Field
0–3	source and destination ports
4–7	sequence number — the number of the first data byte
8–11	acknowledgment number — the next byte expected
12	header length in 4-byte words
13	flags FIN, SYN, RST, PSH, ACK
14–15	receive window
16–17	checksum — as in UDP, with a pseudo-header
20 …	options: kind, length, value

Options are parsed in a loop: kind, length, value; kind 1 is a one-byte padding, kind 0 is the end of the list:

    for (unsigned i = 20; i < t->hlen;) {
        unsigned kind = s[i];
        if (kind == 0)
            break;                      /* конец списка */
        if (kind == 1) {
            i++;                        /* заполнитель для выравнивания */
            continue;
        }
        if (i + 1 >= t->hlen || s[i + 1] < 2 || i + s[i + 1] > t->hlen)
            return 0;
        if (kind == 2 && s[i + 1] == 4)
            t->mss = rd16(s + i + 2);
        i += s[i + 1];
    }

A SYN as Linux sends it has a 40-byte header: MSS 1460, SACK permitted, timestamps, padding and window scale 7. The parser found MSS 1460 and window 64240 — exactly 44 segments of 1460.

106.3 A handshake with a real kernel

Experiment: handshake, data and close built by hand, and a knock on a closed port

Our code in the container at .10 acts as 10.106.106.30 — an address its kernel does not have, so its own kernel ignores the replies. The neighbour .20 is an ordinary Python server: it greets, returns what it received in capital letters and closes the connection. Segments are built by tcp_build:

unsigned tcp_build(uint8_t *s, uint32_t src, uint32_t dst, const tcp_t *t, const uint8_t *data, unsigned n)
{
    unsigned hl = (t->flags & TCP_SYN) ? 24u : 20u;
    wr16(s, t->sport);
    wr16(s + 2, t->dport);
    wr32(s + 4, t->seq);
    wr32(s + 8, t->ack);
    s[12] = (uint8_t)((hl / 4u) << 4);  /* длина заголовка в 32-битных словах */
    s[13] = (uint8_t)t->flags;
    wr16(s + 14, t->win);
    wr16(s + 16, 0);
    wr16(s + 18, 0);
    if (hl == 24u) {
        s[20] = 2;                      /* вид 2 — MSS, длина 4 */
        s[21] = 4;
        wr16(s + 22, t->mss);
    }
    if (n)
        memcpy(s + hl, data, n);
    wr16(s + 16, l4_sum(src, dst, 6u, s, hl + n));
    return hl + n;
}

Our SYN is 24 bytes: 9c 40 1f aa 00 0f 42 40 00 00 00 00 60 02 fa f0 b5 f5 00 00 02 04 05 b4. What happened next:

SYN with sequence number 1000000 and MSS 1460 — the reply is a SYN+ACK with acknowledgment 1000001: the SYN itself takes one number. The neighbour's MSS is also 1460 = 1500 − 20 − 20: the MTU minus the IP and TCP headers.
Our ACK opened the connection, the server accepted it and sent "hello from 106" — 15 bytes with the newline.
Our request "ping 106" is 9 bytes, the reply "PING 106" is the same 9 bytes.
The server closed first — FIN; we acknowledged it and sent our own FIN, whose acknowledgment closed the connection. In total 24 bytes of data arrived.
On the neighbour the connection stayed in TIME_WAIT: this state is held by whoever closed first, so that late segments do not end up in a new connection with the same ports.

A SYN to the closed port 8107 got an RST+ACK with acknowledgment 1000001 — a refusal instead of silence. The datagram "STM32" to port 9106 came back reversed — "23MTS".

The neighbour's kernel checks the checksums and silently drops a segment with a wrong one — since it replied to every one of ours, the checksums and headers were built correctly.

106.4 Sequence numbers wrap around

int seq_before(uint32_t a, uint32_t b)
{
    return (int32_t)(a - b) < 0;
}

Sequence numbers are 32 bits, and 0xFFFFFFFF is followed by 0: 0xFFFFFFF0 comes before 0x10. Comparing by the sign of the difference works as long as the numbers are less than 2^31 apart; at exactly 2^31 it breaks — each of the two numbers turns out to be "before" the other. At 100 Mbit/s the numbers wrap in 343.6 s, at 10 Gbit/s in 3.4 s: there, timestamps protect against old segments.

106.5 Window and throughput

The sender keeps no more than the receiver's window in flight. So the throughput is no higher than the window divided by the round-trip time: 65535 bytes at 50 ms gives 10.49 Mbit/s, however fast the link is. To fill 100 Mbit/s at 50 ms, 625000 bytes must be in flight. The 16-bit window field is helped by the scale from the SYN — a left shift: a shift of 4 would be enough (up to 1048560 bytes), Linux offers 7 — up to 8388480.

An STM32 device with a 2048-byte window at the same 50 ms will squeeze out 0.33 Mbit/s: in small stacks the throughput is limited by memory, not by the wire.

106.6 Retransmissions and the timeout

The retransmission timeout tracks the response time: a smoothed mean plus four times the variation.

void rto_sample(rto_t *e, uint32_t rtt)
{
    if (e->srtt == 0) {
        e->srtt = (int32_t)rtt;
        e->rttvar = (int32_t)rtt / 2;
    } else {
        int32_t d = e->srtt - (int32_t)rtt;
        e->rttvar += ((d < 0 ? -d : d) - e->rttvar) / 4;    /* разброс — с весом 1/4 */
        e->srtt += ((int32_t)rtt - e->srtt) / 8;            /* среднее — с весом 1/8 */
    }
    uint32_t r = (uint32_t)(e->srtt + 4 * e->rttvar);
    e->rto = r < RTO_MIN ? RTO_MIN : r > RTO_MAX ? RTO_MAX : r;
}

void rto_backoff(rto_t *e)
{
    e->rto = e->rto > RTO_MAX / 2u ? RTO_MAX : e->rto * 2u;
}

Samples of 100, 100, 100, 100 ms, a spike of 300 and 100 again give timeouts of 300, 252, 216, 200, 389, 346, 312, 282 ms. The lower limit is 200 ms in Linux, 1 s in the standard. A segment left unacknowledged doubles the timeout: from 282 ms to 564, 1128, 2256 and so on up to the 120 s limit. Linux gives up after 15 retransmissions: from an initial timeout of 200 ms that is 924.6 s — a little more than 15 minutes, and all that time the application considers the connection alive.

TCP retransmission timeout: it tracks the response time and doubles when there is no response

106.7 NAT

A home router replaces the internal address and port with its own external port and remembers the pair in a table. It lets a reply from outside through only if the entry is still alive:

int nat_in(const nat_t *t, uint32_t out_port, uint32_t now, uint32_t timeout, uint32_t *ip, uint32_t *port)
{
    for (unsigned i = 0; i < t->used; i++) {
        const nat_entry_t *x = &t->e[i];
        if (x->out_port == out_port && now - x->last <= timeout) {
            *ip = x->in_ip;
            *port = x->in_port;
            return 1;
        }
    }
    return 0;
}

The model: devices 192.168.1.20 and .21 send from the same port 5000 — outside these are ports 40000 and 40001; a reply to 40001 goes to .21:5000. After 35 s of silence the entry for 40000 is already forgotten — with a UDP timeout of 30 s the reply is dropped. Device .21 sends a keepalive packet every 25 s, and at second 70 its entry is alive. Hence the rule for devices behind NAT: send something more often than the timeout. In Linux a UDP entry with no reply lives 30 s, with traffic in both directions 120 s, an open TCP connection 5 days; routers have their own timeouts, sometimes as short as half a minute. Keepalives are sent for the worst case — every 20–25 s.

106.8 Pitfalls

Your own kernel interferes with the raw socket. In the first run of the experiment no data arrived after the handshake: packet forwarding was enabled in the container, and the kernel "forwarded" replies for a foreign address. With forwarding disabled the exchange went through. On a bare-metal device this does not happen, but on a PC the raw socket and the kernel stack work at the same time.

The connection is "alive" 15 minutes after the cable was pulled. Retransmissions with doubling: 924.6 s until failure — and only when there is unacknowledged data in flight. A silent connection will not notice the break at all. TCP keepalive is enabled on the socket (SO_KEEPALIVE) and shortened with the options TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT: by default both Linux and lwIP send the first probe after 7200 s of silence, probes repeat every 75 s, and after 9 unanswered ones the connection is closed — after 7875 s, more than two hours. In lwIP these options exist only with LWIP_TCP_KEEPALIVE=1. For the sender, the time to wait for an acknowledgment is shortened by TCP_USER_TIMEOUT (Linux). Or use your own link check — with application replies.

A device behind NAT stopped receiving commands. The table entry expired: for UDP after 30 s of silence. Send keepalives more often.

TCP is stuck at hundreds of kilobits on a fast link. A small window: 2048 bytes at 50 ms gives 0.33 Mbit/s.

After a restart the server cannot bind the port. Old connections are in TIME_WAIT; give the listening socket SO_REUSEADDR.

A capture shows wrong TCP checksums on your own packets. The network card fills in the checksum after the capture, and on a virtual interface nobody does.

The comparison a < b of sequence numbers breaks once per wrap. Compare by the sign of the difference.

106.9 Summary

UDP is 8 bytes: ports, length and a checksum with a pseudo-header; nothing more.
TCP is sequence numbers and acknowledgments; SYN and FIN take one number each; MSS 1460 = 1500 − 20 − 20.
A handshake, data and a close built by hand were accepted by a real Linux kernel; TIME_WAIT stayed with the side that closed first.
Throughput is no higher than the window divided by the response time: 65535 bytes at 50 ms gives 10.49 Mbit/s.
The retransmission timeout tracks the response time and doubles; Linux gives up after 924.6 s. A silent connection without keepalive will not notice a break, and the default keepalive starts after 2 hours.
NAT forgets silence: for UDP after 30 s; send keepalives more often.

Next: section 106а "Ethernet on the F407: RMII, PHY over MDIO and DMA descriptors" — how frames get from the wire into the microcontroller's memory.

Open in the interactive handbook