106в. lwIP on STM32: buffers, three APIs and memory

In section 106а frames landed in the DMA descriptor ring, and in section 106б we took them apart byte by byte: Ethernet, ARP, IPv4, ICMP, UDP and TCP. In the firmware all of this work is done by a TCP/IP stack. On STM32 that is usually lwIP: CubeMX puts it into the project, and the ST examples are built on it. In section 156 lwIP is hidden inside ESP-IDF. Here we put it together ourselves: how a frame becomes a pbuf buffer, which settings decide how much RAM it takes, and how the three ways of talking to it differ.

Sources: lwIP 2.1.2 from the STM32Cube_FW_F4 V1.28.0 package (opt.h, pbuf.c, tcp.c, api_msg.c, doc/doxygen/main_page.h) and the ST examples for the STM324xG_EVAL board — LwIP_TCP_Echo_Server (no OS) and LwIP_HTTP_Server_Socket_RTOS (with FreeRTOS). The experiment: lwIP built with gcc 12 under Linux in a container, talking to the Linux kernel through a tap interface. The sizes come from an armclang build for the F407 with the driver from 106а. No board was used in the experiment.

Terms used in the documentation

Term	Meaning
netif	the stack's network interface: addresses and receive and transmit functions
ethernetif.c	the "glue" between the MAC driver and lwIP in the ST examples
pbuf (packet buffer)	a packet buffer; can be a chain of links
memp	pools of equal-size blocks
heap, MEM_SIZE	the lwIP heap — one area, blocks of different sizes
raw API, callback-style API	callbacks; works without an OS
sequential API, netconn	a sequential API on top of threads
socket API	BSD-style sockets on top of netconn
tcpip_thread	the OS thread in which the stack core lives
core locking	the core lock: LOCK_TCPIP_CORE, UNLOCK_TCPIP_CORE
zero-copy	receiving without copying the frame

106в.1 What lwIP is made of

lwIP has three layers.

• netif — the network interface. It is filled in by the file ethernetif.c: three functions, low_level_init (MAC address, MTU, flags), low_level_output (a pbuf chain into DMA) and low_level_input (a frame from DMA into a pbuf).
• The core — ip4.c, etharp.c, tcp_in.c, tcp_out.c, dhcp.c and others. The core functions are not protected against being called at the same time from different places.
• The APIs for the application: raw API, netconn and sockets (106в.5 and 106в.6).

Our netif106d.c is built the same way as ST's ethernetif.c, but simpler: the frame is copied. The init function installs two transmit functions — IP goes out through ARP, a finished frame goes through the driver:

err_t ethernetif_init(struct netif *netif)
{
    netif->name[0] = 's';
    netif->name[1] = 't';
    netif->output = etharp_output;           /* IP → ARP → кадр */
    netif->linkoutput = low_level_output;    /* кадр → провод */
    low_level_init(netif);
    return ERR_OK;
}

The mode decides who calls netif->input. Without an OS it is ethernet_input: the frame is parsed right away, in the same call. With an OS it is tcpip_input: the frame is put into a mailbox, and the tcpip_thread thread parses it.

106в.2 pbuf: types, chains and references

A pbuf is a header (next, payload, tot_len, len, ref, type) plus data. A long packet can be a chain: each link has its own len, and the first one has tot_len — the length of the whole packet. The type sets where the memory comes from.

The PC model is lwIP itself, built with different PBUF_POOL_BUFSIZE values: how many links a 1514-byte frame takes.

A 1524-byte buffer holds a full frame in one link, but a short frame occupies it entirely. In ST's no-OS TCP and UDP echo examples PBUF_POOL_BUFSIZE is 512, in the other examples 1524.

Other pbuf experiments on the PC:

• A 100-byte PBUF_RAM with room for headers (PBUF_TRANSPORT) took 188 bytes of heap: 8 for the heap block header, 24 for struct pbuf on a 64-bit PC, 56 for headers of 14 + 20 + 20 = 54 bytes aligned to 4, and 100 of data. On the F407 struct pbuf is 16 bytes (that is what the pools in the armclang build give, 106в.4).
• PBUF_REF took only the header from the pool; payload points to someone else's data.
• pbuf_ref and then pbuf_free: ref became 1, the buffer is alive. A second pbuf_free returned it to the pool — 0 in use.
• pbuf_cat joined 1000 and 2000 bytes into a chain of 3 links with tot_len 3000 without copying.

The ST driver in CubeF4 1.28.0 receives without copying. ethernetif.c declares its own pool, RX_POOL: an RxBuff_t block is a struct pbuf_custom plus a buffer for DMA aligned to 32 bytes. For the F407 a block takes 1568 bytes (ETH_RX_BUF_SIZE 1528, rounded up to 1536). There are 10 blocks in LwIP_TCP_Echo_Server and 12 in LwIP_HTTP_Server_Socket_RTOS. HAL takes a block in HAL_ETH_RxAllocateCallback, DMA writes the frame straight into it, and the stack gets a pbuf of type PBUF_REF. pbuf_free_custom returns the block to RX_POOL.

106в.3 Memory: heap, pools and what each number means

lwIP has two kinds of memory. The heap of size MEM_SIZE hands out blocks of any size: PBUF_RAM and copies of data to send. The memp pools hand out blocks of one size: pcbs, segments, PBUF_POOL links. A pool does not fragment, and its size is known at build time. For heaps in general, see section 25.

For Ethernet you take TCP_MSS = 1500 − 40 = 1460: that much data fits into a frame after the IP and TCP headers (section 106).

init.c checks the settings already at build time and stops the build with #error:

• MEMP_NUM_TCP_SEG is not less than TCP_SND_QUEUELEN;
• TCP_SND_BUF is not less than 2 · TCP_MSS;
• TCP_SNDQUEUELOWAT is less than TCP_SND_QUEUELEN (the default threshold is the larger of TCP_SND_QUEUELEN / 2 and 5);
• TCP_WND is not more than PBUF_POOL_SIZE · (PBUF_POOL_BUFSIZE − 54): the window must fit into the receive pool.

106в.4 Two configurations for the F407: RAM and code

Two lwipopts.h files, both built with armclang for the F407 (-Os) together with the driver from 106а and the echo server.

• No OS: NO_SYS=1, raw API only, MEM_SIZE 4 KB, 6 links of 1524, 2 connections, TCP_WND and TCP_SND_BUF of 2 · MSS each.
• With FreeRTOS: the numbers from the lwipopts.h of ST's example LwIP_HTTP_Server_Socket_RTOS — MEM_SIZE 10 KB, 8 links of 1524, 10 connections, TCP_SND_BUF 4 · MSS; netconn and sockets enabled.

Pools are taken from the memp_memory_*_base symbols in fromelf -s. Each pool gets 3 extra bytes for alignment.

Build result (armlink --info sizes, lwIP objects only):

Where the no-OS code goes: TCP — 12214 bytes, DHCP — 4109, IPv4 with fragmentation and ICMP — 3034, ARP and Ethernet — 2020. With an OS, netconn, sockets and tcpip.c are added — 8257 bytes. FreeRTOS RAM is almost all its heap, configTOTAL_HEAP_SIZE 25 KB. Thread stacks are taken from it: TCPIP_THREAD_STACKSIZE 1000 in the ST example is in FreeRTOS words, 4000 bytes.

The F407 port, netif106d.c and the no-OS main add another 10896 bytes of RAM: 4 receive buffers and a transmit buffer of 1536 bytes each with descriptors (106а), two intermediate frames of 1536 in netif106d.c, and struct netif. That is the cost of copying frames.

Part of the pools was never asked for: FRAG_PBUF and REASSDATA are enabled by default (IP_FRAG and IP_REASSEMBLY in opt.h are 1) — 526 bytes for IP fragmentation that the echo server does not need.

106в.5 No OS: the raw API and the main loop

With NO_SYS=1 lwIP has no threads and no locks. Everything is called by the main loop — as in main.c of the LwIP_TCP_Echo_Server example:

    for (;;) {
        ethernetif_input(&nif);                /* кадры из кольца DMA — в стек */
        sys_check_timeouts();                  /* таймеры TCP, ARP, DHCP */
    }

sys_check_timeouts runs the expired timers: TCP retransmissions, ARP, DHCP renewal. It takes the time from sys_now() — in the port that is the SysTick millisecond counter. The lwIP documentation (main_page.h) requires that in this mode lwIP is never called from an interrupt. The Ethernet interrupt only puts the frame into a queue, and the main loop parses it.

There is only one API here — the raw API: the application attaches callbacks to a connection, and the core calls them from the same loop. The echo server's receive callback (echo_raw.c):

static err_t echo_recv(void *arg, struct tcp_pcb *pcb, struct pbuf *p, err_t err)
{
    struct pbuf *q = (struct pbuf *)arg;
    if (p == NULL) {                            /* сосед прислал FIN */
        if (q != NULL)
            pbuf_free(q);
        tcp_arg(pcb, NULL);
        return tcp_close(pcb);
    }
    if (err != ERR_OK) {
        pbuf_free(p);
        return err;
    }
    if (q == NULL)
        q = p;
    else
        pbuf_cat(q, p);                         /* в хвост очереди, без копирования */
    tcp_arg(pcb, echo_push(pcb, q));
    return ERR_OK;
}

What was received is not copied: the pbuf from the stack joins the connection's queue (arg). echo_push does the sending:

static struct pbuf *echo_push(struct tcp_pcb *pcb, struct pbuf *q)
{
    while (q != NULL && q->len <= tcp_sndbuf(pcb)) {
        if (tcp_write(pcb, q->payload, q->len, TCP_WRITE_FLAG_COPY) != ERR_OK)
            break;                              /* кончилась куча или очередь сегментов */
        u16_t n = q->len;
        struct pbuf *next = q->next;
        if (next != NULL)
            pbuf_ref(next);                     /* хвост живёт дальше */
        pbuf_free(q);                           /* освобождается только голова */
        tcp_recved(pcb, n);                     /* окно — только после копирования */
        q = next;
    }
    tcp_output(pcb);
    return q;
}

tcp_write with TCP_WRITE_FLAG_COPY copies the data into a PBUF_RAM from the heap, after which the receive link is free. tcp_recved tells the stack that the data has been taken: only then does the receive window grow again. Whatever did not fit into TCP_SND_BUF waits for the echo_sent callback — the peer has acknowledged receipt and space has been freed.

The PC experiment: the same echo_raw.c and netif106d.c, with the Linux kernel's tap0 interface (192.168.7.1) in place of the DMA ring, and lwIP at 192.168.7.2. A Python client on the Linux side sends 8388608 bytes in chunks of 8192 and checks the echo. Three runs, all data came back without errors. lwIP counters (LWIP_STATS):

• heap — at most 3096 of 4096 bytes: exactly two segments in the send queue, 8 + 24 + 56 + 1460 = 1548 bytes each (TCP_SND_BUF = 2 · MSS);
• PBUF_POOL — at most 2 of 6 links, TCP_SEG — 2 of 8;
• SYS_TIMEOUT — 5 of 5: the default pool is exactly the size for the modules' timers;
• memory failures 0.

106в.6 With an OS: tcpip_thread, netconn and sockets

With NO_SYS=0 the core lives in one thread — tcpip_thread, started by tcpip_init. The driver thread puts frames into a mailbox through tcpip_input, and the application threads talk to the core with messages. The OS port is the file sys_arch.c: semaphores, mutexes, mailboxes and threads. ST builds it on the CMSIS-RTOS wrapper over FreeRTOS.

Sockets work on top of netconn: lwip_recv gets a pbuf through netconn_recv_tcp_pbuf_flags and copies the data into the application's buffer.

The lwIP documentation (main_page.h, section Multithreading) says it plainly: raw API functions are not protected against concurrent access, and with an OS they are called only from tcpip_thread. From other threads you may call only api.h, netbuf.h, netdb.h, netifapi.h, sockets.h, sys.h and memory allocation. There are two legitimate ways to call the raw API from your own thread: tcpip_callback (the function will run in tcpip_thread) or the core lock with LWIP_TCPIP_CORE_LOCKING=1 (the default):

    LOCK_TCPIP_CORE();                     /* raw API и netif — только под замком ядра */
    netif_add(&nif, &ip, &mask, &gw, NULL, ethernetif_init, tcpip_input);
    netif_set_default(&nif);
    netif_set_link_up(&nif);
    netif_set_up(&nif);
    echo_raw_init(7);
    UNLOCK_TCPIP_CORE();

Neither works from an interrupt: the lock is an OS mutex. The rules for interrupts in an RTOS are in section 28.

The experiment: the same program with an OS (pthread threads instead of FreeRTOS) runs three echo servers at once: raw on port 7, netconn on 8, sockets on 9. Three runs of 8388608 bytes on each port, all data came back. Speed on the PC — from 74.7 to 389.7 Mbit/s over all runs; that is the speed of the PC's processor and says nothing about the F407.

Something else does — the number of segments. A frame counter on tap0 showed: lwIP sent the same amount of data, 5746 segments, 1459.9 bytes each on average. But empty segments (ACK only) numbered 2 per run for raw, and for netconn and sockets from 8591 to 8612, one and a half for every data segment. The reason is in tcp_recved. If the window has grown by at least TCP_WND_UPDATE_THRESHOLD = min(TCP_WND / 4, 4 · TCP_MSS) — 730 bytes for us — tcp_recved sends an ACK immediately. In the raw echo the data is already in the queue (tcp_write comes before tcp_recved), and the ACK goes out together with it. With netconn the window is returned by lwip_netconn_do_recv at the moment the thread has taken the data — there is no reply in the queue yet, and the ACK goes out empty.

Counters with an OS: heap — up to 4644 of 10240 bytes (three segments of 1548), PBUF_POOL — up to 3 of 8, NETCONN — 3 of 4 (two listening and one connection), TCPIP_MSG_INPKT — up to 3 of 8.

106в.7 Pitfalls

• The build fails on #error lwip_sanity_check. TCP_SND_BUF = 2 · MSS and TCP_SND_QUEUELEN by the formula from the example (2 · TCP_SND_BUF / TCP_MSS = 4) is below the TCP_SNDQUEUELOWAT threshold, and that is never less than 5. You need TCP_SND_QUEUELEN of 6 or more, or a larger TCP_SND_BUF.
• Your own sys_timeout returns an error. MEMP_NUM_SYS_TIMEOUT by default is exactly the number of timers of the enabled modules; in the no-OS experiment 5 of 5 are in use. ST's lwipopts.h sets 10.
• The connection stalls after the first kilobytes. You forgot tcp_recved: the window does not open, and after TCP_WND bytes (2920 for us) the peer waits.
• The raw API from a foreign thread. With NO_SYS=0, tcp_write from an application thread without LOCK_TCPIP_CORE corrupts the core's lists — not right away and not every time. The LWIP_ASSERT_CORE_LOCKED() check exists in lwIP 2.1.2, but by default it is empty.
• lwIP from an interrupt. Not allowed in any mode: an interrupt only wakes a thread or sets a flag for the main loop. The exception with an OS is tcpip_callbackmsg_trycallback_fromisr: a message prepared in advance with tcpip_callbackmsg_new is sent to tcpip_thread.
• The receive pool sits idle. The ST driver receives into its own RX_POOL, and the core itself takes from PBUF_POOL in only one place — udp.c, a copy of a broadcast datagram for several pcbs. With such a driver, PBUF_POOL_SIZE 8 of 1524 is 12323 bytes that receiving never touches.
• pbuf_free on the head of a chain. It frees the whole chain wherever the reference count drops to zero. A tail you still need gets pbuf_ref first.
• The default NETCONN pool is 4. A listening netconn and a listening socket already take two; in the experiment 3 of 4 are in use.

106в.8 Summary

• lwIP is a netif (low_level_init, low_level_output, low_level_input), the core and three APIs: raw API, netconn, sockets.
• A 1514-byte frame takes 1, 3, 6 or 12 PBUF_POOL links with PBUF_POOL_BUFSIZE 1524, 512, 256, 128. The ST driver receives without copying, in RX_POOL blocks of 1568 bytes.
• TCP sending copies data into the heap: 1548 bytes per full segment on the PC; TCP_SND_BUF = 2 · MSS took 3096 bytes.
• F407, armclang build: lwIP without an OS — 26429 bytes of flash and 14905 bytes of RAM; with FreeRTOS and ST's settings — 35322 and 26597, plus FreeRTOS 7392 and 25918.
• NO_SYS=1: the main loop, ethernetif_input and sys_check_timeouts, raw API only. NO_SYS=0: the core in tcpip_thread, the raw API only from there or under LOCK_TCPIP_CORE.
• For the same 5746 data segments, netconn and sockets sent about 8600 empty ACKs versus 2 for the raw API.

Full source code

netif106d.c — a copying netif — the same three functions as in ethernetif.c of the ST examples.

/* 106в: netif с копированием — те же три функции, что в ethernetif.c примеров ST. */
#include "lwip/opt.h"
#include "lwip/netif.h"
#include "lwip/pbuf.h"
#include "lwip/etharp.h"
#include "lwip/stats.h"
#include "netif/ethernet.h"
#include "netif106d.h"

#define FRAME_MAX 1536
static uint8_t rx_frame[FRAME_MAX];      /* сюда порт кладёт кадр целиком */
static uint8_t tx_frame[FRAME_MAX];      /* сюда собирается цепочка pbuf на отправку */

static void low_level_init(struct netif *netif)
{
    port_hwaddr(netif->hwaddr);
    netif->hwaddr_len = ETH_HWADDR_LEN;
    netif->mtu = 1500;
    netif->flags |= NETIF_FLAG_BROADCAST | NETIF_FLAG_ETHARP;
}

static err_t low_level_output(struct netif *netif, struct pbuf *p)
{
    (void)netif;
    u16_t n = pbuf_copy_partial(p, tx_frame, p->tot_len, 0);   /* цепочка → один буфер */
    port_frame_send(tx_frame, n);
    LINK_STATS_INC(link.xmit);
    return ERR_OK;
}

static struct pbuf *low_level_input(struct netif *netif)
{
    (void)netif;
    int len = port_frame_recv(rx_frame, FRAME_MAX);
    if (len <= 0)
        return NULL;
    struct pbuf *p = pbuf_alloc(PBUF_RAW, (u16_t)len, PBUF_POOL);  /* цепочка из пула */
    if (p == NULL) {
        LINK_STATS_INC(link.memerr);
        LINK_STATS_INC(link.drop);
        return NULL;
    }
    pbuf_take(p, rx_frame, (u16_t)len);
    LINK_STATS_INC(link.recv);
    return p;
}

int ethernetif_input(struct netif *netif)
{
    int n = 0;
    struct pbuf *p;
    while ((p = low_level_input(netif)) != NULL) {
        n++;
        if (netif->input(p, netif) != ERR_OK)   /* ethernet_input или tcpip_input */
            pbuf_free(p);
    }
    return n;
}

err_t ethernetif_init(struct netif *netif)
{
    netif->name[0] = 's';
    netif->name[1] = 't';
    netif->output = etharp_output;           /* IP → ARP → кадр */
    netif->linkoutput = low_level_output;    /* кадр → провод */
    low_level_init(netif);
    return ERR_OK;
}

echo_raw.c — a TCP echo server on the raw API: callbacks only, no OS.

/* 106в: эхо-сервер TCP на raw API — только обратные вызовы, без ОС и без потоков.
   В arg соединения хранится очередь принятого, но ещё не отправленного. */
#include "lwip/tcp.h"
#include "echo_raw.h"

/* отправить из очереди, сколько влезет в буфер передачи; вернуть остаток */
static struct pbuf *echo_push(struct tcp_pcb *pcb, struct pbuf *q)
{
    while (q != NULL && q->len <= tcp_sndbuf(pcb)) {
        if (tcp_write(pcb, q->payload, q->len, TCP_WRITE_FLAG_COPY) != ERR_OK)
            break;                              /* кончилась куча или очередь сегментов */
        u16_t n = q->len;
        struct pbuf *next = q->next;
        if (next != NULL)
            pbuf_ref(next);                     /* хвост живёт дальше */
        pbuf_free(q);                           /* освобождается только голова */
        tcp_recved(pcb, n);                     /* окно — только после копирования */
        q = next;
    }
    tcp_output(pcb);
    return q;
}

static err_t echo_recv(void *arg, struct tcp_pcb *pcb, struct pbuf *p, err_t err)
{
    struct pbuf *q = (struct pbuf *)arg;
    if (p == NULL) {                            /* сосед прислал FIN */
        if (q != NULL)
            pbuf_free(q);
        tcp_arg(pcb, NULL);
        return tcp_close(pcb);
    }
    if (err != ERR_OK) {
        pbuf_free(p);
        return err;
    }
    if (q == NULL)
        q = p;
    else
        pbuf_cat(q, p);                         /* в хвост очереди, без копирования */
    tcp_arg(pcb, echo_push(pcb, q));
    return ERR_OK;
}

static err_t echo_sent(void *arg, struct tcp_pcb *pcb, u16_t len)
{
    (void)len;                                  /* сосед подтвердил — место освободилось */
    if (arg != NULL)
        tcp_arg(pcb, echo_push(pcb, (struct pbuf *)arg));
    return ERR_OK;
}

static err_t echo_poll(void *arg, struct tcp_pcb *pcb)
{
    return echo_sent(arg, pcb, 0);              /* раз в секунду: вдруг место есть */
}

static void echo_err(void *arg, err_t err)
{
    (void)err;                                  /* pcb уже освобождён стеком */
    if (arg != NULL)
        pbuf_free((struct pbuf *)arg);
}

static err_t echo_accept(void *arg, struct tcp_pcb *pcb, err_t err)
{
    (void)arg;
    if (err != ERR_OK || pcb == NULL)
        return ERR_VAL;
    tcp_arg(pcb, NULL);
    tcp_recv(pcb, echo_recv);
    tcp_sent(pcb, echo_sent);
    tcp_poll(pcb, echo_poll, 2);                /* 2 × 500 мс */
    tcp_err(pcb, echo_err);
    return ERR_OK;
}

void echo_raw_init(u16_t port)
{
    struct tcp_pcb *pcb = tcp_new();
    tcp_bind(pcb, IP_ADDR_ANY, port);
    pcb = tcp_listen(pcb);                      /* слушающий pcb — из другого пула, поменьше */
    tcp_accept(pcb, echo_accept);
}

main_min.c — F407 without an OS (NO_SYS=1): the main loop.

/* 106в: F407 без ОС (NO_SYS=1) — главный цикл, как в main.c примера ST LwIP_TCP_Echo_Server. */
#include "lwip/init.h"
#include "lwip/netif.h"
#include "lwip/timeouts.h"
#include "netif/ethernet.h"
#include "netif106d.h"
#include "echo_raw.h"

void port_start(void);
static struct netif nif;

int main(void)
{
    ip4_addr_t ip, mask, gw;
    port_start();
    lwip_init();
    IP4_ADDR(&ip, 192, 168, 7, 2);
    IP4_ADDR(&mask, 255, 255, 255, 0);
    IP4_ADDR(&gw, 192, 168, 7, 1);
    netif_add(&nif, &ip, &mask, &gw, NULL, ethernetif_init, ethernet_input);
    netif_set_default(&nif);
    netif_set_link_up(&nif);
    netif_set_up(&nif);
    echo_raw_init(7);
    for (;;) {
        ethernetif_input(&nif);                /* кадры из кольца DMA — в стек */
        sys_check_timeouts();                  /* таймеры TCP, ARP, DHCP */
    }
}

main_rtos.c — F407 with FreeRTOS (NO_SYS=0): tcpip_thread and three echo servers — raw, netconn and sockets.

/* 106в: F407 с FreeRTOS (NO_SYS=0) — tcpip_thread, три эхо-сервера: raw :7, netconn :8, сокеты :9. */
#include "lwip/tcpip.h"
#include "lwip/netif.h"
#include "lwip/api.h"
#include "lwip/sockets.h"
#include "lwip/sys.h"
#include "netif106d.h"
#include "echo_raw.h"

void port_start(void);
static struct netif nif;

static void rx_thread(void *arg)          /* у ST — поток EthIf, будится семафором из прерывания */
{
    (void)arg;
    for (;;) {
        if (ethernetif_input(&nif) == 0)
            sys_msleep(1);
    }
}

static void netconn_echo(void *arg)
{
    (void)arg;
    struct netconn *l = netconn_new(NETCONN_TCP), *c;
    netconn_bind(l, IP_ADDR_ANY, 8);
    netconn_listen(l);
    while (netconn_accept(l, &c) == ERR_OK) {
        struct netbuf *nb;
        while (netconn_recv(c, &nb) == ERR_OK) {
            void *data;
            u16_t len;
            do {
                netbuf_data(nb, &data, &len);
                netconn_write(c, data, len, NETCONN_COPY);
            } while (netbuf_next(nb) >= 0);
            netbuf_delete(nb);
        }
        netconn_close(c);
        netconn_delete(c);
    }
}

static void socket_echo(void *arg)
{
    (void)arg;
    static char buf[1460];
    struct sockaddr_in a = { 0 };
    int l = lwip_socket(AF_INET, SOCK_STREAM, 0);
    a.sin_family = AF_INET;
    a.sin_port = lwip_htons(9);
    lwip_bind(l, (struct sockaddr *)&a, sizeof a);
    lwip_listen(l, 1);
    for (;;) {
        int c = lwip_accept(l, NULL, NULL), n;
        while ((n = lwip_recv(c, buf, sizeof buf, 0)) > 0)
            lwip_send(c, buf, (size_t)n, 0);
        lwip_close(c);
    }
}

static void start_thread(void *arg)
{
    ip4_addr_t ip, mask, gw;
    (void)arg;
    tcpip_init(NULL, NULL);
    IP4_ADDR(&ip, 192, 168, 7, 2);
    IP4_ADDR(&mask, 255, 255, 255, 0);
    IP4_ADDR(&gw, 192, 168, 7, 1);
    LOCK_TCPIP_CORE();
    netif_add(&nif, &ip, &mask, &gw, NULL, ethernetif_init, tcpip_input);
    netif_set_default(&nif);
    netif_set_link_up(&nif);
    netif_set_up(&nif);
    echo_raw_init(7);
    UNLOCK_TCPIP_CORE();
    sys_thread_new("rx", rx_thread, NULL, DEFAULT_THREAD_STACKSIZE, osPriorityRealtime);
    sys_thread_new("netconn", netconn_echo, NULL, DEFAULT_THREAD_STACKSIZE, osPriorityNormal);
    sys_thread_new("socket", socket_echo, NULL, DEFAULT_THREAD_STACKSIZE, osPriorityNormal);
    for (;;)
        sys_msleep(1000);
}

int main(void)
{
    port_start();
    sys_thread_new("start", start_thread, NULL, DEFAULT_THREAD_STACKSIZE, osPriorityNormal);
    osKernelStart();
    for (;;) {
    }
}

Next: section 106г "GSM modem SIM800L over UART: AT commands, SMS, calls and TCP over GPRS — non-blocking response parsing" — when there is no wire and Wi-Fi does not reach: the cellular network through a modem driven by AT commands over a UART.