156. Networking on the ESP32: esp_netif and lwIP, DHCP, SNTP, syslog, HTTP REST, MQTT

The network is the reason people pick the ESP32: Wi-Fi on the chip, the lwIP TCP/IP stack inside ESP-IDF, and ready-made clients and servers for every case. In part X the network was taken apart from the bottom up on the STM32 and on a computer (sections 105–108), and MQTT in sections 87 and 91. Here it is the same thing on the ESP32, but checked with a live exchange. The program in the emulator gets an address over DHCP, takes the time from a real chrony server, sends its log to a real rsyslog, answers curl requests and publishes to a Mosquitto broker. All the servers are neighbouring containers on one Docker network (_verify/esp156/).

The emulator does not model the radio, so the network in it is Ethernet: the OpenCores controller on the chip bus. For the program there is almost no difference: above the driver it is the same esp_netif, the same lwIP and the same sockets as with Wi-Fi. What applies only to Wi-Fi is collected in 156.7.

Terms used in the documentation

Term	Meaning
esp_netif	the ESP-IDF layer between the network driver and the TCP/IP stack
lwIP (lightweight IP)	a TCP/IP stack for microcontrollers
event loop, IP_EVENT	the event loop, network-layer events
station (STA), access point (AP)	Wi-Fi modes: a client of a network and your own access point
SNTP (Simple Network Time Protocol)	a simplified time protocol
syslog, facility, severity, PRI	a log over the network, source, importance, priority
REST, resource, method	a style of HTTP interface: a resource and methods
MQTT broker, topic, QoS	a message broker, a topic, a delivery guarantee level
PUBACK	the publish acknowledgement at QoS 1
keepalive	the time within which the client must show a sign of life
MSS, TCP window	the maximum segment size, the TCP receive window

156.1 The bench: a board and three servers

Three containers on one Docker network 10.156.0.0/24:

The emulator has its own small network 10.0.2.0/24: it hands the board an address over DHCP and forwards its packets outside on behalf of the container, like a home router. So the board sees the servers at their addresses, but from outside the board itself can be reached only through a forwarded port — that is how curl gets to the HTTP server.

156.2 esp_netif, events and the address

In ESP-IDF the network comes up the same way for Ethernet and Wi-Fi: initialize esp_netif, the event loop, the driver, then wait for the "got an address" event. In the ESP-IDF examples this is hidden in example_connect():

    ESP_ERROR_CHECK(nvs_flash_init());
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    ESP_ERROR_CHECK(example_connect());                   /* Ethernet open_eth, DHCP */

адрес 10.0.2.15, маска 255.255.255.0, шлюз 10.0.2.2

Everything network-related in ESP-IDF reports itself with events: IP_EVENT_ETH_GOT_IP or IP_EVENT_STA_GOT_IP for Wi-Fi, WIFI_EVENT_STA_DISCONNECTED, MQTT_EVENT_CONNECTED. The program does not poll the stack; it subscribes to events and waits. NVS is needed by the Wi-Fi and Bluetooth driver: it holds the radio calibration and the network settings. For Ethernet alone it is not required, but example_connect can also do Wi-Fi, so the example always initializes it.

The memory cost of the network shows in the size:

│ Flash Code            │       423254 │    12.66 │        2919050 │       3342304 │
│ IRAM                  │        59043 │    45.05 │          72029 │        131072 │

The image with the network, SNTP, the HTTP server and the MQTT client is 606196 bytes versus 179144 for the empty program of section 151; instruction RAM is 45 % full.

156.3 Time: SNTP

The ESP32 has no battery-backed clock: after power-on time() counts seconds from zero, and without a network the board does not know the date:

до SNTP: time() = 1, sizeof(time_t) = 8
SNTP 10.156.0.10: ESP_OK, год 2026, time() = 1789384076

    esp_sntp_config_t sc = ESP_NETIF_SNTP_DEFAULT_CONFIG(CONFIG_LAB_SERVER_IP);
    esp_netif_sntp_init(&sc);
    esp_err_t e = esp_netif_sntp_sync_wait(pdMS_TO_TICKS(20000));

The SNTP client sends the server a UDP packet to port 123, gets the time and sets the system clock; from then on time(), gettimeofday() and localtime_r() work as on a computer, and the client itself corrects the clock once an hour. time_t in ESP-IDF 5 is 64 bits, so the year 2038 does not threaten the program. The time zone is set with the TZ variable via setenv and tzset; logs and protocols are better kept in UTC.

156.4 A log over the network: syslog

A device without a screen writes its log to a server. The syslog format (RFC 3164) is one line per UDP datagram to port 514:

    int pri = facility * 8 + severity;                     /* RFC 3164: <PRI> = facility·8 + severity */
    int n = snprintf(line, sizeof line, "<%d>%s esp32 %s: %s", pri, stamp, tag, msg);

syslog → 10.156.0.10:514, PRI 134, 42 байт: <134>Sep 14 11:07:56 esp32 net156: boot ok
syslog → 10.156.0.10:514, PRI 131, 49 байт: <131>Sep 14 11:07:56 esp32 net156: sensor timeout

The local0 facility is number 16, the "informational" severity is 6, "error" is 3: 16 × 8 + 6 = 134 and 16 × 8 + 3 = 131. The rsyslog server parsed both lines:

pri=134 local0.info host=esp32 tag=net156: msg= boot ok
pri=131 local0.err host=esp32 tag=net156: msg= sensor timeout

UDP promises nothing: a lost line is lost forever. For an important log there is syslog over TCP and TLS (RFC 5425), but for "just in case" records UDP is enough — the device does not hang if the server is down.

156.5 HTTP REST on the board

esp_http_server is the HTTP server from ESP-IDF: one handler per "URI and method" pair, the response in one call:

static esp_err_t led_put(httpd_req_t *req)
{
    char buf[8] = { 0 };
    int r = httpd_req_recv(req, buf, sizeof buf - 1);
    hits++;
    if (r <= 0 || (buf[0] != '0' && buf[0] != '1'))
        return httpd_resp_send_err(req, HTTPD_400_BAD_REQUEST, "body must be 0 or 1");
    led = buf[0] - '0';
    return led_reply(req);
}

Three curl requests — read, write, write garbage:

{"led":0,"hits":1}
{"led":1,"hits":2}
HTTP 400

HTTP: порт 80, до 8 обработчиков, до 7 сокетов; MSS 1440, окно 5760

By default the server holds up to 8 URIs and up to 7 simultaneous connections: the eighth client waits until a socket frees up. For a device on a local network this is enough; for a page that pulls in images and scripts, the number of sockets is raised in HTTPD_DEFAULT_CONFIG — but together with CONFIG_LWIP_MAX_SOCKETS in the lwIP settings. Its default there is 10, the server keeps three sockets for itself, and httpd_start will not accept max_open_sockets above 7 — it returns ESP_ERR_INVALID_ARG. A browser opens several connections at once by itself, so the "eighth client" is often just the second browser; lru_purge_enable = true closes the connection that has gone unused the longest when there is no room. MSS 1440 and a window of 5760 bytes are the lwIP defaults: four segments in the window, modest, but it eats little RAM. HTTP parsed as text is covered in section 142.

156.6 MQTT

The esp-mqtt client from ESP-IDF keeps the connection to the broker by itself and reports everything with events:

    esp_mqtt_client_config_t mc = { .broker.address.uri = CONFIG_LAB_MQTT_URI, .credentials.client_id = "esp32-156",
                                    .session.keepalive = 30 };
    esp_mqtt_client_handle_t cli = esp_mqtt_client_init(&mc);
    esp_mqtt_client_register_event(cli, ESP_EVENT_ANY_ID, on_mqtt, NULL);
    esp_mqtt_client_start(cli);

On the "connected" event the program subscribes to its command topic; on "subscription acknowledged" it publishes its state and a command to itself, both with QoS 1:

MQTT: соединение с брокером, session_present 0
MQTT: подписка dev/esp32/cmd подтверждена, msg_id 37586
MQTT: PUBACK на msg_id 29042
MQTT: пришло dev/esp32/cmd = blink (QoS 1)
MQTT: PUBACK на msg_id 55183
MQTT: итог 3 (3 — всё прошло), keepalive 30 с

The broker sees the same from its side:

New client connected from 10.156.0.2:36664 as esp32-156 (p4, c1, k30).
Sending CONNACK to esp32-156 (0, 0)
Received PUBLISH from esp32-156 (d0, q1, r0, m29042, 'dev/esp32/status', ... (9 bytes))
Sending PUBACK to esp32-156 (m29042, rc0)

The board came from the container's address 10.156.0.2, not from its own 10.0.2.15 — this is the emulator's address translation from 156.1. p4 is protocol version 3.1.1, c1 is a clean session, k30 is a keepalive of 30 s. A subscriber on the broker received both publications:

dev/esp32/status {"led":1}
dev/esp32/cmd blink

The bytes of MQTT packets, keepalive and QoS levels are covered in section 87. By default Mosquitto 2 lets no one in from outside: without listener 1883 and allow_anonymous true in the configuration the board gets a refused connection — in a product you set passwords and TLS instead. The cost of TLS is covered in section 37а: a client on Mbed TLS takes tens of kilobytes of flash and heap. The certificate is checked against the time, so SNTP (156.3) is synchronized before the first secure connection.

156.7 Wi-Fi: what changes

Nothing above the driver. Instead of example_connect() with Ethernet, the program brings up Wi-Fi itself: esp_netif_create_default_wifi_sta(), esp_wifi_init(), station mode, network name and password, esp_wifi_start() and esp_wifi_connect(), and then waits for the same "got an address" event. Three differences matter in practice:

• A disconnect is normal. The WIFI_EVENT_STA_DISCONNECTED event arrives every time you leave the router's range, and the program must reconnect by itself, with a pause between attempts.
• The radio shares its time. Wi-Fi and Bluetooth work through one radio in turns; ADC2 is not available while Wi-Fi is on (section 154).
• Saving power. In station mode ESP-IDF by default puts the radio to sleep between the router's beacons — this is the modem sleep of section 157, where the current figures are.

An access point (AP) is needed for the first setup: the device brings up its own network, the phone connects and enters the home network name and password in a web form on the HTTP server from 156.5. Better not to send the password in clear text here: ESP-IDF has the wifi_provisioning component with encryption for the first setup.

156.8 Pitfalls

156.9 Summary

• esp_netif + lwIP are the same for Ethernet and Wi-Fi; everything network-related is events.
• DHCP in the emulator: 10.0.2.15/24, gateway 10.0.2.2; outside through NAT.
• SNTP: time() from 1 to the real date, time_t is 64 bits; there is no battery-backed clock.
• syslog: <PRI> = facility × 8 + severity; local0.info is 134, local0.err is 131; UDP 514.
• HTTP server: a handler per URI and method; 8 URIs and 7 sockets by default; MSS 1440, window 5760.
• MQTT: events CONNECTED, SUBSCRIBED, PUBLISHED, DATA; QoS 1 means PUBACK; keepalive 30 s.
• The image with the network is 606196 bytes, IRAM 45 %.

Full source code

main.c — address over DHCP, time over SNTP, log over syslog, HTTP REST and an MQTT client.

/* Раздел 156: сеть ESP32 в QEMU — адрес по DHCP, время по SNTP, журнал по syslog (UDP 514),
 * HTTP-сервер с REST и клиент MQTT. Серверы времени и журнала — в соседнем контейнере
 * CONFIG_LAB_SERVER_IP, брокер MQTT — CONFIG_LAB_MQTT_URI. */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <time.h>
#include "esp_event.h"
#include "esp_http_server.h"
#include "esp_log.h"
#include "esp_netif.h"
#include "esp_netif_sntp.h"
#include "lwip/sockets.h"
#include "mqtt_client.h"
#include "nvs_flash.h"
#include "protocol_examples_common.h"
#include "sdkconfig.h"

static int led;
static int hits;
static volatile int mqtt_state;                         /* 1 — соединились, 2 — PUBACK пришёл, 3 — своё сообщение вернулось */

static void syslog_send(int facility, int severity, const char *tag, const char *msg)
{
    char line[256];
    time_t now = time(NULL);
    struct tm tm;
    gmtime_r(&now, &tm);
    char stamp[20];
    strftime(stamp, sizeof stamp, "%b %e %H:%M:%S", &tm);
    int pri = facility * 8 + severity;                     /* RFC 3164: <PRI> = facility·8 + severity */
    int n = snprintf(line, sizeof line, "<%d>%s esp32 %s: %s", pri, stamp, tag, msg);
    int s = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
    struct sockaddr_in to = { .sin_family = AF_INET, .sin_port = htons(514) };
    inet_aton(CONFIG_LAB_SERVER_IP, &to.sin_addr);
    int sent = sendto(s, line, n, 0, (struct sockaddr *)&to, sizeof to);
    close(s);
    printf("syslog → %s:514, PRI %d, %d байт: %s\n", CONFIG_LAB_SERVER_IP, pri, sent, line);
}

static esp_err_t led_reply(httpd_req_t *req)
{
    char body[64];
    int n = snprintf(body, sizeof body, "{\"led\":%d,\"hits\":%d}", led, hits);
    httpd_resp_set_type(req, "application/json");
    return httpd_resp_send(req, body, n);
}

static esp_err_t led_get(httpd_req_t *req)
{
    hits++;
    return led_reply(req);
}

static esp_err_t led_put(httpd_req_t *req)
{
    char buf[8] = { 0 };
    int r = httpd_req_recv(req, buf, sizeof buf - 1);
    hits++;
    if (r <= 0 || (buf[0] != '0' && buf[0] != '1'))
        return httpd_resp_send_err(req, HTTPD_400_BAD_REQUEST, "body must be 0 or 1");
    led = buf[0] - '0';
    return led_reply(req);
}

static void on_mqtt(void *arg, esp_event_base_t base, int32_t id, void *data)
{
    esp_mqtt_event_handle_t ev = data;
    (void)arg;
    (void)base;
    switch ((esp_mqtt_event_id_t)id) {
    case MQTT_EVENT_CONNECTED:
        printf("MQTT: соединение с брокером, session_present %d\n", ev->session_present);
        esp_mqtt_client_subscribe(ev->client, "dev/esp32/cmd", 1);
        mqtt_state = 1;
        break;
    case MQTT_EVENT_SUBSCRIBED:
        printf("MQTT: подписка dev/esp32/cmd подтверждена, msg_id %d\n", ev->msg_id);
        esp_mqtt_client_publish(ev->client, "dev/esp32/status", "{\"led\":1}", 0, 1, 0);     /* QoS 1 — ждём PUBACK */
        esp_mqtt_client_publish(ev->client, "dev/esp32/cmd", "blink", 0, 1, 0);             /* своя же тема: вернётся через брокер */
        break;
    case MQTT_EVENT_PUBLISHED:
        printf("MQTT: PUBACK на msg_id %d\n", ev->msg_id);
        if (mqtt_state < 2) mqtt_state = 2;
        break;
    case MQTT_EVENT_DATA:
        printf("MQTT: пришло %.*s = %.*s (QoS %d)\n", ev->topic_len, ev->topic, ev->data_len, ev->data, ev->qos);
        mqtt_state = 3;
        break;
    case MQTT_EVENT_ERROR:
        printf("MQTT: ошибка, тип %d\n", ev->error_handle ? ev->error_handle->error_type : -1);
        break;
    default:
        break;
    }
}

void app_main(void)
{
    ESP_ERROR_CHECK(nvs_flash_init());
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    ESP_ERROR_CHECK(example_connect());                   /* Ethernet open_eth, DHCP */

    esp_netif_ip_info_t ip;
    esp_netif_get_ip_info(get_example_netif(), &ip);
    printf("адрес " IPSTR ", маска " IPSTR ", шлюз " IPSTR "\n", IP2STR(&ip.ip), IP2STR(&ip.netmask), IP2STR(&ip.gw));
    printf("до SNTP: time() = %lld, sizeof(time_t) = %u\n", (long long)time(NULL), (unsigned)sizeof(time_t));

    esp_sntp_config_t sc = ESP_NETIF_SNTP_DEFAULT_CONFIG(CONFIG_LAB_SERVER_IP);
    esp_netif_sntp_init(&sc);
    esp_err_t e = esp_netif_sntp_sync_wait(pdMS_TO_TICKS(20000));
    time_t now = time(NULL);
    struct tm tm;
    gmtime_r(&now, &tm);
    printf("SNTP %s: %s, год %d, time() = %lld\n", CONFIG_LAB_SERVER_IP, esp_err_to_name(e), tm.tm_year + 1900, (long long)now);

    syslog_send(16, 6, "net156", "boot ok");              /* local0.info */
    syslog_send(16, 3, "net156", "sensor timeout");       /* local0.err */

    esp_mqtt_client_config_t mc = { .broker.address.uri = CONFIG_LAB_MQTT_URI, .credentials.client_id = "esp32-156",
                                    .session.keepalive = 30 };
    esp_mqtt_client_handle_t cli = esp_mqtt_client_init(&mc);
    esp_mqtt_client_register_event(cli, ESP_EVENT_ANY_ID, on_mqtt, NULL);
    esp_mqtt_client_start(cli);
    for (int i = 0; i < 100 && mqtt_state < 3; i++)
        vTaskDelay(pdMS_TO_TICKS(100));
    printf("MQTT: итог %d (3 — всё прошло), keepalive %d с\n", mqtt_state, mc.session.keepalive);

    httpd_config_t hc = HTTPD_DEFAULT_CONFIG();
    httpd_handle_t srv = NULL;
    ESP_ERROR_CHECK(httpd_start(&srv, &hc));
    httpd_uri_t g = { .uri = "/api/led", .method = HTTP_GET, .handler = led_get };
    httpd_uri_t p = { .uri = "/api/led", .method = HTTP_PUT, .handler = led_put };
    httpd_register_uri_handler(srv, &g);
    httpd_register_uri_handler(srv, &p);
    printf("HTTP: порт %d, до %d обработчиков, до %d сокетов; MSS %d, окно %d\n", hc.server_port, hc.max_uri_handlers, hc.max_open_sockets,
           CONFIG_LWIP_TCP_MSS, CONFIG_LWIP_TCP_WND_DEFAULT);
    printf("ready\n");
}

Next: section 157 — sleep, RTC memory and the ULP coprocessor.