88. Distributed systems and APIs

A device's data rarely stops at the broker. From there it is taken in by a system of many parts: an ingestion service, a database, rules, notifications, reports. The parts live on different machines and talk to each other through APIs — programming interfaces over the network. The device becomes a participant in such a system too: it publishes to a broker (section 87) or calls the server API itself.

This section is about how such systems are built and what follows from that for the firmware: how to build a request so that it is understood, how to retry without doing harm, and what a token actually protects. The client side is written in portable C and checked by experiment: two nginx servers behind an nginx load balancer in containers.

Terms used in the documentation

Term	Meaning
file server, client–server, multi-tier	the file-server, client–server and layered architectures
vertical / horizontal scaling	a bigger machine / more machines
replication, sharding	read copies of a database, splitting data by key
monolith, microservices	one application, a set of small services
API, REST, endpoint	a programming interface, REST, an address you call
idempotent	a repeat does not change the result
status code	the response code
publish–subscribe, request–reply	delivery through a broker, a call with an answer
round-robin	in turn
exponential backoff, jitter	a growing delay, a random spread
thundering herd	everyone at once after a failure
JWT, bearer token	a web token, a token in the Authorization header
OpenAPI (Swagger)	a machine-readable API description

88.1 Information system architectures

Devices are almost always clients of a multi-tier system: a load balancer stands in front of the services, and the database behind them.

88.2 Scaling

• Vertical — a more powerful machine. Simple, but growth has a ceiling, and there is still only one machine.
• Horizontal — more identical machines behind a load balancer. It grows almost without limit and survives the failure of one of the machines.

A database is scaled in two ways. Replication — read copies: writes go to the primary, reads are spread across the copies. Splitting by key (sharding) — the data is divided into parts, for example by device number, and each part lives on its own machine.

Devices become more numerous, not more powerful, — so the systems for them grow horizontally.

88.3 Monolith and microservices

For a device system the sensible path is to start with a monolith and split out services where the load or the team requires it: message ingestion, storage, rules and alarms, reports.

88.4 How systems agree

Ways to join systems, from simple to proper: by hand (copying), through shared files, through a shared database, through a data bus, and through an API over HTTP — REST or SOAP. For devices today it is almost always REST or messaging.

REST works with resources at addresses and with four main methods:

The property "a repeat does not change the result" is called idempotency, and for a device it is the main one. The link drops, the response is lost, the device repeats the request. A repeated POST creates a duplicate, a repeated PUT is harmless. So device data is written idempotently: PUT by the device's own number or POST with a unique request key by which the server recognizes a repeat.

The status code decides what to do next: 2xx — done; 4xx — the request is wrong, and repeating it in the same form is pointless; 5xx and no response — repeat, but with a growing delay (88.6). There are exceptions among the 4xx codes, and for a device they matter more than the rule: 429 "too many requests" and 408 "request timeout" are repeated with the same growing delay, and if the server sent a Retry-After header, — no earlier than the time it gives; 401 after the token expires — get a new token and repeat (88.7). A device that drops a request on 429 loses data.

Who waits for whom. A chain of synchronous calls "service → service → service" adds up the delays, and the failure of one link breaks the whole chain. An asynchronous request–reply answers "accepted" and sends the result later. Publish–subscribe is MQTT (sections 37.2 and 87): the device publishes, and it does not know and should not know who is subscribed or how many receivers there are.

Why devices do not poll the server every few seconds: polling once every 10 s is 8640 requests per day for one device, 8.64 million for a thousand devices, and almost all of them for nothing. A subscription or sending on an event spends the network and the battery only when there is something to say.

88.5 The device as a REST client

A request from the firmware is text (section 78.6), and two things in it break most often: the JSON body and its length.

JSON is built so that a quote in a string does not break the document:

/* Строка в JSON: кавычка, обратная косая и управляющие символы — через «\» */
size_t json_escape(char *out, size_t cap, const char *s)
{
    static const char hex[] = "0123456789abcdef";
    size_t k = 0;
    for (; *s; s++) {
        unsigned char c = (unsigned char)*s;
        char tmp[7];
        size_t n = 0;
        if (c == '"' || c == '\\') {
            tmp[n++] = '\\';
            tmp[n++] = (char)c;
        } else if (c < 0x20) {
            tmp[n++] = '\\';
            tmp[n++] = 'u';
            tmp[n++] = '0';
            tmp[n++] = '0';
            tmp[n++] = hex[c >> 4];
            tmp[n++] = hex[c & 15];
        } else {
            tmp[n++] = (char)c;
        }
        size_t need = n;
        if (c >= 0xC0u)                     /* первый байт буквы UTF-8: она должна влезть целиком */
            need = c >= 0xF0u ? 4u : c >= 0xE0u ? 3u : 2u;
        if (k + need + 1 > cap)
            break;                          /* не влезло — обрезаем, букву UTF-8 не разрезая */
        memcpy(out + k, tmp, n);
        k += n;
    }
    out[k] = '\0';
    return k;
}

/* Телеметрия: температура в сотых долях градуса — без float на приборе */
int json_telemetry(char *out, size_t cap, const char *device, int32_t temp_centi, uint32_t uptime_s)
{
    char name[64];
    json_escape(name, sizeof name, device);
    int32_t a = temp_centi < 0 ? -temp_centi : temp_centi;
    return snprintf(out, cap, "{\"device\":\"%s\",\"temp\":%s%ld.%02ld,\"uptime\":%lu}",
                    name, temp_centi < 0 ? "-" : "", (long)(a / 100), (long)(a % 100),
                    (unsigned long)uptime_s);
}

For a device named stm32 "цех-1", with a temperature of −1234 hundredths of a degree and an uptime of 86400 s, the result was {"device":"stm32 \"цех-1\"","temp":-12.34,"uptime":86400} — Python parsed it without errors. The fraction is printed from integers, without floating-point arithmetic (section 69).

The body length is in bytes, not in characters: a Cyrillic letter in UTF-8 is two bytes. strlen counts exactly bytes:

int http_request(char *out, size_t cap, const char *method, const char *host,
                 const char *path, const char *body)
{
    if (body)
        return snprintf(out, cap,
                        "%s %s HTTP/1.1\r\nHost: %s\r\n"
                        "Content-Type: application/json\r\nContent-Length: %zu\r\n"
                        "Connection: close\r\n\r\n%s",
                        method, path, host, strlen(body), body);
    return snprintf(out, cap, "%s %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n",
                    method, path, host);
}

Experiment. On the container network there are two nginx servers, backend-1 and backend-2, and in front of them an nginx load balancer named lb. The "device" is the same code built for a computer:

The load balancer takes a few lines:

upstream backends {
    zone backends 64k;
    server b1:80;
    server b2:80;
}
server {
    listen 80;
    location / {
        proxy_pass http://backends;
    }
}

⚠️ The zone line here is not for decoration. Each nginx worker process keeps its own round-robin queue, and connections from devices are short and land in different processes. In the first try, without a shared zone, all four requests went to the first server, and the load balancer looked broken. zone makes the group state shared by all processes.

The stack frame of json_telemetry on Cortex-M3 is 136 bytes, 64 of them the name buffer; the other functions get by with 48 bytes or less. For an RTOS task that is not much, but the stack of the communication task is sized with a margin (section 26).

88.6 Retry, but not all at once

The server went down and came back up. Thousands of devices that were waiting will repeat the request — and if all of them do it after exactly one second, they will bring the server down again. This is the "thundering herd". The cure is a growing delay with a random spread (jitter):

/* Задержка перед попыткой attempt (0, 1, 2…): случайная в пределах
 * 1 с · 2^attempt, но не больше 5 минут. */
uint32_t backoff_ms(unsigned attempt, uint32_t *seed)
{
    uint32_t cap = attempt >= 9 ? 300000u : 1000u << attempt;
    if (cap > 300000u)
        cap = 300000u;
    *seed = *seed * 1103515245u + 12345u;
    return (*seed >> 8) % (cap + 1);
}

The delay limit grows — 1, 2, 4, 8… 256 s — and from the tenth attempt it hits 5 minutes; within the limit the delay is random. Calculated for 10000 devices after a failure: without the spread all 10000 arrive in the same second, and with the spread, on the fifth attempt, the requests stretch over 16 s: 62 per 0.1 second on average, about 90 in the densest 0.1 s. Each device takes its own seed for the generator — for example, from the unique ID of the chip (section 79.7), otherwise the "random" delays will be the same for everyone.

88.7 Tokens: what they protect and what they do not

The server has to know who has come to it. The device presents a token in the Authorization: Bearer … header. A common format is JWT: three parts separated by dots — header, payload and signature, each encoded as base64url:

/* base64url без «=» в конце — так кодируют части токена JWT */
size_t b64url(const uint8_t *in, size_t n, char *out)
{
    static const char tab[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
    size_t k = 0;
    for (size_t i = 0; i < n; i += 3) {
        uint32_t v = (uint32_t)in[i] << 16;
        if (i + 1 < n)
            v |= (uint32_t)in[i + 1] << 8;
        if (i + 2 < n)
            v |= in[i + 2];
        out[k++] = tab[v >> 18 & 63];
        out[k++] = tab[v >> 12 & 63];
        if (i + 1 < n)
            out[k++] = tab[v >> 6 & 63];
        if (i + 2 < n)
            out[k++] = tab[v & 63];
    }
    out[k] = '\0';
    return k;
}

The header {"alg":"HS256","typ":"JWT"} turns into eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 — the start of every such token.

The main thing to remember: base64 is an encoding, not a cipher. Anyone who has the token can read its contents. The token is protected against forgery by the signature, and against eavesdropping only by encrypting the channel (TLS, section 37.4). The payload holds an expiry time: the server will not accept an expired token, and the device has to get a new one in time. A token stored in plain text in the firmware is visible in its image (section 77.9).

Before writing the client side, you try the API by hand: a description in the OpenAPI (Swagger) format shows all the methods, and tools for sending requests let you see the server's responses before the first line of firmware.

88.8 Examples of distributed technologies

• BitTorrent — peer-to-peer distribution: the file is cut into pieces, each one is checked by its hash and fetched from different peers, the rarest first. Here a file's popularity loads no one, unlike a single server. Checking every piece by its hash is the same idea as checking the firmware image during an update (section 48). The "slow start" mechanism often mentioned alongside is congestion control in TCP itself, not a BitTorrent algorithm.
• VPN — an encrypted tunnel over the internet: remote access to a network or a link between sites. For a fleet of devices it is a way to keep them in one closed network; WireGuard, OpenVPN and IPsec are common.
• Tor — "onion" routing: a request goes through several nodes, and each one removes its own layer of encryption; no single node sees both the sender and the contents.
• Blockchain — a chain of blocks, each of which stores the hash of the previous one: you cannot change an old record without recalculating all the following ones, and every participant has a copy of the chain. Devices rarely need it, but the idea of a log linked by hashes is useful where you need to prove that the records were not edited.

88.9 Pitfalls

Duplicate measurements in the database. The device repeated a POST after a dropped link; you need a PUT by the device's own number or a request key.

The server cannot come back up after a failure. All devices repeat their requests at the same time — there is no random spread, or everyone has the same seed.

The server answers 400 to the telemetry of one device. The device name contains a quote, and the JSON was built without escaping.

The request body is cut off on Cyrillic text. Content-Length was counted in characters, not in bytes.

The load balancer sends everything to one server. There is no shared zone — each nginx process has its own queue.

The token was stolen from intercepted traffic. base64 does not encrypt; you need TLS.

The device stopped sending data after a day. The token expired, and the firmware cannot renew it.

A report takes minutes to build. A chain of synchronous service calls, where each one waits for the next.

The server bill went up, but there is no more data. Thousands of devices poll the server every few seconds instead of subscribing.

88.10 Summary

• Devices are clients of multi-tier systems; such systems grow horizontally, and databases grow with read copies and splitting by key.
• A monolith is simpler at the start, microservices pay off as you grow; services are split out when needed.
• REST: GET, PUT, DELETE are safe to retry, POST is not; device retries are made idempotent. 4xx are not repeated, except 408 and 429 (and 401 — with a new token); 5xx and timeouts are repeated with a delay.
• Polling once every 10 s is 8640 requests per day per device; a subscription and sending on an event are cheaper.
• JSON is built with escaping, Content-Length is in bytes; the experiment with two servers behind nginx confirmed both.
• Retries use a growing delay with a spread: 10000 devices after a failure give 62 requests per 0.1 s on average, about 90 at the peak, instead of 10000 in one second.
• JWT is protected against forgery by the signature, against eavesdropping only by TLS; base64 does not encrypt.

Next: section 89 "Network protocols and mesh networks".