Embedded HandbookApps
All sections › XIII. Other languages and their build tools
STM32F407 · level intermediate · 25 min read
Open in the interactive handbook

115г. Exchange formats: TLV, protobuf (nanopb) and CBOR on a microcontroller

Section 115в rebuilt the frames of section 194 in Python: a binary packet of eight bytes 01 00 09 C4 0C E4 00 00 — the sequence number, the temperature in hundredths of a degree and the voltage in millivolts — inside a COBS frame with a CRC, 12 bytes in total. The packet is short, but it has a weak spot: only someone who knows the layout knows what the bytes mean. Add a field, and the old parser on the computer reads someone else's bytes as its own. Here the same measurement is written in three self-describing ways: TLV, protobuf and CBOR — and, for comparison, as JSON text.

Each format is encoded by hand in C, without libraries, and checked against other implementations: protobuf against the nanopb library in C and against python protobuf, CBOR against the zcbor library in C and against the cbor package for Python. The same bytes come out on the PC (gcc in the stm-sys:1 image) and on a Cortex-M4 in qemu-system-arm -M netduinoplus2 (the stm-arm:1 image, output over semihosting, section 35а). The code size is measured with armclang for the F407.

References: nanopb 0.4.9 and zcbor 0.9.1 — modules of the Zephyr tree on disk (modules/lib/nanopb, modules/lib/zcbor); python protobuf 7.35.1 and its internal/wire_format.py — wire types and zigzag; the cbor 1.0.0 package from the Zephyr environment; the zcbor README — on canonical CBOR per RFC 8949. The files of this section are in _verify/fmt115e/, checked by _verify/fmt115e_check.py.

What it is called in the documentation

Original Meaning
TLV (type, length, value) tag, length, value
varint variable-length number: 7 bits per byte
zigzag encoding zigzag: −1 → 1, 1 → 2, −2 → 3 — numbers small in magnitude stay short
wire type wire type: how to step over a field without knowing it
field number, tag field number; key = number and wire type
schema evolution schema evolution: new fields without breaking old devices
unknown fields foreign (unknown) fields
major type, additional information CBOR major type, the extra 5 bits of the first byte
definite / indefinite length length given up front / up to a terminator byte
deterministic (canonical) encoding unambiguous encoding: one value — one byte sequence
preferred serialization preferred encoding: the shortest one

115г.1 One measurement and five ways to write it

The message is the same throughout the section: number 1, 25.00 °C (2500 in hundredths), 3300 mV, as in packet 194. For negative and large numbers there is a second one: number 300, −20.00 °C, 3300 mV.

One measurement — number 1, 25.00 °C, 3300 mV — byte by byte: the 115в binary packet 8 bytes, TLV 11 bytes, protobuf 8 bytes, CBOR map with numeric keys 11 bytes, CBOR array 8 bytes, JSON 25 bytes; colour marks the number, the temperature, the voltage and the overhead bytes

Format Bytes What a parser knows without the schema New field
115в binary packet 8 (12 in a frame) nothing breaks the layout
TLV 11 where each field ends stepped over by its length
protobuf 8 the field number and how to step over it stepped over by its wire type
CBOR, map with numeric keys 11 everything: numbers, strings, maps stepped over
CBOR, array 8 everything, but a field is identified by its position only at the end
CBOR, map with names "n", "t", "u" 14 everything, names included stepped over
JSON 25 everything, it is text stepped over

The main difference is not size but self-description. A binary packet cannot be read at all without the schema. TLV and protobuf tell you where a field ends, but not what it means: the meaning of a field number lives in the schema. CBOR and JSON also carry the value type: number, string, map.


115г.2 TLV, varint and zigzag

TLV is the simplest self-describing format: a tag (1 byte), a length (1 byte), a value. In the experiment the value is a signed integer, most significant byte first, in the fewest bytes possible:

size_t tlv_put_int(uint8_t *p, uint8_t tag, int32_t v)
{
    uint8_t n = 4;                         /* убрать лишние старшие байты, не меняя знака */
    while (n > 1) {
        int32_t top = v >> (8 * (n - 1) - 1);   /* старший байт и знаковый бит следующего */
        if (top != 0 && top != -1)
            break;
        n--;
    }
    p[0] = tag;
    p[1] = n;
    for (uint8_t i = 0; i < n; i++)
        p[2 + i] = (uint8_t)((uint32_t)v >> (8 * (n - 1 - i)));
    return 2u + n;
}

The first measurement is 01 01 01 02 02 09 C4 03 02 0C E4, 11 bytes; the second is 01 02 01 2C 02 02 F8 30 03 02 0C E4, 12 bytes: number 300 and −2000 took two bytes each. The parser reads the tag and the length; an unknown tag it steps over by its length without looking inside. The price is two overhead bytes per field and a length limit of 255 bytes.

varint saves space differently: each byte holds 7 bits of the number, the top bit means "one more byte follows", and the low 7 bits come first. This is the basis of protobuf:

Number varint
0, 1, 127 00, 01, 7F — one byte
128 80 01
300 AC 02
3300 E4 19
16383 / 16384 FF 7F / 80 80 01
0xFFFFFFFF FF FF FF FF 0F — five bytes
−2000 as 64-bit B0 F0 FF FF FF FF FF FF FF 01 — ten bytes

The last row is the trouble with negatives: in two's complement all their upper bits are ones, and varint drags them all along. The cure is zigzag: (v << 1) ^ (v >> 31), the sign moves into the lowest bit.

v 0 −1 1 −2 2 2500 −2000 2147483647 −2147483648
zigzag 0 1 2 3 4 5000 3999 4294967294 4294967295

ZigZagEncode from python protobuf gives the same numbers; the experiment checks all of them.


115г.3 protobuf from the wire specification, and nanopb

The schema is the file meas.proto:

message Measurement {        // версия 1
  uint32 n = 1;              // номер измерения
  sint32 t = 2;              // температура, сотые доли °C (sint32: отрицательные — zigzag)
  uint32 u = 3;              // напряжение, мВ
}

On the wire every field is a key followed by a value. The key is the varint (field number << 3) | wire type. Four wire types are in use (the same in wire_format.py: WIRETYPE_VARINT = 0 … WIRETYPE_FIXED32 = 5):

Wire type What follows Fields
0 varint uint32, sint32, int32, bool, enum
1 8 bytes fixed64, double
2 a varint length and that many bytes strings, bytes, nested messages
5 4 bytes fixed32, float

Types 3 and 4 are groups (WIRETYPE_START_GROUP, WIRETYPE_END_GROUP); the section's schema has none. Our own encoder is about ten lines:

size_t meas_pb_encode(const meas_t *m, int version, uint8_t *out)
{
    size_t k = 0;
    k += pb_uint(out + k, 1, m->n);
    k += pb_uint(out + k, 2, zigzag32(m->t));      /* sint32: −1 → 1, 1 → 2, −2000 → 3999 */
    k += pb_uint(out + k, 3, m->u);
    if (version >= 2)
        k += pb_uint(out + k, 4, m->rh);
    return k;
}

pb_uint writes the key and the varint, and does not write a zero at all: in proto3 the default value is not transmitted. The result is 08 01 10 88 27 18 E4 19, 8 bytes: 08 — field 1, type 0; 10 — field 2; 88 27 — 5000, the zigzag of 2500; 18 — field 3; E4 19 — 3300. The second measurement is 9 bytes.

The field type decides the size. The same schema with int32 t instead of sint32 t: for +25.00 °C the only difference is in the bytes (C4 13 instead of 88 27), but −20.00 °C takes 10 bytes in one field, and the message grows from 9 to 17 bytes. float t (degrees) is wire type 5 and 4 bytes, least significant first: 15 00 00 C8 41, a 10-byte message.

nanopb is a protobuf library in C for microcontrollers: a message is a plain struct, and the field description is a table in flash memory. The description is usually produced by the generator nanopb_generator.py, but it needs the schema compiler protoc, and the machine does not have it. The files meas.pb.h and meas.pb.c are typed by hand in the same form the generator writes:

#define Measurement_FIELDLIST(X, a) \
X(a, STATIC,   SINGULAR, UINT32,   n,                 1) \
X(a, STATIC,   SINGULAR, SINT32,   t,                 2) \
X(a, STATIC,   SINGULAR, UINT32,   u,                 3)

It is hard to get them wrong: PB_BIND checks at compile time that the struct's fields fit the description. The maximum message length is known in advance — Measurement_size equals 18 (three fields, each a key plus five bytes of varint), and the experiment with the largest values gave exactly 18 bytes; for the schema with int32 it is 23.

Checking the three implementations against each other:

What Own code nanopb python protobuf 7.35.1
measurement 1 08 01 10 88 27 18 E4 19 same same
measurement 2 08 AC 02 10 9F 1F 18 E4 19 same same
with int32 and with float 8 and 10 bytes / 17 and 11 bytes same same
1000 random measurements 10003 bytes byte for byte in 1000 of 1000 same CRC-32 over all bytes

The schema for python protobuf was not built with protoc either: the check reads meas.proto itself and builds the message descriptions in memory.


115г.4 CBOR: the first byte says it all

CBOR (RFC 8949) is built as a "binary JSON". The first byte of every item holds the major type in the top three bits and the additional information in the low five:

Major type What Example from the experiment
0 integer ≥ 0 01 — 1; 19 09 C4 — 2500
1 integer < 0, stored as −1 − v 39 07 CF — −2000 (1999 = 0x07CF)
3 UTF-8 text 61 6E — the string "n"
4 array 83 — three items
5 map A3 — three pairs
7 float and simple values F9 4E 40 — 25.0 in half precision

Additional information 0…23 is the number itself; 24, 25, 26, 27 mean that 1, 2, 4 or 8 bytes of the number follow, most significant first; 31 means the length is not given, and items run up to the byte FF. Our own header encoder:

size_t cbor_head(uint8_t *p, uint8_t major, uint64_t v)
{
    uint8_t mt = (uint8_t)(major << 5);
    if (v < 24) {
        p[0] = mt | (uint8_t)v;            /* число прямо в первом байте */
        return 1;
    }
    int n = v <= 0xFF ? 1 : v <= 0xFFFF ? 2 : v <= 0xFFFFFFFFu ? 4 : 8;
    p[0] = mt | (uint8_t)(n == 1 ? 24 : n == 2 ? 25 : n == 4 ? 26 : 27);
    for (int i = 0; i < n; i++)
        p[1 + i] = (uint8_t)(v >> (8 * (n - 1 - i)));   /* старшим вперёд */
    return 1u + (size_t)n;
}

The measurement as a map {1: 1, 2: 2500, 3: 3300} is A3 01 01 02 19 09 C4 03 19 0C E4, 11 bytes; as an array [1, 2500, 3300] it is 8 bytes, like protobuf; with the names "n", "t", "u" it is 14 bytes.

Cross-check: the cbor package gave the same bytes for both measurements in all three forms, zcbor with ZCBOR_CANONICAL did so for the map, and on 1000 random measurements our own encoder, zcbor and cbor agreed in 1000 cases out of 1000. Without ZCBOR_CANONICAL zcbor writes the map with indefinite length: BF … FF, 12 bytes instead of 11. This is valid CBOR too, and the cbor package reads it into the same map, but one value now has two encodings — you can no longer compare messages byte by byte or compute a signature over them.

Numbers with a fraction. Our own encoder writes a float in the shortest way without loss: 25.0 → F9 4E 40, half precision, 3 bytes; −20.0 → F9 CD 00. The cbor package on the PC wrote the same map with 25.0 using FB — an 8-byte double, and the message grew from 11 to 17 bytes. Both variants are valid, and a parser has to understand all three float sizes: the cbor package read our encoder's half precision as 25.0 and −20.0.


115г.5 Schema version and backward compatibility

A device in the field is not updated right away, and neither is the computer. So a new message will reach an old parser, and the other way round. Version 2 of the schema adds humidity under a new field number: uint32 rh = 4; (45 %).

Experiment protobuf TLV CBOR
v2 message … 20 2D, 10 bytes … 04 01 2D, 14 bytes A4 … 04 18 2D, 14 bytes
old parser reads v2 nanopb: success, n, t, u correct; own: foreign fields 1 foreign tags 1 foreign keys 1
new parser reads v1 nanopb and python: rh = 0 rh = 0 rh = 0
tag 9 from a future v3 arrives in v2 — v2 skipped 1 tag, v1 — 2 —

Python protobuf also keeps the foreign field: a v2 message read with the old schema and written back matched the original byte for byte. nanopb simply throws the foreign field away.

What you must not do with a schema has been tested too:

Mistake What happened
field 2 was sint32, became uint32 −20.00 °C read as 3999 — "39.99 °C", without a single error (nanopb and python)
field 2 was sint32, became float nanopb: rejected with wrong wire type; python: t = 0.0 without an error, the field's bytes went into the foreign ones
message cut short by one byte nanopb: end-of-stream; python: DecodeError; our own parsers — an error code

Hence the rules common to all three formats:


115г.6 What it costs: message bytes and code bytes

Message size.

Format Measurement 1 Measurement 2 1000 random, average Maximum
115в binary packet 8 8 — 8
protobuf (sint32) 8 9 10.0 18
TLV 11 12 12.6 18
CBOR, map 11 13 13.7 19
CBOR, array 8 10 — —
JSON 25 28 29.1 —

The random measurements: number up to 100000, temperature from −100.00 to +100.00 °C, voltage up to 5000 mV. protobuf and the CBOR array are shorter than TLV because they spend no length byte; JSON is three times longer — digits as text and field names in every message.

Code size — armclang 6.24, -Os, Cortex-M4 with FPU, a full image with the C library; the table shows how much the image grew compared with a program without the format:

What Flash, bytes
own TLV, encode 176
own protobuf, encode 304
own protobuf, decode 644
nanopb, encode 3204
nanopb, decode 4992
own CBOR, encode map 316
zcbor, encode map, ZCBOR_CANONICAL 1548
zcbor without ZCBOR_CANONICAL 868
JSON via snprintf, integers only 692
JSON via snprintf with %.2f 5724

A hand-written encoder for one message is hundreds of bytes, a library is kilobytes: nanopb and zcbor handle all field types, nested messages and checks. A library pays off when there are dozens of messages and the schema changes. A single %.2f in a format string costs more than all of nanopb: library code in the image went up to 5824 bytes from 996. The zcbor README warns that ZCBOR_CANONICAL increases code size — in the experiment by 680 bytes.


115г.7 Pitfalls

A negative temperature took 10 bytes. The field is declared int32: varint drags along all the upper one-bits. For signed numbers use sint32.

After a schema update, "39.99 °C" instead of −20.00. The type was changed under the old field number. Neither nanopb nor python reports an error — the bytes were honestly read by the new schema. A field number is forever.

Python reads it, the device rejects it. The field's wire type changed (sint32 → float): nanopb answers wrong wire type, while python quietly puts the field among the foreign ones and returns t = 0.0. A silent zero is worse than a rejection.

"Humidity 0 %" on old devices. A new proto3 field without optional: "no field" and 0 look the same.

CBOR from two libraries does not match byte for byte. One writes the map with a length, the other up to FF; one writes 25.0 in three bytes, the other in nine. For comparison and signing use the unambiguous encoding (ZCBOR_CANONICAL, shortest numbers).

The message buffer is too small. The protobuf length depends on the values: 8 bytes for the first measurement, 18 for the largest ones. Size the buffer by Measurement_size, not by an example.

Parsing ran past the end of the buffer. A length in TLV or CBOR came from someone else — check it against the rest of the message before reading. Our own parsers return an error on a truncated message, nanopb returns end-of-stream.

JSON cost 5.7 KB of flash. The snprintf had a %.2f. Keep hundredths of a degree as an integer, and printing stays at 692 bytes.


115г.8 Summary

Full source code

fmt.c — our own encoders for varint, TLV, protobuf and CBOR.

/* Раздел 115г: свои кодировщики TLV, protobuf и CBOR — см. fmt.h. */
#include "fmt.h"

/* ════ varint и zigzag ════ */
size_t put_varint(uint8_t *p, uint64_t v)
{
    size_t n = 0;
    while (v >= 0x80) {
        p[n++] = (uint8_t)(v | 0x80);      /* младшие 7 бит и флаг «дальше ещё» */
        v >>= 7;
    }
    p[n++] = (uint8_t)v;
    return n;
}

size_t get_varint(const uint8_t *p, size_t len, uint64_t *v)
{
    uint64_t r = 0;
    for (size_t i = 0; i < len && i < 10; i++) {
        r |= (uint64_t)(p[i] & 0x7F) << (7 * i);
        if (!(p[i] & 0x80)) {
            *v = r;
            return i + 1;
        }
    }
    return 0;                              /* обрыв посреди числа или больше 10 байт */
}

/* ════ TLV ════ */
size_t tlv_put_int(uint8_t *p, uint8_t tag, int32_t v)
{
    uint8_t n = 4;                         /* убрать лишние старшие байты, не меняя знака */
    while (n > 1) {
        int32_t top = v >> (8 * (n - 1) - 1);   /* старший байт и знаковый бит следующего */
        if (top != 0 && top != -1)
            break;
        n--;
    }
    p[0] = tag;
    p[1] = n;
    for (uint8_t i = 0; i < n; i++)
        p[2 + i] = (uint8_t)((uint32_t)v >> (8 * (n - 1 - i)));
    return 2u + n;
}

size_t meas_tlv_encode(const meas_t *m, int version, uint8_t *out)
{
    size_t k = 0;
    k += tlv_put_int(out + k, 1, (int32_t)m->n);
    k += tlv_put_int(out + k, 2, m->t);
    k += tlv_put_int(out + k, 3, (int32_t)m->u);
    if (version >= 2)
        k += tlv_put_int(out + k, 4, (int32_t)m->rh);
    return k;
}

int meas_tlv_decode(const uint8_t *p, size_t len, int version, meas_t *m)
{
    int skipped = 0, last = version >= 2 ? 4 : 3;   /* сколько тегов знает эта версия */
    size_t i = 0;
    *m = (meas_t){0};
    while (i < len) {
        if (len - i < 2 || p[i + 1] > len - i - 2)
            return -1;                     /* длина выходит за конец сообщения */
        uint8_t tag = p[i], l = p[i + 1];
        const uint8_t *v = p + i + 2;
        i += 2u + l;
        if (tag < 1 || tag > last) {          /* чужой тег: длина известна — просто перешагнуть */
            skipped++;
            continue;
        }
        if (l < 1 || l > 4)
            return -2;
        uint32_t x = (v[0] & 0x80) ? 0xFFFFFFFFu : 0;   /* расширить знак */
        for (uint8_t j = 0; j < l; j++)
            x = (x << 8) | v[j];
        if (tag == 1) m->n = x;
        if (tag == 2) m->t = (int32_t)x;
        if (tag == 3) m->u = x;
        if (tag == 4) m->rh = x;
    }
    return skipped;
}

/* ════ protobuf ════ */
size_t pb_key(uint8_t *p, uint32_t field, uint32_t wt)
{
    return put_varint(p, ((uint64_t)field << 3) | wt);
}

static size_t pb_uint(uint8_t *p, uint32_t field, uint64_t v)
{
    if (v == 0)
        return 0;                          /* proto3: значение по умолчанию не передаётся */
    size_t k = pb_key(p, field, WT_VARINT);
    return k + put_varint(p + k, v);
}

size_t meas_pb_encode(const meas_t *m, int version, uint8_t *out)
{
    size_t k = 0;
    k += pb_uint(out + k, 1, m->n);
    k += pb_uint(out + k, 2, zigzag32(m->t));      /* sint32: −1 → 1, 1 → 2, −2000 → 3999 */
    k += pb_uint(out + k, 3, m->u);
    if (version >= 2)
        k += pb_uint(out + k, 4, m->rh);
    return k;
}

size_t meas_pb_encode_int32(const meas_t *m, uint8_t *out)
{
    size_t k = 0;
    k += pb_uint(out + k, 1, m->n);
    k += pb_uint(out + k, 2, (uint64_t)(int64_t)m->t);  /* int32: отрицательное расширяется до 64 бит */
    k += pb_uint(out + k, 3, m->u);
    return k;
}

static uint32_t float_bits(float f)
{
    union { float f; uint32_t u; } c = {f};
    return c.u;
}

size_t meas_pb_encode_float(const meas_t *m, uint8_t *out)
{
    size_t k = pb_uint(out, 1, m->n);
    float deg = (float)m->t / 100.0f;
    if (deg != 0.0f) {
        k += pb_key(out + k, 2, WT_I32);
        uint32_t b = float_bits(deg);
        for (int i = 0; i < 4; i++)
            out[k++] = (uint8_t)(b >> (8 * i));    /* fixed32 и float — младшим байтом вперёд */
    }
    k += pb_uint(out + k, 3, m->u);
    return k;
}

int meas_pb_decode(const uint8_t *p, size_t len, int version, meas_t *m)
{
    uint32_t last = version >= 2 ? 4 : 3;
    int skipped = 0;
    size_t i = 0;
    *m = (meas_t){0};
    while (i < len) {
        uint64_t key, v = 0;
        size_t k = get_varint(p + i, len - i, &key);
        if (!k)
            return -1;
        i += k;
        uint32_t field = (uint32_t)(key >> 3), wt = (uint32_t)(key & 7);
        if (wt == WT_VARINT) {
            k = get_varint(p + i, len - i, &v);
            if (!k)
                return -1;
            i += k;
        } else if (wt == WT_I64 || wt == WT_I32) {
            size_t w = wt == WT_I64 ? 8 : 4;
            if (len - i < w)
                return -1;
            i += w;
        } else if (wt == WT_LEN) {
            k = get_varint(p + i, len - i, &v);
            if (!k || v > len - i - k)
                return -1;
            i += k + (size_t)v;
        } else {
            return -3;                     /* типы 3, 4, 6, 7 — группы и мусор */
        }
        if (field >= 1 && field <= last && wt != WT_VARINT)
            return -2;                     /* знакомое поле пришло с другим типом провода */
        if (field == 1) m->n = (uint32_t)v;
        else if (field == 2) m->t = unzigzag32((uint32_t)v);
        else if (field == 3) m->u = (uint32_t)v;
        else if (field == 4 && last >= 4) m->rh = (uint32_t)v;
        else skipped++;                    /* чужое поле: тип провода говорит, сколько перешагнуть */
    }
    return skipped;
}

/* ════ CBOR ════ */
size_t cbor_head(uint8_t *p, uint8_t major, uint64_t v)
{
    uint8_t mt = (uint8_t)(major << 5);
    if (v < 24) {
        p[0] = mt | (uint8_t)v;            /* число прямо в первом байте */
        return 1;
    }
    int n = v <= 0xFF ? 1 : v <= 0xFFFF ? 2 : v <= 0xFFFFFFFFu ? 4 : 8;
    p[0] = mt | (uint8_t)(n == 1 ? 24 : n == 2 ? 25 : n == 4 ? 26 : 27);
    for (int i = 0; i < n; i++)
        p[1 + i] = (uint8_t)(v >> (8 * (n - 1 - i)));   /* старшим вперёд */
    return 1u + (size_t)n;
}

size_t cbor_int(uint8_t *p, int64_t v)
{
    return v >= 0 ? cbor_head(p, 0, (uint64_t)v) : cbor_head(p, 1, (uint64_t)(-1 - v));  /* −1 → 0x20 */
}

size_t cbor_tstr(uint8_t *p, const char *s)
{
    size_t n = 0;
    while (s[n])
        n++;
    size_t k = cbor_head(p, 3, n);
    for (size_t i = 0; i < n; i++)
        p[k + i] = (uint8_t)s[i];
    return k + n;
}

size_t cbor_float(uint8_t *p, float f)
{
    uint32_t b = float_bits(f);
    uint32_t s = b >> 31, e = (b >> 23) & 0xFF, m = b & 0x7FFFFF;
    int he = (int)e - 127 + 15;
    if ((b & 0x7FFFFFFF) == 0 || (e != 0xFF && he >= 1 && he <= 30 && (m & 0x1FFF) == 0)) {
        uint16_t h = (uint16_t)(s << 15);
        if (b & 0x7FFFFFFF)
            h |= (uint16_t)((uint32_t)he << 10 | m >> 13);
        p[0] = 0xF9;                       /* половинная точность: 2 байта без потерь */
        p[1] = (uint8_t)(h >> 8);
        p[2] = (uint8_t)h;
        return 3;
    }
    p[0] = 0xFA;                           /* иначе одинарная: 4 байта */
    for (int i = 0; i < 4; i++)
        p[1 + i] = (uint8_t)(b >> (8 * (3 - i)));
    return 5;
}

size_t meas_cbor_map(const meas_t *m, int version, uint8_t *out)
{
    size_t k = cbor_head(out, 5, version >= 2 ? 4 : 3);
    k += cbor_int(out + k, 1); k += cbor_int(out + k, m->n);
    k += cbor_int(out + k, 2); k += cbor_int(out + k, m->t);
    k += cbor_int(out + k, 3); k += cbor_int(out + k, m->u);
    if (version >= 2) {
        k += cbor_int(out + k, 4); k += cbor_int(out + k, m->rh);
    }
    return k;
}

size_t meas_cbor_array(const meas_t *m, uint8_t *out)
{
    size_t k = cbor_head(out, 4, 3);
    k += cbor_int(out + k, m->n);
    k += cbor_int(out + k, m->t);
    k += cbor_int(out + k, m->u);
    return k;
}

size_t meas_cbor_smap(const meas_t *m, uint8_t *out)
{
    size_t k = cbor_head(out, 5, 3);
    k += cbor_tstr(out + k, "n"); k += cbor_int(out + k, m->n);
    k += cbor_tstr(out + k, "t"); k += cbor_int(out + k, m->t);
    k += cbor_tstr(out + k, "u"); k += cbor_int(out + k, m->u);
    return k;
}

size_t meas_cbor_map_float(const meas_t *m, uint8_t *out)
{
    size_t k = cbor_head(out, 5, 3);
    k += cbor_int(out + k, 1); k += cbor_int(out + k, m->n);
    k += cbor_int(out + k, 2); k += cbor_float(out + k, (float)m->t / 100.0f);
    k += cbor_int(out + k, 3); k += cbor_int(out + k, m->u);
    return k;
}

/* разбор: заголовок элемента */
static int rd_head(const uint8_t *p, size_t len, size_t *i, uint8_t *major, uint64_t *v, int *indef)
{
    if (*i >= len)
        return 0;
    uint8_t b = p[(*i)++], ai = b & 31;
    *major = b >> 5;
    *indef = 0;
    *v = 0;
    if (ai < 24) {
        *v = ai;
        return 1;
    }
    if (ai == 31) {                        /* неопределённая длина: до байта 0xFF */
        *indef = 1;
        return 1;
    }
    if (ai > 27)
        return 0;
    size_t n = (size_t)1 << (ai - 24);
    if (len - *i < n)
        return 0;
    for (size_t j = 0; j < n; j++)
        *v = (*v << 8) | p[(*i)++];
    return 1;
}

static int skip_item(const uint8_t *p, size_t len, size_t *i, int depth)
{
    uint8_t mt;
    uint64_t v;
    int indef;
    if (depth > 8 || !rd_head(p, len, i, &mt, &v, &indef))
        return 0;
    if (mt <= 1 || (mt == 7 && !indef))
        return 1;                          /* числа, простые значения и float — только заголовок */
    if (mt == 2 || mt == 3) {
        if (indef || v > len - *i)
            return 0;
        *i += (size_t)v;
        return 1;
    }
    if (mt == 6)
        return skip_item(p, len, i, depth + 1);   /* тег + одно значение */
    uint64_t items = mt == 5 ? 2 * v : v;
    if (indef) {
        while (*i < len && p[*i] != 0xFF)
            if (!skip_item(p, len, i, depth + 1))
                return 0;
        return *i < len ? (++*i, 1) : 0;
    }
    for (uint64_t j = 0; j < items; j++)
        if (!skip_item(p, len, i, depth + 1))
            return 0;
    return 1;
}

int meas_cbor_decode(const uint8_t *p, size_t len, int version, meas_t *m)
{
    uint64_t last = version >= 2 ? 4 : 3;
    size_t i = 0;
    uint8_t mt;
    uint64_t cnt, key, v;
    int indef, skipped = 0;
    *m = (meas_t){0};
    if (!rd_head(p, len, &i, &mt, &cnt, &indef) || mt != 5)
        return -1;
    for (uint64_t j = 0; indef ? (i < len && p[i] != 0xFF) : j < cnt; j++) {
        int ki;
        if (!rd_head(p, len, &i, &mt, &key, &ki))
            return -1;                     /* обрыв */
        if (mt != 0)
            return -2;                     /* ключи в этой схеме — только неотрицательные числа */
        if (key < 1 || key > last) {
            if (!skip_item(p, len, &i, 0))
                return -1;
            skipped++;
            continue;
        }
        if (!rd_head(p, len, &i, &mt, &v, &ki))
            return -1;
        if (mt > 1)
            return -2;
        int64_t x = mt == 0 ? (int64_t)v : -1 - (int64_t)v;
        if (key == 1) m->n = (uint32_t)x;
        if (key == 2) m->t = (int32_t)x;
        if (key == 3) m->u = (uint32_t)x;
        if (key == 4) m->rh = (uint32_t)x;
    }
    if (indef) {
        if (i >= len)
            return -1;
        i++;                               /* байт 0xFF — конец словаря */
    }
    return i == len ? skipped : -1;
}

meas.pb.h — the message descriptions for nanopb in the form the generator writes them.

/* Раздел 115г: описание сообщений meas.proto для nanopb 0.4.9 — в том виде, в каком его пишет
 * генератор nanopb_generator.py. Генератору нужен protoc, а его на машине нет, поэтому файл
 * набран вручную; правильность сверяет сама nanopb (PB_BIND проверяет размеры и смещения полей
 * при компиляции) и опыт: байты nanopb = байты своего кодировщика = байты python protobuf. */
#ifndef PB_MEAS_PB_H_INCLUDED
#define PB_MEAS_PB_H_INCLUDED
#include <pb.h>

#if PB_PROTO_HEADER_VERSION != 40
#error Regenerate this file with the current version of nanopb generator.
#endif

/* Struct definitions */
typedef struct _Measurement {
    uint32_t n;
    int32_t t;
    uint32_t u;
} Measurement;

typedef struct _Measurement2 {
    uint32_t n;
    int32_t t;
    uint32_t u;
    uint32_t rh;
} Measurement2;

typedef struct _MeasurementI {
    uint32_t n;
    int32_t t;
    uint32_t u;
} MeasurementI;

typedef struct _MeasurementU {
    uint32_t n;
    uint32_t t;
    uint32_t u;
} MeasurementU;

typedef struct _MeasurementF {
    uint32_t n;
    float t;
    uint32_t u;
} MeasurementF;

/* Initializer values for message structs */
#define Measurement_init_zero                    {0, 0, 0}
#define Measurement2_init_zero                   {0, 0, 0, 0}
#define MeasurementI_init_zero                   {0, 0, 0}
#define MeasurementU_init_zero                   {0, 0, 0}
#define MeasurementF_init_zero                   {0, 0, 0}

/* Struct field encoding specification for nanopb */
#define Measurement_FIELDLIST(X, a) \
X(a, STATIC,   SINGULAR, UINT32,   n,                 1) \
X(a, STATIC,   SINGULAR, SINT32,   t,                 2) \
X(a, STATIC,   SINGULAR, UINT32,   u,                 3)
#define Measurement_CALLBACK NULL
#define Measurement_DEFAULT NULL

#define Measurement2_FIELDLIST(X, a) \
X(a, STATIC,   SINGULAR, UINT32,   n,                 1) \
X(a, STATIC,   SINGULAR, SINT32,   t,                 2) \
X(a, STATIC,   SINGULAR, UINT32,   u,                 3) \
X(a, STATIC,   SINGULAR, UINT32,   rh,                4)
#define Measurement2_CALLBACK NULL
#define Measurement2_DEFAULT NULL

#define MeasurementI_FIELDLIST(X, a) X(a, STATIC,   SINGULAR, UINT32,   n,                 1) X(a, STATIC,   SINGULAR, INT32,    t,                 2) X(a, STATIC,   SINGULAR, UINT32,   u,                 3)
#define MeasurementI_CALLBACK NULL
#define MeasurementI_DEFAULT NULL

#define MeasurementU_FIELDLIST(X, a) \
X(a, STATIC,   SINGULAR, UINT32,   n,                 1) \
X(a, STATIC,   SINGULAR, UINT32,   t,                 2) \
X(a, STATIC,   SINGULAR, UINT32,   u,                 3)
#define MeasurementU_CALLBACK NULL
#define MeasurementU_DEFAULT NULL

#define MeasurementF_FIELDLIST(X, a) \
X(a, STATIC,   SINGULAR, UINT32,   n,                 1) \
X(a, STATIC,   SINGULAR, FLOAT,    t,                 2) \
X(a, STATIC,   SINGULAR, UINT32,   u,                 3)
#define MeasurementF_CALLBACK NULL
#define MeasurementF_DEFAULT NULL

extern const pb_msgdesc_t Measurement_msg;
extern const pb_msgdesc_t Measurement2_msg;
extern const pb_msgdesc_t MeasurementI_msg;
extern const pb_msgdesc_t MeasurementU_msg;
extern const pb_msgdesc_t MeasurementF_msg;

#define Measurement_fields &Measurement_msg
#define Measurement2_fields &Measurement2_msg
#define MeasurementI_fields &MeasurementI_msg
#define MeasurementU_fields &MeasurementU_msg
#define MeasurementF_fields &MeasurementF_msg

/* Maximum encoded size of messages (where known) */
#define Measurement_size                         18
#define Measurement2_size                        24
#define MeasurementI_size                        23
#define MeasurementU_size                        18
#define MeasurementF_size                        17

#endif

Next: section 116 "Typical approaches and algorithms: from a threshold to a small neural network".

Open in the interactive handbook