115г. Exchange formats: TLV, protobuf (nanopb) and CBOR on a microcontroller
Section 115в rebuilt the frames of section 194 in Python: a binary packet of eight
bytes 01 00 09 C4 0C E4 00 00 — the sequence number, the temperature in hundredths of a degree and
the voltage in millivolts — inside a COBS frame with a CRC, 12 bytes in total. The packet is short, but it has a weak spot: only someone who knows the layout
knows what the bytes mean. Add a field, and the old parser on
the computer reads someone else's bytes as its own. Here the same measurement is written
in three self-describing ways: TLV, protobuf and CBOR — and, for comparison,
as JSON text.
Each format is encoded by hand in C, without libraries, and checked against
other implementations: protobuf against the nanopb library in C and against python
protobuf, CBOR against the zcbor library in C and against the cbor package for Python.
The same bytes come out on the PC (gcc in the stm-sys:1 image) and on a Cortex-M4
in qemu-system-arm -M netduinoplus2 (the stm-arm:1 image, output over
semihosting, section 35а). The code size is measured with armclang for the F407.
References: nanopb 0.4.9 and zcbor 0.9.1 — modules of the Zephyr tree on disk
(modules/lib/nanopb, modules/lib/zcbor); python protobuf 7.35.1 and its
internal/wire_format.py — wire types and zigzag; the cbor 1.0.0 package from
the Zephyr environment; the zcbor README — on canonical CBOR per RFC 8949. The files of
this section are in _verify/fmt115e/, checked by _verify/fmt115e_check.py.
What it is called in the documentation
Original Meaning TLV (type, length, value) tag, length, value varint variable-length number: 7 bits per byte zigzag encoding zigzag: −1 → 1, 1 → 2, −2 → 3 — numbers small in magnitude stay short wire type wire type: how to step over a field without knowing it field number, tag field number; key = number and wire type schema evolution schema evolution: new fields without breaking old devices unknown fields foreign (unknown) fields major type, additional information CBOR major type, the extra 5 bits of the first byte definite / indefinite length length given up front / up to a terminator byte deterministic (canonical) encoding unambiguous encoding: one value — one byte sequence preferred serialization preferred encoding: the shortest one
115г.1 One measurement and five ways to write it
The message is the same throughout the section: number 1, 25.00 °C (2500 in hundredths), 3300 mV, as in packet 194. For negative and large numbers there is a second one: number 300, −20.00 °C, 3300 mV.
| Format | Bytes | What a parser knows without the schema | New field |
|---|---|---|---|
| 115в binary packet | 8 (12 in a frame) | nothing | breaks the layout |
| TLV | 11 | where each field ends | stepped over by its length |
| protobuf | 8 | the field number and how to step over it | stepped over by its wire type |
| CBOR, map with numeric keys | 11 | everything: numbers, strings, maps | stepped over |
| CBOR, array | 8 | everything, but a field is identified by its position | only at the end |
CBOR, map with names "n", "t", "u" |
14 | everything, names included | stepped over |
| JSON | 25 | everything, it is text | stepped over |
The main difference is not size but self-description. A binary packet cannot be read at all without the schema. TLV and protobuf tell you where a field ends, but not what it means: the meaning of a field number lives in the schema. CBOR and JSON also carry the value type: number, string, map.
115г.2 TLV, varint and zigzag
TLV is the simplest self-describing format: a tag (1 byte), a length (1 byte), a value. In the experiment the value is a signed integer, most significant byte first, in the fewest bytes possible:
size_t tlv_put_int(uint8_t *p, uint8_t tag, int32_t v)
{
uint8_t n = 4; /* убрать лишние старшие байты, не меняя знака */
while (n > 1) {
int32_t top = v >> (8 * (n - 1) - 1); /* старший байт и знаковый бит следующего */
if (top != 0 && top != -1)
break;
n--;
}
p[0] = tag;
p[1] = n;
for (uint8_t i = 0; i < n; i++)
p[2 + i] = (uint8_t)((uint32_t)v >> (8 * (n - 1 - i)));
return 2u + n;
}
The first measurement is 01 01 01 02 02 09 C4 03 02 0C E4, 11 bytes; the second is
01 02 01 2C 02 02 F8 30 03 02 0C E4, 12 bytes: number 300 and −2000 took
two bytes each. The parser reads the tag and the length; an unknown tag it steps over by
its length without looking inside. The price is two overhead bytes per field and a length
limit of 255 bytes.
varint saves space differently: each byte holds 7 bits of the number, the top bit means "one more byte follows", and the low 7 bits come first. This is the basis of protobuf:
| Number | varint |
|---|---|
| 0, 1, 127 | 00, 01, 7F — one byte |
| 128 | 80 01 |
| 300 | AC 02 |
| 3300 | E4 19 |
| 16383 / 16384 | FF 7F / 80 80 01 |
| 0xFFFFFFFF | FF FF FF FF 0F — five bytes |
| −2000 as 64-bit | B0 F0 FF FF FF FF FF FF FF 01 — ten bytes |
The last row is the trouble with negatives: in two's complement all their
upper bits are ones, and varint drags them all along. The cure is zigzag:
(v << 1) ^ (v >> 31), the sign moves into the lowest bit.
| v | 0 | −1 | 1 | −2 | 2 | 2500 | −2000 | 2147483647 | −2147483648 |
|---|---|---|---|---|---|---|---|---|---|
| zigzag | 0 | 1 | 2 | 3 | 4 | 5000 | 3999 | 4294967294 | 4294967295 |
ZigZagEncode from python protobuf gives the same numbers; the experiment checks all of them.
115г.3 protobuf from the wire specification, and nanopb
The schema is the file meas.proto:
message Measurement { // версия 1
uint32 n = 1; // номер измерения
sint32 t = 2; // температура, сотые доли °C (sint32: отрицательные — zigzag)
uint32 u = 3; // напряжение, мВ
}
On the wire every field is a key followed by a value. The key is the varint
(field number << 3) | wire type. Four wire types are in use
(the same in wire_format.py: WIRETYPE_VARINT = 0 … WIRETYPE_FIXED32 = 5):
| Wire type | What follows | Fields |
|---|---|---|
| 0 | varint | uint32, sint32, int32, bool, enum |
| 1 | 8 bytes | fixed64, double |
| 2 | a varint length and that many bytes | strings, bytes, nested messages |
| 5 | 4 bytes | fixed32, float |
Types 3 and 4 are groups (WIRETYPE_START_GROUP, WIRETYPE_END_GROUP); the section's schema
has none. Our own encoder is about ten lines:
size_t meas_pb_encode(const meas_t *m, int version, uint8_t *out)
{
size_t k = 0;
k += pb_uint(out + k, 1, m->n);
k += pb_uint(out + k, 2, zigzag32(m->t)); /* sint32: −1 → 1, 1 → 2, −2000 → 3999 */
k += pb_uint(out + k, 3, m->u);
if (version >= 2)
k += pb_uint(out + k, 4, m->rh);
return k;
}
pb_uint writes the key and the varint, and does not write a zero at all: in proto3 the default
value is not transmitted. The result is 08 01 10 88 27 18 E4 19, 8 bytes:
08 — field 1, type 0; 10 — field 2; 88 27 — 5000, the zigzag of 2500; 18 —
field 3; E4 19 — 3300. The second measurement is 9 bytes.
The field type decides the size. The same schema with int32 t instead of sint32 t: for
+25.00 °C the only difference is in the bytes (C4 13 instead of 88 27), but −20.00 °C
takes 10 bytes in one field, and the message grows from 9 to 17 bytes.
float t (degrees) is wire type 5 and 4 bytes, least significant first:
15 00 00 C8 41, a 10-byte message.
nanopb is a protobuf library in C for microcontrollers: a message is
a plain struct, and the field description is a table in flash memory. The description is usually
produced by the generator nanopb_generator.py, but it needs the schema compiler
protoc, and the machine does not have it. The files meas.pb.h and meas.pb.c are typed
by hand in the same form the generator writes:
#define Measurement_FIELDLIST(X, a) \
X(a, STATIC, SINGULAR, UINT32, n, 1) \
X(a, STATIC, SINGULAR, SINT32, t, 2) \
X(a, STATIC, SINGULAR, UINT32, u, 3)
It is hard to get them wrong: PB_BIND checks at compile time that the struct's fields
fit the description. The maximum message length is known
in advance — Measurement_size equals 18 (three fields, each a key plus five bytes of
varint), and the experiment with the largest values gave exactly 18 bytes; for the schema with
int32 it is 23.
Checking the three implementations against each other:
| What | Own code | nanopb | python protobuf 7.35.1 |
|---|---|---|---|
| measurement 1 | 08 01 10 88 27 18 E4 19 |
same | same |
| measurement 2 | 08 AC 02 10 9F 1F 18 E4 19 |
same | same |
with int32 and with float |
8 and 10 bytes / 17 and 11 bytes | same | same |
| 1000 random measurements | 10003 bytes | byte for byte in 1000 of 1000 | same CRC-32 over all bytes |
The schema for python protobuf was not built with protoc either: the check reads
meas.proto itself and builds the message descriptions in memory.
115г.4 CBOR: the first byte says it all
CBOR (RFC 8949) is built as a "binary JSON". The first byte of every item holds the major type in the top three bits and the additional information in the low five:
| Major type | What | Example from the experiment |
|---|---|---|
| 0 | integer ≥ 0 | 01 — 1; 19 09 C4 — 2500 |
| 1 | integer < 0, stored as −1 − v | 39 07 CF — −2000 (1999 = 0x07CF) |
| 3 | UTF-8 text | 61 6E — the string "n" |
| 4 | array | 83 — three items |
| 5 | map | A3 — three pairs |
| 7 | float and simple values | F9 4E 40 — 25.0 in half precision |
Additional information 0…23 is the number itself; 24, 25, 26, 27 mean that 1, 2, 4 or 8
bytes of the number follow, most significant first; 31 means the length is not given, and items run up to the byte
FF. Our own header encoder:
size_t cbor_head(uint8_t *p, uint8_t major, uint64_t v)
{
uint8_t mt = (uint8_t)(major << 5);
if (v < 24) {
p[0] = mt | (uint8_t)v; /* число прямо в первом байте */
return 1;
}
int n = v <= 0xFF ? 1 : v <= 0xFFFF ? 2 : v <= 0xFFFFFFFFu ? 4 : 8;
p[0] = mt | (uint8_t)(n == 1 ? 24 : n == 2 ? 25 : n == 4 ? 26 : 27);
for (int i = 0; i < n; i++)
p[1 + i] = (uint8_t)(v >> (8 * (n - 1 - i))); /* старшим вперёд */
return 1u + (size_t)n;
}
The measurement as a map {1: 1, 2: 2500, 3: 3300} is A3 01 01 02 19 09 C4 03 19 0C E4,
11 bytes; as an array [1, 2500, 3300] it is 8 bytes, like protobuf; with the names
"n", "t", "u" it is 14 bytes.
Cross-check: the cbor package gave the same bytes for both measurements in all three forms,
zcbor with ZCBOR_CANONICAL did so for the map, and on 1000 random measurements
our own encoder, zcbor and cbor agreed in 1000 cases out of 1000. Without ZCBOR_CANONICAL zcbor writes the map
with indefinite length: BF … FF, 12 bytes instead of 11. This is valid
CBOR too, and the cbor package reads it into the same map, but one value
now has two encodings — you can no longer compare messages byte by byte or compute
a signature over them.
Numbers with a fraction. Our own encoder writes a float in the shortest way without
loss: 25.0 → F9 4E 40, half precision, 3 bytes; −20.0 → F9 CD 00.
The cbor package on the PC wrote the same map with 25.0 using FB — an 8-byte
double, and the message grew from 11 to 17 bytes. Both variants are valid, and
a parser has to understand all three float sizes: the cbor package read
our encoder's half precision as 25.0 and −20.0.
115г.5 Schema version and backward compatibility
A device in the field is not updated right away, and neither is the computer. So a new
message will reach an old parser, and the other way round. Version 2 of the schema
adds humidity under a new field number: uint32 rh = 4; (45 %).
| Experiment | protobuf | TLV | CBOR |
|---|---|---|---|
| v2 message | … 20 2D, 10 bytes |
… 04 01 2D, 14 bytes |
A4 … 04 18 2D, 14 bytes |
| old parser reads v2 | nanopb: success, n, t, u correct; own: foreign fields 1 | foreign tags 1 | foreign keys 1 |
| new parser reads v1 | nanopb and python: rh = 0 |
rh = 0 |
rh = 0 |
| tag 9 from a future v3 arrives in v2 | — | v2 skipped 1 tag, v1 — 2 | — |
Python protobuf also keeps the foreign field: a v2 message read with the old schema and written back matched the original byte for byte. nanopb simply throws the foreign field away.
What you must not do with a schema has been tested too:
| Mistake | What happened |
|---|---|
field 2 was sint32, became uint32 |
−20.00 °C read as 3999 — "39.99 °C", without a single error (nanopb and python) |
field 2 was sint32, became float |
nanopb: rejected with wrong wire type; python: t = 0.0 without an error, the field's bytes went into the foreign ones |
| message cut short by one byte | nanopb: end-of-stream; python: DecodeError; our own parsers — an error code |
Hence the rules common to all three formats:
- a field number is forever: do not change its type and do not give it to another field, even after the old one has been removed;
- a new field gets only a new number, and old code must work without it;
- in proto3 zero and "field absent" are indistinguishable: if 0 is a meaningful value,
declare the field
optional(per the nanopb CHANGELOG — since protoc 3.12), and it gets ahas_<field>flag, or agree that 0 means "no data"; - a binary packet like 115в needs a version byte at the start, otherwise an old parser will not know the layout is different.
115г.6 What it costs: message bytes and code bytes
Message size.
| Format | Measurement 1 | Measurement 2 | 1000 random, average | Maximum |
|---|---|---|---|---|
| 115в binary packet | 8 | 8 | — | 8 |
protobuf (sint32) |
8 | 9 | 10.0 | 18 |
| TLV | 11 | 12 | 12.6 | 18 |
| CBOR, map | 11 | 13 | 13.7 | 19 |
| CBOR, array | 8 | 10 | — | — |
| JSON | 25 | 28 | 29.1 | — |
The random measurements: number up to 100000, temperature from −100.00 to +100.00 °C, voltage up to 5000 mV. protobuf and the CBOR array are shorter than TLV because they spend no length byte; JSON is three times longer — digits as text and field names in every message.
Code size — armclang 6.24, -Os, Cortex-M4 with FPU, a full image with
the C library; the table shows how much the image grew compared with a program without
the format:
| What | Flash, bytes |
|---|---|
| own TLV, encode | 176 |
| own protobuf, encode | 304 |
| own protobuf, decode | 644 |
| nanopb, encode | 3204 |
| nanopb, decode | 4992 |
| own CBOR, encode map | 316 |
zcbor, encode map, ZCBOR_CANONICAL |
1548 |
zcbor without ZCBOR_CANONICAL |
868 |
JSON via snprintf, integers only |
692 |
JSON via snprintf with %.2f |
5724 |
A hand-written encoder for one message is hundreds of bytes, a library is kilobytes:
nanopb and zcbor handle all field types, nested messages and checks.
A library pays off when there are dozens of messages and the schema changes. A single %.2f in
a format string costs more than all of nanopb: library code in
the image went up to 5824 bytes from 996. The zcbor README warns that
ZCBOR_CANONICAL increases code size — in the experiment by 680 bytes.
115г.7 Pitfalls
A negative temperature took 10 bytes. The field is declared int32: varint
drags along all the upper one-bits. For signed numbers use sint32.
After a schema update, "39.99 °C" instead of −20.00. The type was changed under the old field number. Neither nanopb nor python reports an error — the bytes were honestly read by the new schema. A field number is forever.
Python reads it, the device rejects it. The field's wire type changed (sint32 →
float): nanopb answers wrong wire type, while python quietly puts the field among
the foreign ones and returns t = 0.0. A silent zero is worse than a rejection.
"Humidity 0 %" on old devices. A new proto3 field without optional:
"no field" and 0 look the same.
CBOR from two libraries does not match byte for byte. One writes the map with
a length, the other up to FF; one writes 25.0 in three bytes, the other in nine.
For comparison and signing use the unambiguous encoding (ZCBOR_CANONICAL, shortest
numbers).
The message buffer is too small. The protobuf length depends on the values: 8 bytes for
the first measurement, 18 for the largest ones. Size the buffer by Measurement_size, not
by an example.
Parsing ran past the end of the buffer. A length in TLV or CBOR came from someone else —
check it against the rest of the message before reading. Our own parsers return
an error on a truncated message, nanopb returns end-of-stream.
JSON cost 5.7 KB of flash. The snprintf had a %.2f. Keep hundredths of a
degree as an integer, and printing stays at 692 bytes.
115г.8 Summary
- One measurement: binary packet 8 bytes, protobuf 8, CBOR array 8, TLV 11, CBOR map 11, JSON 25; over 1000 random ones the average is protobuf 10.0, TLV 12.6, CBOR 13.7, JSON 29.1 bytes.
- varint is 7 bits per byte; a negative
int32is 10 bytes, zigzag (sint32) turns −2000 into 3999 and two bytes. - Our own protobuf matched nanopb and python protobuf on all vectors and on
1000 random measurements; our own CBOR matched zcbor (
ZCBOR_CANONICAL) and thecborpackage; the bytes are the same on the PC and on a Cortex-M4 in QEMU. - Compatibility: all three formats step over a new field with a new number; changing the type under an old number gives a silent "39.99 °C"; in proto3 "field absent" = 0.
- F407 flash: own encoders — 176…316 bytes, nanopb — 3204 for encoding and
4992 for decoding, zcbor — 1548; JSON — 692 bytes with integers and 5724 with
%.2f.
Full source code
fmt.c — our own encoders for varint, TLV, protobuf and CBOR.
/* Раздел 115г: свои кодировщики TLV, protobuf и CBOR — см. fmt.h. */
#include "fmt.h"
/* ════ varint и zigzag ════ */
size_t put_varint(uint8_t *p, uint64_t v)
{
size_t n = 0;
while (v >= 0x80) {
p[n++] = (uint8_t)(v | 0x80); /* младшие 7 бит и флаг «дальше ещё» */
v >>= 7;
}
p[n++] = (uint8_t)v;
return n;
}
size_t get_varint(const uint8_t *p, size_t len, uint64_t *v)
{
uint64_t r = 0;
for (size_t i = 0; i < len && i < 10; i++) {
r |= (uint64_t)(p[i] & 0x7F) << (7 * i);
if (!(p[i] & 0x80)) {
*v = r;
return i + 1;
}
}
return 0; /* обрыв посреди числа или больше 10 байт */
}
/* ════ TLV ════ */
size_t tlv_put_int(uint8_t *p, uint8_t tag, int32_t v)
{
uint8_t n = 4; /* убрать лишние старшие байты, не меняя знака */
while (n > 1) {
int32_t top = v >> (8 * (n - 1) - 1); /* старший байт и знаковый бит следующего */
if (top != 0 && top != -1)
break;
n--;
}
p[0] = tag;
p[1] = n;
for (uint8_t i = 0; i < n; i++)
p[2 + i] = (uint8_t)((uint32_t)v >> (8 * (n - 1 - i)));
return 2u + n;
}
size_t meas_tlv_encode(const meas_t *m, int version, uint8_t *out)
{
size_t k = 0;
k += tlv_put_int(out + k, 1, (int32_t)m->n);
k += tlv_put_int(out + k, 2, m->t);
k += tlv_put_int(out + k, 3, (int32_t)m->u);
if (version >= 2)
k += tlv_put_int(out + k, 4, (int32_t)m->rh);
return k;
}
int meas_tlv_decode(const uint8_t *p, size_t len, int version, meas_t *m)
{
int skipped = 0, last = version >= 2 ? 4 : 3; /* сколько тегов знает эта версия */
size_t i = 0;
*m = (meas_t){0};
while (i < len) {
if (len - i < 2 || p[i + 1] > len - i - 2)
return -1; /* длина выходит за конец сообщения */
uint8_t tag = p[i], l = p[i + 1];
const uint8_t *v = p + i + 2;
i += 2u + l;
if (tag < 1 || tag > last) { /* чужой тег: длина известна — просто перешагнуть */
skipped++;
continue;
}
if (l < 1 || l > 4)
return -2;
uint32_t x = (v[0] & 0x80) ? 0xFFFFFFFFu : 0; /* расширить знак */
for (uint8_t j = 0; j < l; j++)
x = (x << 8) | v[j];
if (tag == 1) m->n = x;
if (tag == 2) m->t = (int32_t)x;
if (tag == 3) m->u = x;
if (tag == 4) m->rh = x;
}
return skipped;
}
/* ════ protobuf ════ */
size_t pb_key(uint8_t *p, uint32_t field, uint32_t wt)
{
return put_varint(p, ((uint64_t)field << 3) | wt);
}
static size_t pb_uint(uint8_t *p, uint32_t field, uint64_t v)
{
if (v == 0)
return 0; /* proto3: значение по умолчанию не передаётся */
size_t k = pb_key(p, field, WT_VARINT);
return k + put_varint(p + k, v);
}
size_t meas_pb_encode(const meas_t *m, int version, uint8_t *out)
{
size_t k = 0;
k += pb_uint(out + k, 1, m->n);
k += pb_uint(out + k, 2, zigzag32(m->t)); /* sint32: −1 → 1, 1 → 2, −2000 → 3999 */
k += pb_uint(out + k, 3, m->u);
if (version >= 2)
k += pb_uint(out + k, 4, m->rh);
return k;
}
size_t meas_pb_encode_int32(const meas_t *m, uint8_t *out)
{
size_t k = 0;
k += pb_uint(out + k, 1, m->n);
k += pb_uint(out + k, 2, (uint64_t)(int64_t)m->t); /* int32: отрицательное расширяется до 64 бит */
k += pb_uint(out + k, 3, m->u);
return k;
}
static uint32_t float_bits(float f)
{
union { float f; uint32_t u; } c = {f};
return c.u;
}
size_t meas_pb_encode_float(const meas_t *m, uint8_t *out)
{
size_t k = pb_uint(out, 1, m->n);
float deg = (float)m->t / 100.0f;
if (deg != 0.0f) {
k += pb_key(out + k, 2, WT_I32);
uint32_t b = float_bits(deg);
for (int i = 0; i < 4; i++)
out[k++] = (uint8_t)(b >> (8 * i)); /* fixed32 и float — младшим байтом вперёд */
}
k += pb_uint(out + k, 3, m->u);
return k;
}
int meas_pb_decode(const uint8_t *p, size_t len, int version, meas_t *m)
{
uint32_t last = version >= 2 ? 4 : 3;
int skipped = 0;
size_t i = 0;
*m = (meas_t){0};
while (i < len) {
uint64_t key, v = 0;
size_t k = get_varint(p + i, len - i, &key);
if (!k)
return -1;
i += k;
uint32_t field = (uint32_t)(key >> 3), wt = (uint32_t)(key & 7);
if (wt == WT_VARINT) {
k = get_varint(p + i, len - i, &v);
if (!k)
return -1;
i += k;
} else if (wt == WT_I64 || wt == WT_I32) {
size_t w = wt == WT_I64 ? 8 : 4;
if (len - i < w)
return -1;
i += w;
} else if (wt == WT_LEN) {
k = get_varint(p + i, len - i, &v);
if (!k || v > len - i - k)
return -1;
i += k + (size_t)v;
} else {
return -3; /* типы 3, 4, 6, 7 — группы и мусор */
}
if (field >= 1 && field <= last && wt != WT_VARINT)
return -2; /* знакомое поле пришло с другим типом провода */
if (field == 1) m->n = (uint32_t)v;
else if (field == 2) m->t = unzigzag32((uint32_t)v);
else if (field == 3) m->u = (uint32_t)v;
else if (field == 4 && last >= 4) m->rh = (uint32_t)v;
else skipped++; /* чужое поле: тип провода говорит, сколько перешагнуть */
}
return skipped;
}
/* ════ CBOR ════ */
size_t cbor_head(uint8_t *p, uint8_t major, uint64_t v)
{
uint8_t mt = (uint8_t)(major << 5);
if (v < 24) {
p[0] = mt | (uint8_t)v; /* число прямо в первом байте */
return 1;
}
int n = v <= 0xFF ? 1 : v <= 0xFFFF ? 2 : v <= 0xFFFFFFFFu ? 4 : 8;
p[0] = mt | (uint8_t)(n == 1 ? 24 : n == 2 ? 25 : n == 4 ? 26 : 27);
for (int i = 0; i < n; i++)
p[1 + i] = (uint8_t)(v >> (8 * (n - 1 - i))); /* старшим вперёд */
return 1u + (size_t)n;
}
size_t cbor_int(uint8_t *p, int64_t v)
{
return v >= 0 ? cbor_head(p, 0, (uint64_t)v) : cbor_head(p, 1, (uint64_t)(-1 - v)); /* −1 → 0x20 */
}
size_t cbor_tstr(uint8_t *p, const char *s)
{
size_t n = 0;
while (s[n])
n++;
size_t k = cbor_head(p, 3, n);
for (size_t i = 0; i < n; i++)
p[k + i] = (uint8_t)s[i];
return k + n;
}
size_t cbor_float(uint8_t *p, float f)
{
uint32_t b = float_bits(f);
uint32_t s = b >> 31, e = (b >> 23) & 0xFF, m = b & 0x7FFFFF;
int he = (int)e - 127 + 15;
if ((b & 0x7FFFFFFF) == 0 || (e != 0xFF && he >= 1 && he <= 30 && (m & 0x1FFF) == 0)) {
uint16_t h = (uint16_t)(s << 15);
if (b & 0x7FFFFFFF)
h |= (uint16_t)((uint32_t)he << 10 | m >> 13);
p[0] = 0xF9; /* половинная точность: 2 байта без потерь */
p[1] = (uint8_t)(h >> 8);
p[2] = (uint8_t)h;
return 3;
}
p[0] = 0xFA; /* иначе одинарная: 4 байта */
for (int i = 0; i < 4; i++)
p[1 + i] = (uint8_t)(b >> (8 * (3 - i)));
return 5;
}
size_t meas_cbor_map(const meas_t *m, int version, uint8_t *out)
{
size_t k = cbor_head(out, 5, version >= 2 ? 4 : 3);
k += cbor_int(out + k, 1); k += cbor_int(out + k, m->n);
k += cbor_int(out + k, 2); k += cbor_int(out + k, m->t);
k += cbor_int(out + k, 3); k += cbor_int(out + k, m->u);
if (version >= 2) {
k += cbor_int(out + k, 4); k += cbor_int(out + k, m->rh);
}
return k;
}
size_t meas_cbor_array(const meas_t *m, uint8_t *out)
{
size_t k = cbor_head(out, 4, 3);
k += cbor_int(out + k, m->n);
k += cbor_int(out + k, m->t);
k += cbor_int(out + k, m->u);
return k;
}
size_t meas_cbor_smap(const meas_t *m, uint8_t *out)
{
size_t k = cbor_head(out, 5, 3);
k += cbor_tstr(out + k, "n"); k += cbor_int(out + k, m->n);
k += cbor_tstr(out + k, "t"); k += cbor_int(out + k, m->t);
k += cbor_tstr(out + k, "u"); k += cbor_int(out + k, m->u);
return k;
}
size_t meas_cbor_map_float(const meas_t *m, uint8_t *out)
{
size_t k = cbor_head(out, 5, 3);
k += cbor_int(out + k, 1); k += cbor_int(out + k, m->n);
k += cbor_int(out + k, 2); k += cbor_float(out + k, (float)m->t / 100.0f);
k += cbor_int(out + k, 3); k += cbor_int(out + k, m->u);
return k;
}
/* разбор: заголовок элемента */
static int rd_head(const uint8_t *p, size_t len, size_t *i, uint8_t *major, uint64_t *v, int *indef)
{
if (*i >= len)
return 0;
uint8_t b = p[(*i)++], ai = b & 31;
*major = b >> 5;
*indef = 0;
*v = 0;
if (ai < 24) {
*v = ai;
return 1;
}
if (ai == 31) { /* неопределённая длина: до байта 0xFF */
*indef = 1;
return 1;
}
if (ai > 27)
return 0;
size_t n = (size_t)1 << (ai - 24);
if (len - *i < n)
return 0;
for (size_t j = 0; j < n; j++)
*v = (*v << 8) | p[(*i)++];
return 1;
}
static int skip_item(const uint8_t *p, size_t len, size_t *i, int depth)
{
uint8_t mt;
uint64_t v;
int indef;
if (depth > 8 || !rd_head(p, len, i, &mt, &v, &indef))
return 0;
if (mt <= 1 || (mt == 7 && !indef))
return 1; /* числа, простые значения и float — только заголовок */
if (mt == 2 || mt == 3) {
if (indef || v > len - *i)
return 0;
*i += (size_t)v;
return 1;
}
if (mt == 6)
return skip_item(p, len, i, depth + 1); /* тег + одно значение */
uint64_t items = mt == 5 ? 2 * v : v;
if (indef) {
while (*i < len && p[*i] != 0xFF)
if (!skip_item(p, len, i, depth + 1))
return 0;
return *i < len ? (++*i, 1) : 0;
}
for (uint64_t j = 0; j < items; j++)
if (!skip_item(p, len, i, depth + 1))
return 0;
return 1;
}
int meas_cbor_decode(const uint8_t *p, size_t len, int version, meas_t *m)
{
uint64_t last = version >= 2 ? 4 : 3;
size_t i = 0;
uint8_t mt;
uint64_t cnt, key, v;
int indef, skipped = 0;
*m = (meas_t){0};
if (!rd_head(p, len, &i, &mt, &cnt, &indef) || mt != 5)
return -1;
for (uint64_t j = 0; indef ? (i < len && p[i] != 0xFF) : j < cnt; j++) {
int ki;
if (!rd_head(p, len, &i, &mt, &key, &ki))
return -1; /* обрыв */
if (mt != 0)
return -2; /* ключи в этой схеме — только неотрицательные числа */
if (key < 1 || key > last) {
if (!skip_item(p, len, &i, 0))
return -1;
skipped++;
continue;
}
if (!rd_head(p, len, &i, &mt, &v, &ki))
return -1;
if (mt > 1)
return -2;
int64_t x = mt == 0 ? (int64_t)v : -1 - (int64_t)v;
if (key == 1) m->n = (uint32_t)x;
if (key == 2) m->t = (int32_t)x;
if (key == 3) m->u = (uint32_t)x;
if (key == 4) m->rh = (uint32_t)x;
}
if (indef) {
if (i >= len)
return -1;
i++; /* байт 0xFF — конец словаря */
}
return i == len ? skipped : -1;
}
meas.pb.h — the message descriptions for nanopb in the form the generator writes them.
/* Раздел 115г: описание сообщений meas.proto для nanopb 0.4.9 — в том виде, в каком его пишет
* генератор nanopb_generator.py. Генератору нужен protoc, а его на машине нет, поэтому файл
* набран вручную; правильность сверяет сама nanopb (PB_BIND проверяет размеры и смещения полей
* при компиляции) и опыт: байты nanopb = байты своего кодировщика = байты python protobuf. */
#ifndef PB_MEAS_PB_H_INCLUDED
#define PB_MEAS_PB_H_INCLUDED
#include <pb.h>
#if PB_PROTO_HEADER_VERSION != 40
#error Regenerate this file with the current version of nanopb generator.
#endif
/* Struct definitions */
typedef struct _Measurement {
uint32_t n;
int32_t t;
uint32_t u;
} Measurement;
typedef struct _Measurement2 {
uint32_t n;
int32_t t;
uint32_t u;
uint32_t rh;
} Measurement2;
typedef struct _MeasurementI {
uint32_t n;
int32_t t;
uint32_t u;
} MeasurementI;
typedef struct _MeasurementU {
uint32_t n;
uint32_t t;
uint32_t u;
} MeasurementU;
typedef struct _MeasurementF {
uint32_t n;
float t;
uint32_t u;
} MeasurementF;
/* Initializer values for message structs */
#define Measurement_init_zero {0, 0, 0}
#define Measurement2_init_zero {0, 0, 0, 0}
#define MeasurementI_init_zero {0, 0, 0}
#define MeasurementU_init_zero {0, 0, 0}
#define MeasurementF_init_zero {0, 0, 0}
/* Struct field encoding specification for nanopb */
#define Measurement_FIELDLIST(X, a) \
X(a, STATIC, SINGULAR, UINT32, n, 1) \
X(a, STATIC, SINGULAR, SINT32, t, 2) \
X(a, STATIC, SINGULAR, UINT32, u, 3)
#define Measurement_CALLBACK NULL
#define Measurement_DEFAULT NULL
#define Measurement2_FIELDLIST(X, a) \
X(a, STATIC, SINGULAR, UINT32, n, 1) \
X(a, STATIC, SINGULAR, SINT32, t, 2) \
X(a, STATIC, SINGULAR, UINT32, u, 3) \
X(a, STATIC, SINGULAR, UINT32, rh, 4)
#define Measurement2_CALLBACK NULL
#define Measurement2_DEFAULT NULL
#define MeasurementI_FIELDLIST(X, a) X(a, STATIC, SINGULAR, UINT32, n, 1) X(a, STATIC, SINGULAR, INT32, t, 2) X(a, STATIC, SINGULAR, UINT32, u, 3)
#define MeasurementI_CALLBACK NULL
#define MeasurementI_DEFAULT NULL
#define MeasurementU_FIELDLIST(X, a) \
X(a, STATIC, SINGULAR, UINT32, n, 1) \
X(a, STATIC, SINGULAR, UINT32, t, 2) \
X(a, STATIC, SINGULAR, UINT32, u, 3)
#define MeasurementU_CALLBACK NULL
#define MeasurementU_DEFAULT NULL
#define MeasurementF_FIELDLIST(X, a) \
X(a, STATIC, SINGULAR, UINT32, n, 1) \
X(a, STATIC, SINGULAR, FLOAT, t, 2) \
X(a, STATIC, SINGULAR, UINT32, u, 3)
#define MeasurementF_CALLBACK NULL
#define MeasurementF_DEFAULT NULL
extern const pb_msgdesc_t Measurement_msg;
extern const pb_msgdesc_t Measurement2_msg;
extern const pb_msgdesc_t MeasurementI_msg;
extern const pb_msgdesc_t MeasurementU_msg;
extern const pb_msgdesc_t MeasurementF_msg;
#define Measurement_fields &Measurement_msg
#define Measurement2_fields &Measurement2_msg
#define MeasurementI_fields &MeasurementI_msg
#define MeasurementU_fields &MeasurementU_msg
#define MeasurementF_fields &MeasurementF_msg
/* Maximum encoded size of messages (where known) */
#define Measurement_size 18
#define Measurement2_size 24
#define MeasurementI_size 23
#define MeasurementU_size 18
#define MeasurementF_size 17
#endif
Next: section 116 "Typical approaches and algorithms: from a threshold to a small neural network".