198б. Mini-project: a USB data logger and a PC program

The device in this project is a data logger: an F407 measures two ADC channels a thousand times a second and continuously streams the data to a computer, where a program receives it, checks it and writes a file with a plot. The computer sees the device as an ordinary serial port — the CDC class from section 20. What is new here is not USB (its mechanics have already been covered) but everything between the measurement and the file: a binary frame with an integrity check, a queue for the receiver's pauses, and a PC program that honestly counts losses.

QEMU is no help here: it does not model USB OTG — neither enumeration nor endpoints. So the project is cut along a seam that can be checked without a board (the technique of sections 184 and 196): the frame and the queue are shared code, logger.c, without a single register, built both for Cortex-M4 and with ordinary gcc; a generator drives the stream of frames through a pipe into the receiver. The files are in _verify/usb198b/: logger.h, logger.c, sim_queue.c, gen_frames.py, logger_rx.py; the runs are recorded in _verify/usb198b/*_host.txt, checked by _verify/usb198b_check.py.

Terms used in the documentation

Term	Meaning
data logger	a device that records a stream of measurements
payload	the frame's data without the header and CRC
magic number	distinctive bytes that mark the start of a frame
sequence number	the frame's serial number: gaps in it reveal losses
throughput	how many bytes per second get through
goodput	the useful part of it — without headers, CRC and retries
drop-oldest / drop-newest	evict the old / discard the new on overflow

198б.1 What the device does and who is responsible for what

Two decisions define the whole architecture. The firmware stamps the time: frames reach the program in batches and with a delay — the host polls the device on its own schedule (section 20), and the receiver reads the port when the operating system lets it. So every frame carries t_ms from SysTick, and plots are built only from it, not from the PC clock. And the firmware buffers: there is no return channel, nobody to ask to "repeat" — a receiver pause can be survived only by the queue in the device's RAM (198б.3).

There are more than enough endpoints: CDC needs three besides endpoint zero — bulk IN and OUT for data and an interrupt IN for notifications. OTG FS in the F407 provides four IN and four OUT, including endpoint zero (the header stm32f407xx.h from the pack: USB_OTG_FS_MAX_IN_ENDPOINTS 4). The endpoint buffers share 1280 bytes of FIFO (USB_OTG_FS_TOTAL_FIFO_SIZE), that is 320 words, which the firmware divides between the receive FIFO and the transmit FIFOs; any reasonable split is enough for 20-byte frames.

198б.2 The frame: twenty bytes, byte by byte

A text format ("t=1000 ch0=-123…") would be two to three times fatter and would require string parsing; a fixed-size binary frame is simpler for both the device and the program. Fields are little-endian: that is how Cortex-M writes them to memory, and the PC almost everywhere reads them the same way (byte order — section 3). The frame is assembled in logger.c literally byte by byte, without structures with alignment: the format is visible to the eye and does not depend on the compiler.

The eviction counter is cumulative, not "since the previous frame": an evicted frame takes its own field with it, and per-frame counters would be lost along with it. The difference of the cumulative counter in the first and last received frames always tells how many frames the queue evicted — no matter how many frames went missing in between.

The CRC closes the first 16 bytes — exactly four words, the way the STM32 hardware unit consumes them (section 43): words from memory, without byte reversal. On the computer the same thing takes about ten lines:

uint32_t lg_crc32_words(const uint8_t *p, uint32_t n)
{
    uint32_t crc = 0xFFFFFFFFu;
    for (uint32_t i = 0; i < n; i += 4) {
        /* слово так, как его прочитал бы процессор из памяти (младшим вперёд) */
        crc ^= (uint32_t)p[i] | (uint32_t)p[i + 1] << 8 |
               (uint32_t)p[i + 2] << 16 | (uint32_t)p[i + 3] << 24;
        for (int b = 0; b < 32; b++)
            crc = (crc & 0x80000000u) ? (crc << 1) ^ 0x04C11DB7u : crc << 1;
    }
    return crc;
}

⚠️ Check the implementation against the numbers of section 43 before running frames. The model prints three check values: bytewise over 123456789 it gives 0x0376E6E7 (CRC-32/MPEG-2), wordwise over 12345678 — 0xFEFC54F9, and the same number comes from a bytewise run over the string 43218765 — the byte order within a word is reversed, exactly as described there. If all three match, the firmware can write CRC->DR without __REV, and the PC can compute with this function.

The counters are deliberately narrow. The frame number is 16 bits: it wraps after 65536 frames, at 1 kHz — every 65.5 s; that is enough for the receiver, because it counts gaps as a difference modulo 2¹⁶. The time is 32 bits of milliseconds: it wraps after 49.7 days of continuous recording.

198б.3 The queue: evict the old or stop

The queue is a ring of 32 frames (640 bytes of buffers; the whole structure with counters is 652 bytes, as printed by the model). At 1 kHz that is a cushion of 32 ms of receiver silence — the usual pauses of the scheduler and of disk writes it absorbs silently. Things get interesting when the pause is longer. There are two options, and both lose data — the only question is which:

• "stop" (drop-newest): the queue is full — the new frame is discarded. The start of the pause survives, its end is lost — the freshest data;
• "new evicts old" (drop-oldest): full — the oldest frame is thrown out and the new one goes in. The fresh data survives.

Both policies sit side by side in the core (lg_push and lg_push_stop) so that the model can pit them against each other. Eviction is three lines at the start of lg_push; at the same time the flag is set and the cumulative counter grows:

void lg_push(struct lg_queue *q, uint32_t t_ms, int16_t ch0, int16_t ch1)
{
    uint8_t evicted = 0;
    if (q->count == LG_DEPTH) {                    /* вытеснить самый старый */
        q->head = (uint16_t)((q->head + 1) % LG_DEPTH);
        q->count--;
        q->drops_total++;
        evicted = 1;
    }
    uint16_t tail = (uint16_t)((q->head + q->count) % LG_DEPTH);
    lg_build(q, q->buf[tail], t_ms, ch0, ch1, evicted);
    q->count++;
}

With "stop" there is a subtlety: the frame is not created, but its number is used up — otherwise the receiver would not see a gap, and the loss would go unnoticed.

The model sim_queue.c runs 10000 ms: the producer puts a frame every millisecond, the consumer takes up to four frames per millisecond, but falls silent on [3000; 3250) ms — for a quarter of a second the host is busy with something of its own. Both policies, the very same run:

The losses are equal — more frames were born during the pause than fit, and the extra ones perished under either policy. The difference is in which survived: after "stop" the record breaks off at number 3030 — the event that made the host hang is missing from the log; with eviction the gap lies at the start of the pause, and the last 250 ms are in place. A data logger almost always needs the second. And note: 219 frames were lost, not 218 — the queue met the pause not empty (frame 2999 was still waiting to be sent), so there were 31 free slots. The books balance: the sum of the gaps in the numbers of the received frames equals the cumulative counter in the last one — 219 in both, account=ok.

The core is also built for the firmware: arm-none-eabi-gcc -mcpu=cortex-m4 -Os gives 504 bytes of code for the whole of logger.c — frame assembly, CRC and both policies; the core has no data of its own, the queue lives with its owner. On the board lg_push is called from the ADC interrupt, and the main loop uses lg_pop to feed frames to CDC_Transmit_FS (recipe 20.7): while the class answers "busy", the next frame is simply not taken from the queue.

198б.4 🔧 The receiver on the PC: magic, CRC, loss accounting

From the computer's side the device is a serial port, and pyserial can read it from Python: serial.Serial("COM5") and port.read(4096) in a loop. Everything interesting is in parsing what has been read.

⚠️ The port returns chunks of arbitrary size. USB carries packets of 64 bytes, the operating system glues them into a buffer, and read returns whatever has accumulated: one and a half frames, a third of a frame, ten frames. Frame boundaries exist nowhere except in the format itself — so the parser must be stream-based: receive a chunk, append it to the tail, take out whole frames, and keep the remainder until the next chunk.

Parsing is the feed method of the Parser class in logger_rx.py; it does not care where the chunk came from — a port, a file or a pipe:

    def feed(self, chunk):
        self.buf += chunk
        while True:
            i = self.buf.find(MAGIC)
            if i < 0:                          # сигнатуры нет — всё, кроме
                keep = max(0, len(self.buf) - 1)   # последнего байта, мусор
                self.bytes_skipped += keep
                self.buf = self.buf[keep:]
                return
            self.bytes_skipped += i
            self.buf = self.buf[i:]
            if len(self.buf) < FRAME:          # кадр ещё не дошёл целиком
                return
            cand = self.buf[:FRAME]
            if crc32_words(cand[:16]) == struct.unpack("<I", cand[16:])[0]:
                self._accept(cand)
                self.buf = self.buf[FRAME:]
            else:                              # ложная сигнатура или порча
                self.crc_errors += 1
                self.bytes_skipped += 1
                self.buf = self.buf[1:]

The order is mandatory: the magic is only a candidate for a frame, the CRC confirms it. The two bytes 0xA5 0x5A also occur inside the payload and in garbage; the CRC of a false candidate will not match, the parser shifts by one byte and continues the search — that is recovery after a failure. A tail shorter than a frame is left alone: the next read will deliver the rest.

Received frames are counted in three columns: frames — how many arrived, lost_by_seq — the sum of the gaps in the numbers, drops_accounted — the difference of the cumulative eviction counter between the last and the first frame. The difference lost_by_seq − drops_accounted is the losses not caused by the queue: corruption on the line, parsing errors. The CSV is written in columns t_ms,ch0,ch1 — the plot can be built with anything, from a spreadsheet to matplotlib.

198б.5 Throughput: the ceiling and the margins

How much can CDC carry at full speed at all? It is counted honestly, from the bus frame (not to be confused with the logger's frame): at 12 Mbit/s 1500 "raw" bytes pass in a one-millisecond bus frame. A bulk transaction with a 64-byte data field costs 77 bytes: 13 bytes of overhead — one SYNC byte and one PID byte for each of the three packets (token, data, handshake), 2 bytes of address with CRC5, 2 bytes of CRC16 for the data field and about three bytes of inter-packet gaps. A bus frame holds 1500 / 77 = 19 transactions — the ceiling for bulk transfers:

19 × 64 = 1216 bytes per millisecond ≈ 1.2 MB/s.

This is the ceiling for the whole bus: it is shared between devices on the same controller and cut by other traffic; a real stack will not even come close to it. But the scale is clear at once:

Two conclusions follow. First: at these rates USB does not even break a sweat — it gets tight at tens of kilohertz. Second: the backup UART channel (section 16) at the usual 115200 baud cannot carry our stream — a byte on the wire costs 10 bits, and 11520 B/s is less than 20000; you need either a port twice as fast or decimation. Meanwhile the payload in a frame is 4 bytes out of 20 — 80 % overhead; what to do about that is in 198б.7.

198б.6 A run without a board: the generator drives the stream into the receiver

gen_frames.py assembles the same frames as logger.c — the model printed the reference frame A5 5A 01 00 00 00 E8 03 00 00 85 FF C8 01 00 00 90 E0 61 E8, and on the same data the generator must reproduce it byte for byte (the check compares them) — and feeds them through a pipe to logger_rx.py, as a port would.

Scenario one — a clean channel, 10000 frames in a row:

frames=10000 crc_errors=0 lost_by_seq=0 drops_accounted=0 lost_unexplained=0 bytes_skipped=0

No losses, no CRC errors, no discarded bytes; the CSV is 10001 lines including the header.

Scenario two — a dirty channel: 157 bytes of garbage in two insertions (inside one of them a deliberately planted false magic A5 5A), five frames "evicted by the queue" (numbers skipped, the cumulative counter moved by 5), and in three frames a payload byte corrupted after the CRC was computed:

frames=9992 crc_errors=4 lost_by_seq=8 drops_accounted=5 lost_unexplained=3 bytes_skipped=217

Everything adds up to the byte. Delivered 9992 = 10000 − 5 evicted − 3 corrupted. There are four CRC errors: three corruptions plus the false magic in the garbage. There are eight gaps in the numbers: five are explained by the eviction counter from the frames themselves, three remain "on the line" — exactly the corrupted ones. 217 bytes were discarded: 157 of garbage and 60 bytes of the three frames that failed the CRC. The receiver not only survived the dirt — it reported whose loss was whose.

198б.7 What to improve

• Batches of samples: put not one pair of channels into a frame but 24 pairs — a frame of 112 bytes, and the useful share grows from 20 % to 86 %. The format is ready: a different frame type.
• Commands to the device: CDC already has an OUT channel, and receiving strings is covered in recipe 20.7; "start", "stop", "rate 5000".
• High Speed: the second F407 controller is OTG HS with a 4096-byte FIFO and six IN endpoints (the pack header), but full HS speed needs an external physical-layer chip over ULPI.
• A file instead of a port: the MSC class (section 20.4) plus a memory card (sections 41–42) — a logger without a computer; the queue and frames stay the same.
• Sleep between samples: WFI in the pauses of the main loop (section 10), the ADC triggered by a timer, wakeup on an interrupt.

198б.8 Pitfalls

The baud rate set when opening the port has no effect. That is by design: in CDC the "baud rate" is a request from the old world; it never reaches any wires, the bytes travel in USB packets. The real arithmetic is in 198б.5.

The CRC "does not match the computer's". The hardware unit consumes words least-significant byte first (section 43). Either the firmware feeds CRC->DR through __REV, or — as here — the PC computes by words; the check: 12345678 by words gives 0xFEFC54F9.

Every other frame is "corrupted". The receiver believes that read returns a whole frame. It does not: there are no frame boundaries in the port, a frame is cut at any byte. Stream parsing with a tail is the only honest way.

The record breaks off a moment before the event. The "stop" policy: while the host was silent, the queue kept the old frames and threw away the fresh ones — the event itself. A data logger needs eviction of the old (the run in 198б.3).

Losses grow although there are no pauses. If lost_by_seq is ahead of drops_accounted, the queue is not to blame — frames die on the way: the cable, the hub's power, a race in writing the frame in the firmware. Queue depth will not help.

The port is "busy" or the device "is not recognized". The port is held by a forgotten terminal — CDC is exclusive; and "not recognized" on a home-made board is almost always the clocking and the surrounding circuitry from section 20, not this project.

198б.9 Summary

• The frame is 20 bytes: magic 0xA5 0x5A, type, flags, number, t_ms, two channels, the cumulative eviction counter and a CRC-32 over the first 16 bytes — exactly four words of the hardware unit.
• The firmware stamps the time and the firmware buffers: a queue of 32 frames (652 bytes with buffers and counters) absorbs a receiver pause of up to 32 ms.
• A 250 ms pause costs 219 frames under either policy; the choice of policy decides which ones survive: "stop" loses the end of the pause, eviction loses the start. A data logger should evict the old.
• The ceiling for full-speed bulk transfers is 19 transactions of 64 bytes per bus frame: 1216000 B/s. A stream of 1 kHz × 20 bytes = 20000 B/s — a margin of 60.8 times; UART at 115200 cannot carry such a stream (11520 B/s), 230400 can, with a margin of 15.2 %.
• Receiver: the magic is a candidate, the CRC confirms; 10000 frames through a pipe — zero losses and errors. With dirt: 8 gaps = 5 evicted (per the counter in the frames) + 3 corrupted; 217 bytes discarded.
• The logger core on Cortex-M4 is 504 bytes of code; CRC check: 0x0376E6E7 bytewise over 123456789, 0xFEFC54F9 wordwise over 12345678.

Full source code

logger.c — frame assembly and the queue.

/* logger.c — сборка кадра и очередь. Собирается arm-none-eabi-gcc для
 * Cortex-M4 (прошивка) и обычным gcc (модель на компьютере) без правок. */
#include "logger.h"

uint32_t lg_crc32_words(const uint8_t *p, uint32_t n)
{
    uint32_t crc = 0xFFFFFFFFu;
    for (uint32_t i = 0; i < n; i += 4) {
        /* слово так, как его прочитал бы процессор из памяти (младшим вперёд) */
        crc ^= (uint32_t)p[i] | (uint32_t)p[i + 1] << 8 |
               (uint32_t)p[i + 2] << 16 | (uint32_t)p[i + 3] << 24;
        for (int b = 0; b < 32; b++)
            crc = (crc & 0x80000000u) ? (crc << 1) ^ 0x04C11DB7u : crc << 1;
    }
    return crc;
}

void lg_init(struct lg_queue *q)
{
    q->head = 0; q->count = 0; q->seq = 0; q->drops_total = 0;
}

/* Кадр целиком, байт за байтом; flags — бит 0: перед кадром были вытеснения */
static void lg_build(struct lg_queue *q, uint8_t *f, uint32_t t_ms,
                     int16_t ch0, int16_t ch1, uint8_t flags)
{
    uint16_t drops16 = (uint16_t)q->drops_total;    /* младшие 16 бит */
    f[0]  = LG_MAGIC0;
    f[1]  = LG_MAGIC1;
    f[2]  = LG_TYPE;
    f[3]  = flags;
    f[4]  = (uint8_t)(q->seq);          f[5]  = (uint8_t)(q->seq >> 8);
    f[6]  = (uint8_t)(t_ms);            f[7]  = (uint8_t)(t_ms >> 8);
    f[8]  = (uint8_t)(t_ms >> 16);      f[9]  = (uint8_t)(t_ms >> 24);
    f[10] = (uint8_t)((uint16_t)ch0);   f[11] = (uint8_t)((uint16_t)ch0 >> 8);
    f[12] = (uint8_t)((uint16_t)ch1);   f[13] = (uint8_t)((uint16_t)ch1 >> 8);
    f[14] = (uint8_t)(drops16);         f[15] = (uint8_t)(drops16 >> 8);
    uint32_t crc = lg_crc32_words(f, 16);
    f[16] = (uint8_t)(crc);             f[17] = (uint8_t)(crc >> 8);
    f[18] = (uint8_t)(crc >> 16);       f[19] = (uint8_t)(crc >> 24);
    q->seq++;
}

void lg_push(struct lg_queue *q, uint32_t t_ms, int16_t ch0, int16_t ch1)
{
    uint8_t evicted = 0;
    if (q->count == LG_DEPTH) {                    /* вытеснить самый старый */
        q->head = (uint16_t)((q->head + 1) % LG_DEPTH);
        q->count--;
        q->drops_total++;
        evicted = 1;
    }
    uint16_t tail = (uint16_t)((q->head + q->count) % LG_DEPTH);
    lg_build(q, q->buf[tail], t_ms, ch0, ch1, evicted);
    q->count++;
}

void lg_push_stop(struct lg_queue *q, uint32_t t_ms, int16_t ch0, int16_t ch1)
{
    if (q->count == LG_DEPTH) {                    /* места нет — кадр пропал */
        q->drops_total++;
        q->seq++;                    /* номер расходуется: приёмник увидит дыру */
        return;
    }
    uint16_t tail = (uint16_t)((q->head + q->count) % LG_DEPTH);
    lg_build(q, q->buf[tail], t_ms, ch0, ch1, 0);
    q->count++;
}

int lg_pop(struct lg_queue *q, uint8_t out[LG_FRAME])
{
    if (q->count == 0)
        return 0;
    const uint8_t *f = q->buf[q->head];
    for (uint32_t i = 0; i < LG_FRAME; i++)
        out[i] = f[i];
    q->head = (uint16_t)((q->head + 1) % LG_DEPTH);
    q->count--;
    return 1;
}

logger_rx.py — the receiver on the computer: magic, CRC, loss accounting, CSV.

# -*- coding: utf-8 -*-
"""logger_rx.py — приёмник самописца: сигнатура -> CRC -> счёт потерь -> CSV.

python logger_rx.py [--csv файл]        поток кадров ждёт на stdin

С прибором вместо stdin открывается порт (pyserial):
    port = serial.Serial("COM5")        # скорость у CDC ни на что не влияет
и вместо read() у файла зовётся port.read(4096) в том же цикле.
"""
import struct
import sys

FRAME = 20
MAGIC = b"\xA5\x5A"


def crc32_words(data):
    """Тот же CRC, что считает блок STM32 и logger.c: по словам, без переворота."""
    crc = 0xFFFFFFFF
    for i in range(0, len(data), 4):
        crc ^= int.from_bytes(data[i:i + 4], "little")
        for _ in range(32):
            crc = ((crc << 1) ^ 0x04C11DB7 if crc & 0x80000000 else crc << 1) & 0xFFFFFFFF
    return crc


class Parser:
    """Потоковый разбор: feed() принимает куски любого размера."""

    def __init__(self, on_frame):
        self.buf = b""
        self.on_frame = on_frame
        self.frames = 0
        self.crc_errors = 0
        self.bytes_skipped = 0
        self.lost_by_seq = 0
        self.prev_seq = None
        self.first_field = None
        self.last_field = 0

    def feed(self, chunk):
        self.buf += chunk
        while True:
            i = self.buf.find(MAGIC)
            if i < 0:                          # сигнатуры нет — всё, кроме
                keep = max(0, len(self.buf) - 1)   # последнего байта, мусор
                self.bytes_skipped += keep
                self.buf = self.buf[keep:]
                return
            self.bytes_skipped += i
            self.buf = self.buf[i:]
            if len(self.buf) < FRAME:          # кадр ещё не дошёл целиком
                return
            cand = self.buf[:FRAME]
            if crc32_words(cand[:16]) == struct.unpack("<I", cand[16:])[0]:
                self._accept(cand)
                self.buf = self.buf[FRAME:]
            else:                              # ложная сигнатура или порча
                self.crc_errors += 1
                self.bytes_skipped += 1
                self.buf = self.buf[1:]

    def _accept(self, f):
        (_, _, _, _, seq, t_ms, ch0, ch1, field) = struct.unpack("<BBBBHIhhH", f[:16])
        if self.prev_seq is not None:
            self.lost_by_seq += (seq - self.prev_seq - 1) & 0xFFFF
        self.prev_seq = seq
        if self.first_field is None:
            self.first_field = field
        self.last_field = field
        self.frames += 1
        self.on_frame(t_ms, ch0, ch1)

    def report(self):
        accounted = 0
        if self.first_field is not None:
            accounted = (self.last_field - self.first_field) & 0xFFFF
        return ("frames=%d crc_errors=%d lost_by_seq=%d drops_accounted=%d "
                "lost_unexplained=%d bytes_skipped=%d"
                % (self.frames, self.crc_errors, self.lost_by_seq, accounted,
                   self.lost_by_seq - accounted, self.bytes_skipped))


def main():
    csv = None
    if len(sys.argv) >= 3 and sys.argv[1] == "--csv":
        csv = open(sys.argv[2], "w", encoding="utf-8")
        csv.write("t_ms,ch0,ch1\n")

    def on_frame(t_ms, ch0, ch1):
        if csv:
            csv.write("%d,%d,%d\n" % (t_ms, ch0, ch1))

    p = Parser(on_frame)
    src = sys.stdin.buffer
    while True:
        chunk = src.read(4096)      # порция произвольная: границ кадров нет
        if not chunk:
            break
        p.feed(chunk)
    if csv:
        csv.close()
    print(p.report())


if __name__ == "__main__":
    main()

Next: section 198в "Mini-project: a Modbus RTU I/O module" — the logger talked in one direction; now the device will start answering requests.