All sections › XII. Networks and DevOps for the engineer

STM32F103, STM32F407 · level intermediate · 12 min read

110. CI/CD for firmware: build in a container, tests, artifacts

"It builds on my machine" is the most expensive phrase in firmware development. It builds on one computer, with one compiler version, from a tree where something is not saved. A pipeline takes the human out of this chain: the build runs in a container with pinned versions, tests run by themselves, and the output is an image with the version from the tag and a checksum. This section is not about the buttons of a particular service but about how it works: what a pipeline must do, why repeatability matters more than speed and what must stop a release.

The experiments are a real pipeline in a container: arm-none-eabi-gcc 12.2.1 builds firmware for F103 and F407, the same source is checked by tests on the computer, and the artifacts are stored with a manifest.

Terms used in the documentation

Term Meaning

continuous integration (CI) every commit is built and checked

continuous delivery (CD) a ready image after a successful check

pipeline, stage, job the whole chain, one step of it, one task within a step

runner, agent the machine where the build runs

artifact a file the pipeline hands out

build matrix one build for several boards or settings

reproducible build the same input gives a byte-for-byte identical image

pinned version a fixed version of a tool

red build the pipeline has failed

manifest, checksum a list of outputs, a control sum of a file

cache files kept between runs

Term	Meaning
continuous integration (CI)	every commit is built and checked
continuous delivery (CD)	a ready image after a successful check
pipeline, stage, job	the whole chain, one step of it, one task within a step
runner, agent	the machine where the build runs
artifact	a file the pipeline hands out
build matrix	one build for several boards or settings
reproducible build	the same input gives a byte-for-byte identical image
pinned version	a fixed version of a tool
red build	the pipeline has failed
manifest, checksum	a list of outputs, a control sum of a file
cache	files kept between runs

110.1 What a pipeline fixes

Three troubles familiar to anyone who has released firmware more than once.

Built with the wrong thing. One engineer has a 2019 compiler, a colleague has a 2024 one; the images differ, the bugs differ. The cure is a pinned version in a container.

Built from the wrong thing. Someone forgot to commit an edit to the tree, or built from a modified working tree. The cure is building from a clean clone and a version from git describe (section 109): -dirty in the name is a signal that the image cannot be reproduced.

Forgot to test. Tests exist, but they are run "when someone remembers". The cure is that the pipeline runs them itself and does not hand out an artifact if even one has failed.

110.2 Stages

The firmware pipeline: commit, build in a container, tests, artifact; a failed test stops the release

The order is the same almost everywhere: fetch the sources for the commit → build → check → package. Each stage either passes completely or fails the pipeline. Then branches are added: deployment only from the main branch, image signing only on a tag, a nightly build with slow tests.

A subtlety for firmware: the "build" stage produces not a program that can be run on the same computer, but an image for a different processor. So checking splits: part of the code is tested on the computer (logic, packet parsing, checksums), and what cannot be checked without a board is left to the test bench and to observing devices in the field (section 112).

110.3 Building in a container with pinned versions

The build image is described by a file and is built from it — this is "a build machine you can reproduce":

FROM debian:12-slim
RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
      gcc-arm-none-eabi=15:12.2.rel1-1 \
      libnewlib-arm-none-eabi \
      build-essential \
      git \
      ca-certificates \

The compiler version is given explicitly, not as "latest": otherwise one morning an update will silently change the image's size and behaviour. The run shows it: arm-none-eabi-gcc 12.2.1. But the version line only works while that package is on the mirror: distribution updates remove old versions, and a year later the build image may fail to build, while debian:12-slim under the same name will by then be a different image. For the build image to be reproducible a year later too, the base image is taken by hash (debian:12-slim@sha256:…), packages come from the Debian snapshot archive for the required date, and the finished build image is kept in your own registry and is also referenced by hash.

110.4 The version comes from the history, not from someone's head

The first thing the pipeline does is find out what it is building:

VERSION=$(git describe --tags --dirty)
echo "version=$VERSION"
printf '#ifndef VERSION_H\n#define VERSION_H\n#define FW_VERSION "%s"\n#endif\n' "$VERSION" > src/version.h

The resulting string — v2.0.0 in the experiment — goes into the firmware (as a macro), into the artifact name and into the manifest. From then on the device itself says what is flashed into it, and the warehouse knows from the file name what it is shipping.

110.5 The matrix: two boards in one run

The same firmware is built for F103 and F407 one after the other — only the core options and the memory map change:

build f103 "-mcpu=cortex-m3 -mthumb" f103.ld
build f407 "-mcpu=cortex-m4 -mthumb -mfloat-abi=soft" f407.ld

-mfloat-abi=soft is used here because the experiment has no floating point and the start-up code does not enable the FPU. In real F407 firmware you set -mfpu=fpv4-sp-d16 -mfloat-abi=hard and enable the FPU at start-up, otherwise the FPU sits idle.

Both builds produced a 160-byte image, but the content differs — the code is built for different cores. This is a check too: if one of the boards stops building, the pipeline turns red at once, not a month later when someone gets round to it.

110.6 Tests: the same source on the computer

A piece of firmware not tied to registers is built with ordinary gcc and checked on the computer in a fraction of a second. A frame checksum is exactly such a piece:

uint16_t crc16(const uint8_t *p, unsigned n)
{
    uint16_t crc = 0xFFFFu;
    while (n--) {
        crc ^= (uint16_t)*p++ << 8;
        for (unsigned b = 0; b < 8u; b++)
            crc = (crc & 0x8000u) ? (uint16_t)(crc << 1 ^ 0x1021u) : (uint16_t)(crc << 1);
    }
    return crc;
}

The expected values are taken not from your own implementation but from an independent source: for CRC-16/CCITT-FALSE the string "123456789" gives 0x29B1 — this is a well-known check value, it is printed in descriptions of the algorithm. In the run the test answered exactly that.

110.7 A red build stops the release

Checking that the pipeline can fail matters more than checking that it can build: broken protection goes unnoticed until it is needed. In the experiment the CRC polynomial was replaced with a neighbouring one, and the test saw the mismatch at once — two checks out of three failed, the program returned non-zero, the pipeline stopped and no artifact appeared.

Hence the rule: a stage fails if even one check has not passed, and no "oh well, let's build it by hand". Otherwise one day a release will go out with the very bug the whole thing was set up to catch.

110.8 Repeatability: the same image byte for byte

The pipeline built the firmware twice — in different directories — and got a byte-for-byte identical image. This does not happen by itself: three things get in the way.

Build time inside the image (__DATE__, __TIME__) — fixed by the SOURCE_DATE_EPOCH variable.
Paths from the build directory end up in the debug data — removed with the -ffile-prefix-map option.
The build ID, if enabled, is a hash of the whole file including the debug data: different paths change it, and the note holding it goes into the image. It is turned off with -Wl,--build-id=none; arm-none-eabi-gcc 12 does not add it by itself, but many compiler builds and IDEs enable it.

Why this matters: an identical hash is proof that the published image was built from exactly that commit. Otherwise you have to take it on trust. For firmware that is signed and distributed to devices (bootloader and updates — section 48), this is the foundation of trust.

A reproducible build: two directories, one and the same image, byte for byte

110.9 Artifacts and the manifest

The output is three files: two images with the version in the name (fw-v2.0.0-f103.bin) and a manifest of four lines: the version, the compiler and the checksums of both images. The rules are simple:

the file name contains the version — otherwise in a month nobody will tell two fw.bin apart;
next to it lies the manifest with checksums — it is what you check the device contents against;
artifacts are stored separately from the repository, but with the commit noted;
anything that has not passed the tests does not become an artifact;
the build fails on warnings (-Werror) and on size too: the output of arm-none-eabi-size is compared with the Flash and RAM limits.

110.10 Pitfalls

A "latest version" compiler. One morning the build changes all by itself. Pin the version in the image.

Building from an engineer's working tree. Unsaved work ends up in the image: git describe will show -dirty. Build from a clean clone in a container.

A test fails, but the pipeline is green. The check's exit code is lost in a command pipeline: for make test | tee log the exit code comes from tee. In the script — set -eu -o pipefail (bash), and check the exit code, not the output.

The cache "speeds up" the build to a wrong result. Object files from another compiler version are cached. Cache only what is unambiguously tied to versions, and be able to clear it.

The image is different every time. Time, paths and the build ID inside. SOURCE_DATE_EPOCH, -ffile-prefix-map, --build-id=none.

An artifact without a version in the name. A month later the warehouse has several fw.bin, and nobody knows which one is in the devices.

Secrets in the pipeline log. A signing key printed in the log is considered leaked. Secrets only as environment variables, and masked.

110.11 Summary

The pipeline answers the question "what was it built from": a container with a pinned arm-none-eabi-gcc 12.2.1, the version from the tag — v2.0.0.
Two boards in one run: a 160-byte image for each, different content.
Part of the firmware is checked on the computer with the same source: CRC-16 "123456789" = 0x29B1.
A failed test stops the pipeline: there is no artifact.
A repeated build gives the same image byte for byte — SOURCE_DATE_EPOCH, -ffile-prefix-map, --build-id=none.
The artifact is named fw-v2.0.0-f103.bin and lies next to the manifest of checksums.

Full sources

pipeline.sh — the whole pipeline: version from git, build for two boards, tests, a repeated build, artifacts.

#!/bin/sh
# Раздел 110: НАСТОЯЩИЙ конвейер сборки прошивки в контейнере stm-ci:1.
#
# Этапы: версия из git → сборка для F103 и F407 → модульные тесты на компьютере
# → упаковка артефакта с манифестом. Проверяются два свойства, ради которых
# конвейер и заводят: сборка повторяется байт в байт и падение теста
# останавливает выпуск. Печатает ключ=значение для _verify/ci110_check.py.
set -eu
export GIT_CONFIG_GLOBAL=/tmp/gitconfig GIT_CONFIG_NOSYSTEM=1
export GIT_AUTHOR_NAME=CI GIT_AUTHOR_EMAIL=ci@example.test
export GIT_COMMITTER_NAME=CI GIT_COMMITTER_EMAIL=ci@example.test
export GIT_AUTHOR_DATE="2026-01-02T03:04:05+00:00" GIT_COMMITTER_DATE="2026-01-02T03:04:05+00:00"
# Одинаковое время сборки — иначе __DATE__ и пути сделали бы образы разными.
export SOURCE_DATE_EPOCH=1767322800 TZ=UTC LC_ALL=C

SRC=/v/ci110/fw
W=/tmp/ci
rm -rf $W
mkdir -p $W/work $W/out
cp -r $SRC/. $W/work/
cd $W/work

echo "cc_version=$(arm-none-eabi-gcc -dumpversion)"

# --- 1. Версия: репозиторий с тегом, как в разделе 109 ----------------------
git init -q -b main
git add -A
git commit -q -m "прошивка"
git tag -a v2.0.0 -m "выпуск 2.0.0"
VERSION=$(git describe --tags --dirty)
echo "version=$VERSION"
printf '#ifndef VERSION_H\n#define VERSION_H\n#define FW_VERSION "%s"\n#endif\n' "$VERSION" > src/version.h

CFLAGS="-Os -Wall -Wextra -Werror -ffunction-sections -fdata-sections -ffile-prefix-map=$W/work=. -g0"
LDFLAGS="-nostdlib -Wl,--gc-sections -Wl,--build-id=none"

build() {                      # build <плата> <ключи ядра> <карта памяти>
  board=$1; cpu=$2; ld=$3
  # shellcheck disable=SC2086
  arm-none-eabi-gcc $cpu $CFLAGS -Isrc -c src/crc.c -o crc-$board.o
  # shellcheck disable=SC2086
  arm-none-eabi-gcc $cpu $CFLAGS -Isrc -c src/main.c -o main-$board.o
  # shellcheck disable=SC2086
  arm-none-eabi-gcc $cpu $CFLAGS -c crt0.S -o crt0-$board.o
  # shellcheck disable=SC2086
  arm-none-eabi-gcc $cpu $CFLAGS $LDFLAGS -T $ld -L. crt0-$board.o main-$board.o crc-$board.o -o fw-$board.elf
  arm-none-eabi-objcopy -O binary fw-$board.elf fw-$board.bin
  echo "size_$board=$(arm-none-eabi-size -A fw-$board.elf | awk '/^.text/ {t=$2} /^.data/ {d=$2} END {print t+d}')"
  echo "bin_$board=$(wc -c < fw-$board.bin | tr -d ' ')"
  echo "sha_$board=$(sha256sum fw-$board.bin | cut -c1-16)"
}

# --- 2. Сборка для двух плат в одном прогоне --------------------------------
build f103 "-mcpu=cortex-m3 -mthumb" f103.ld
build f407 "-mcpu=cortex-m4 -mthumb -mfloat-abi=soft" f407.ld

# --- 3. Повторяемость: та же сборка ещё раз в другом каталоге ---------------
mkdir -p $W/again
cp -r $SRC/. $W/again/
cp src/version.h $W/again/src/version.h
( cd $W/again
  arm-none-eabi-gcc -mcpu=cortex-m3 -mthumb -Os -Wall -Wextra -Werror -ffunction-sections -fdata-sections \
     -ffile-prefix-map=$W/again=. -g0 -Isrc -c src/crc.c -o crc.o
  arm-none-eabi-gcc -mcpu=cortex-m3 -mthumb -Os -Wall -Wextra -Werror -ffunction-sections -fdata-sections \
     -ffile-prefix-map=$W/again=. -g0 -Isrc -c src/main.c -o main.o
  arm-none-eabi-gcc -mcpu=cortex-m3 -mthumb -Os -ffile-prefix-map=$W/again=. -g0 -c crt0.S -o crt0.o
  arm-none-eabi-gcc -mcpu=cortex-m3 -mthumb -Os -nostdlib -Wl,--gc-sections -Wl,--build-id=none -T f103.ld -L. \
     crt0.o main.o crc.o -o fw.elf
  arm-none-eabi-objcopy -O binary fw.elf fw.bin )
A=$(sha256sum fw-f103.bin | cut -d' ' -f1)
B=$(sha256sum $W/again/fw.bin | cut -d' ' -f1)
echo "reproducible=$([ "$A" = "$B" ] && echo same || echo differ)"

# --- 4. Тесты на компьютере: тот же исходник, обычный gcc -------------------
gcc -O1 -Wall -Wextra -Werror -o $W/test tests/test_crc.c src/crc.c
$W/test > $W/test.log 2>&1 && TEST_RC=0 || TEST_RC=$?
echo "test_rc=$TEST_RC"
echo "test_crc_123456789=$(grep -o '123456789 = 0x[0-9A-F]*' $W/test.log | cut -d'x' -f2)"

# --- 5. Что будет, если тест не прошёл: конвейер обязан остановиться --------
sed 's/0x1021u/0x1022u/' src/crc.c > /tmp/broken.c
gcc -O1 -w -o $W/test_broken tests/test_crc.c /tmp/broken.c -I src
$W/test_broken > /tmp/broken.log 2>&1 && BROKEN_RC=0 || BROKEN_RC=$?
echo "broken_rc=$BROKEN_RC"
echo "broken_failed=$(grep -c '^FAIL' /tmp/broken.log || true)"

# --- 6. Упаковка артефакта: имя с версией и манифест ------------------------
if [ "$TEST_RC" -eq 0 ]; then
  NAME="fw-$VERSION"
  cp fw-f103.bin $W/out/$NAME-f103.bin
  cp fw-f407.bin $W/out/$NAME-f407.bin
  ( cd $W/out
    printf 'version=%s\ncc=%s\n' "$VERSION" "$(arm-none-eabi-gcc -dumpversion)" > manifest.txt
    sha256sum ./*.bin >> manifest.txt )
  echo "artifacts=$(ls $W/out | wc -l | tr -d ' ')"
  echo "artifact_name=$(ls $W/out | grep f103 | head -1 | sed 's/-g[0-9a-f]*//')"
  echo "manifest_lines=$(wc -l < $W/out/manifest.txt | tr -d ' ')"
fi
echo "done=1"

crc.c — the frame checksum — code shared by the firmware and the unit tests.

/* Раздел 110: кусочек прошивки, который проверяется и на плате, и на компьютере.
 * Контрольная сумма кадра: тот же код собирается для F103/F407 и для тестов. */
#include "crc.h"

uint16_t crc16(const uint8_t *p, unsigned n)
{
    uint16_t crc = 0xFFFFu;
    while (n--) {
        crc ^= (uint16_t)*p++ << 8;
        for (unsigned b = 0; b < 8u; b++)
            crc = (crc & 0x8000u) ? (uint16_t)(crc << 1 ^ 0x1021u) : (uint16_t)(crc << 1);
    }
    return crc;
}

Next: section 110а "Firmware under the pipeline's eye: a size delta on every change and assert without strings in the release build" — how the pipeline guards the image size on every merge request and why assertions are worth keeping in release firmware, but without strings.

Open in the interactive handbook