195. A/B update: two slots, confirmation and rollback

In section 193 you updated the device program by replacing a file and restarting the service. You cannot do that to update the whole system — the kernel, the libraries and the service together: if power is lost in the middle of the write, the device will not boot at all. The A/B scheme solves this with two copies of the system. One copy runs, the new one is written to the other; the bootloader switches to it only after the write is complete, and if the new one does not work, it goes back to the old one on its own.

This section is a model of such a scheme in C (_verify/linux195/ab195.c): files stand in for flash partitions, but the logic is the same as in real update systems. The model was run through four scenarios, including a power loss in the middle of writing the image and in the middle of writing the metadata. Checked by _verify/linux195_check.py.

Terms used in the documentation

Term	Meaning
A/B (dual-copy) update	an update that keeps two copies of the system
slot, active / inactive slot	a slot: the running one and the spare one
boot counter, boot attempts	the counter of boot attempts
mark good, commit	confirmation: the new version is declared working
rollback, fallback	going back to the previous version
atomic update	an indivisible update: all or nothing
power-fail safe	survives a loss of power

195.1 Three participants

The key word is confirmation. A new version counts as working not when it has been written and not when it has booted, but when it has said "I am fine" itself. If it has not said so within three attempts, there is a rollback.

195.2 Metadata: small but essential

Besides the two slots, you need a record of what they contain and which one to boot. In the model it is 52 bytes: the record number, which slot to boot, and for each slot its state, version, remaining attempts, size and the CRC-32 of the image; the whole record is protected by its own CRC-32.

The metadata changes at every boot of a trial version, and its write can also be cut off halfway. That is why there are two copies: a new record always goes to the older copy, and on reading the newest intact one is taken:

int va = read_file("meta0.bin", &a, sizeof a) == sizeof a && meta_ok(&a);
int vb = read_file("meta1.bin", &b, sizeof b) == sizeof b && meta_ok(&b);
if (!va && !vb) return -1;
*m = (va && (!vb || a.seq > b.seq)) ? a : b;

If a write is cut off, the CRC of that copy will not match, and the bootloader takes the other one, with the previous state. Atomicity here comes not from the file system but from the order of writes.

195.3 Four scenarios

Factory state: version 1 in slot A, confirmed.

1. A normal update to v2. The image is written to B, B is "on trial", 3 attempts. Boot: slot B, 2 attempts left. The application checked itself and confirmed B. Result: A — v1 good, B — v2 good, B boots. The old version stays in place — that is exactly the reserve for a rollback.

2. An update to v3 that crashes at every start. v3 is written to A (now A is the spare one). Three boots in a row, attempts left 2, 1, 0 — and every time the application crashes before confirming. The fourth boot: "no attempts left — slot A is bad, rolling back to B". Result: A — v3 bad, B — v2 good, the device runs v2 without any human intervention.

3. Power lost in the middle of writing the v4 image. 3000 bytes out of 8000 were written; the updater read the slot back, the CRC did not match, and the slot was not enabled. The metadata had declared this slot empty before the write, so the bootloader did not even try to boot it. Result: B, v2 boots.

4. Power lost in the middle of writing the metadata. The write was cut off at the 10th byte. After that only one copy remained intact — the previous one: record number 12, as before the cut. The bootloader took it, and the device booted as if nothing had happened.

195.4 The order of actions is the whole point

The scheme is robust only with a strict order of writes. For the updater:

1. Declare the spare slot empty — before touching its contents.
2. Write the image.
3. Read it back and verify the CRC (or the signature, section 48а).
4. Only now: the slot is "on trial", 3 attempts, the next boot is from it.

Swap 1 and 2, and a cut-off at step 2 leaves a "good" slot with half an image in the metadata. Remove step 3, and the bootloader gets an image that nobody has checked.

For the bootloader: first decrement the attempts and write that down, then boot. The other way round does not work — a hung new version would boot forever, because the record of the attempt would never be written.

For the application: confirm after a real health check — the service is up, the link to the microcontroller works (section 194), the network responds. Otherwise the rollback will never trigger: a broken version will manage to confirm itself.

Who reboots a hung version. The attempt counter works only if the broken system reboots by itself. After a panic, the Linux kernel reboots the device only with the panic= parameter on the command line (the number is the seconds before the reboot; in a typical kernel build the default is 0 — the kernel just stays halted), and a hung service or kernel will not reboot the device at all: no attempts are used up, and v3 hangs until a person arrives. You need a hardware watchdog timer: the bootloader enables it even before the kernel starts, and the service feeds it after the health check. Then a hung v3 is rebooted by the watchdog, and the attempts run out just as with a crash.

195.5 What it looks like in a real system

• Slots are two partitions with a root file system on eMMC or SD (or two volumes in UBI on NAND); often read-only.
• The metadata and the attempt counter live in the bootloader environment variables or in a separate small partition; bootloaders for embedded Linux have a ready-made boot counting mechanism for this.
• The updater: ready-made update systems do the same as the model: they write to the inactive slot, check the signature and the CRC, and switch.
• User data goes into a third partition that the update does not touch: otherwise a rollback would also bring back the old settings.
• The price of the scheme is double the space for the system. On devices with little memory, instead of a second full copy you keep a small recovery system.

195.6 Pitfalls

The device does not boot after a power loss. The update was done "in place", over the running system. Write only to the spare slot.

The update "succeeded", but the device boots half an image. The slot was marked before the write or was not read back afterwards. First "empty", then the write, then the check, then "on trial".

A broken version boots again and again. The attempt counter is decremented after the boot, not before. Or the application confirms itself right at startup.

The new version hung, and there is no rollback. There is no watchdog timer, panic= is not set: the device does not reboot, and no attempts are used up.

Chaos after a cut-off while writing the metadata. There is only one copy of the metadata. You need two, with a record number and a CRC; write to the older one.

The rollback also brought back the old settings. User data lives in the system slot. It belongs in a separate partition.

There is nothing to roll back to. A second update was started while the first was still "on trial", and it overwrote the spare slot, the only working one. Start a new update only after the current version has been confirmed.

195.7 Summary

• Two slots: one runs, the new one is written to the other; the old one is the reserve for a rollback.
• A new version is "on trial" with three attempts; it becomes working only after the application itself confirms it.
• The model: the broken v3 rolled back to v2 on the 4th boot; a cut-off while writing the image did not enable the slot; a cut-off while writing the metadata left the previous copy intact (record number 12), and the device booted.
• The metadata is 52 bytes in two copies, each under a CRC-32; the newest intact one is read.
• The order: "empty" → write → check → "on trial"; the attempt is recorded before the boot; confirmation comes after the health check.

Full sources

ab195.c — a model of an A/B update with rollback, on files.

/* Раздел 195: обновление по схеме A/B с откатом — модель на файлах.
 *
 * «Флеш» — каталог: slotA.img, slotB.img и две копии метаданных meta0.bin, meta1.bin.
 * Метаданные: номер записи (растёт), какой слот загружать, для каждого слота — состояние,
 * версия, оставшиеся попытки, CRC-32 образа; вся запись под своим CRC-32.
 * Запись метаданных — всегда в ту копию, что старее; при чтении берётся самая новая целая.
 * Загрузчик: слот «на проверке» грузит, уменьшив попытки; попытки кончились — слот «плохой»,
 * грузится «хороший». Приложение после проверки здоровья подтверждает слот. */
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>

enum state { EMPTY, PENDING, GOOD, BAD };
static const char *const ST[] = {"пусто", "на проверке", "хороший", "плохой"};

struct slot { uint32_t state, version, tries, size, crc; };
struct meta { uint32_t seq, boot; struct slot s[2]; uint32_t crc; };

static char dir[256];

static uint32_t crc32(const void *p, size_t n)
{
    const uint8_t *b = p;
    uint32_t c = 0xFFFFFFFFu;
    while (n--) {
        c ^= *b++;
        for (int k = 0; k < 8; k++)
            c = (c >> 1) ^ (0xEDB88320u & -(c & 1));
    }
    return ~c;
}

static void path(char *out, const char *name) { snprintf(out, 300, "%s/%s", dir, name); }

static int write_file(const char *name, const void *p, size_t n, size_t cut)   /* cut — оборвать после стольких байт */
{
    char f[300];
    path(f, name);
    FILE *fp = fopen(f, "wb");
    if (!fp) return -1;
    fwrite(p, 1, cut < n ? cut : n, fp);
    fflush(fp);
    fsync(fileno(fp));
    fclose(fp);
    return cut < n ? -1 : 0;
}

static size_t read_file(const char *name, void *p, size_t n)
{
    char f[300];
    path(f, name);
    FILE *fp = fopen(f, "rb");
    if (!fp) return 0;
    size_t r = fread(p, 1, n, fp);
    fclose(fp);
    return r;
}

static int meta_ok(const struct meta *m) { return m->crc == crc32(m, offsetof(struct meta, crc)); }

static int load_meta(struct meta *m)
{
    struct meta a, b;
    int va = read_file("meta0.bin", &a, sizeof a) == sizeof a && meta_ok(&a);
    int vb = read_file("meta1.bin", &b, sizeof b) == sizeof b && meta_ok(&b);
    if (!va && !vb) return -1;
    *m = (va && (!vb || a.seq > b.seq)) ? a : b;
    return (va && vb) ? 2 : 1;                       /* сколько копий целы */
}

static void save_meta(struct meta *m, size_t cut)
{
    struct meta cur;
    int have = load_meta(&cur);
    m->seq = have > 0 ? cur.seq + 1 : 1;
    m->crc = crc32(m, offsetof(struct meta, crc));
    /* пишем в ту копию, что НЕ содержит текущей записи */
    struct meta a;
    int va = read_file("meta0.bin", &a, sizeof a) == sizeof a && meta_ok(&a);
    const char *target = (va && have > 0 && a.seq == cur.seq) ? "meta1.bin" : "meta0.bin";
    write_file(target, m, sizeof *m, cut);
}

static uint8_t img[8192];

static void make_image(uint32_t version, size_t n)
{
    for (size_t i = 0; i < n; i++)
        img[i] = (uint8_t)(version * 31 + i * 7);
}

/* ── обновитель: пишет в неактивный слот, проверяет, ставит «на проверке» ── */
static int update(uint32_t version, size_t n, size_t power_cut_at)
{
    struct meta m;
    load_meta(&m);
    int tgt = 1 - (int)m.boot;
    make_image(version, n);
    const char *name = tgt ? "slotB.img" : "slotA.img";
    m.s[tgt] = (struct slot){EMPTY, 0, 0, 0, 0};       /* сначала объявить слот пустым */
    save_meta(&m, (size_t)-1);
    write_file(name, img, n, power_cut_at);
    /* после (возможного) обрыва — проверка того, что реально лежит в слоте */
    static uint8_t back[8192];
    size_t got = read_file(name, back, sizeof back);
    if (got != n || crc32(back, n) != crc32(img, n)) {
        printf("  обновление v%u: образ в слоте %c не сошёлся по CRC (%zu из %zu байт) — слот не включён\n",
               version, "AB"[tgt], got, n);
        return -1;
    }
    load_meta(&m);
    m.s[tgt] = (struct slot){PENDING, version, 3, (uint32_t)n, crc32(img, n)};
    m.boot = (uint32_t)tgt;
    save_meta(&m, (size_t)-1);
    printf("  обновление v%u: записано в слот %c, слот «на проверке», попыток 3\n", version, "AB"[tgt]);
    return 0;
}

/* ── загрузчик ── */
static int boot(void)
{
    struct meta m;
    int copies = load_meta(&m);
    int b = (int)m.boot;
    if (m.s[b].state == PENDING) {
        if (m.s[b].tries == 0) {
            m.s[b].state = BAD;
            b = 1 - b;
            m.boot = (uint32_t)b;
            save_meta(&m, (size_t)-1);
            printf("  загрузка: попытки кончились — слот %c плохой, откат на %c\n", "AB"[1 - b], "AB"[b]);
        } else {
            m.s[b].tries--;
            save_meta(&m, (size_t)-1);
        }
    }
    /* образ проверяется всегда: загрузчик не запускает то, что не сходится с CRC в метаданных */
    static uint8_t back[8192];
    size_t got = read_file(b ? "slotB.img" : "slotA.img", back, sizeof back);
    int intact = got == m.s[b].size && crc32(back, got) == m.s[b].crc;
    printf("  загрузка: слот %c, v%u, %s, попыток осталось %u, образ %s, копий метаданных целы: %d\n",
           "AB"[b], m.s[b].version, ST[m.s[b].state], m.s[b].tries, intact ? "цел" : "ИСПОРЧЕН", copies);
    return b;
}

/* ── приложение: проверка здоровья и подтверждение ── */
static void confirm(int healthy)
{
    struct meta m;
    load_meta(&m);
    int b = (int)m.boot;
    if (healthy && m.s[b].state == PENDING) {
        m.s[b].state = GOOD;
        m.s[b].tries = 0;
        save_meta(&m, (size_t)-1);
        printf("  приложение v%u: проверка прошла, слот %c подтверждён\n", m.s[b].version, "AB"[b]);
    } else if (!healthy) {
        printf("  приложение v%u: упало до подтверждения\n", m.s[b].version);
    }
}

static void status(const char *title)
{
    struct meta m;
    load_meta(&m);
    printf("  [%s] seq=%u загружать %c | A: v%u %s | B: v%u %s\n", title, m.seq, "AB"[m.boot],
           m.s[0].version, ST[m.s[0].state], m.s[1].version, ST[m.s[1].state]);
}

int main(void)
{
    snprintf(dir, sizeof dir, "/tmp/ab195");
    mkdir(dir, 0755);
    /* завод: v1 в слоте A, подтверждён */
    make_image(1, 6000);
    write_file("slotA.img", img, 6000, (size_t)-1);
    struct meta m = {0, 0, {{GOOD, 1, 0, 6000, crc32(img, 6000)}, {EMPTY, 0, 0, 0, 0}}, 0};
    save_meta(&m, (size_t)-1);
    status("завод");

    printf("1) обычное обновление до v2\n");
    update(2, 7000, (size_t)-1);
    boot();
    confirm(1);
    status("после 1");

    printf("2) обновление до v3, которое падает при каждом запуске\n");
    update(3, 7500, (size_t)-1);
    for (int i = 0; i < 4; i++) {
        int b = boot();
        struct meta mm;
        load_meta(&mm);
        if (mm.s[b].version == 3)
            confirm(0);
        else
            confirm(1);
    }
    status("после 2");

    printf("3) пропало питание посреди записи образа v4\n");
    update(4, 8000, 3000);
    boot();
    status("после 3");

    printf("4) пропало питание посреди записи метаданных\n");
    load_meta(&m);
    struct meta before = m;
    m.s[0].version = 99;                                /* какая-то правка метаданных... */
    save_meta(&m, 10);                                  /* ...оборвалась на 10-м байте */
    struct meta after;
    int copies = load_meta(&after);
    printf("  метаданные после обрыва: целых копий %d, seq=%u (был %u), A: v%u\n", copies, after.seq, before.seq,
           after.s[0].version);
    boot();
    status("после 4");
    printf("meta_bytes=%zu\n", sizeof(struct meta));
    return 0;
}

Next: section 196 "Mini-project: a thermostat on the BluePill".