126. Data in memory: alignment, packing, bit fields, byte order

A C structure is not a packet on the wire and not a register, however much you want to treat it as both. The compiler inserts empty bytes between the fields, lays out bit fields as it sees fit, and a number lies in memory least significant byte first. Firmware that sends a structure "as is" over UART or reads a packet by casting a pointer works until the compiler, the core or the other side changes. This section shows the layout by experiment and what each approach costs on Cortex-M3 and on M0.

Terms used in the documentation

Term	Meaning
alignment	the address is a multiple of the size
padding	empty bytes between fields
packed structure	a structure without padding
unaligned access	an access to an address that is not aligned
bit-field	a field a few bits wide inside a word
endianness	the order of bytes in a number
little-endian / big-endian	least significant byte first / most significant first
network byte order	the network order — most significant byte first
serialization	writing data into a byte stream
byte swap (REV)	reversing the order of bytes

126.1 Alignment and padding

Every type wants to lie at an address that is a multiple of its size: uint16_t — at an even one, uint32_t — at a multiple of four. That is why the compiler inserts empty bytes. The experiment: three layouts of the same fields:

After the one-byte type there are three empty bytes — value starts at 4. At the end there are two more: the size of a structure is a multiple of its strictest alignment, so that in an array the next structure is aligned too. Fields ordered from large to small save a third of the memory — on arrays of hundreds of records that is kilobytes of RAM.

126.2 Padding is real bytes

The empty bytes are not zeros but whatever was in memory before. In the experiment the structure lay in memory filled with 0xAA:

01 AA AA AA 02 00 00 00 03 00 AA AA

Five bytes out of twelve are garbage. Sending the whole structure puts them on the channel, and comparing structures with memcmp gives wrong answers. It is also a data leak: pieces of other variables remain in the padding.

126.3 Packing

__attribute__((packed)) removes the padding: 7 bytes, as in the packet. The price is unaligned fields. What the compiler does with them can be seen in the object file:

uint32_t wire_value(const struct wire *w)
{
    return w->value;
}

• Cortex-M3/M4 can read an unaligned word: one instruction LDR r0, [r0, #1].
• Cortex-M0 cannot: the same line becomes four LDRB and shifts, 11 instructions.

The compiler coped on its own because it knew the field is packed. The danger is something else: a pointer cast *(uint32_t *)(buf + 1): the compiler assumes the address is aligned, emits a plain LDR, and on M0 that is a HardFault. On M3/M4 the same cast works, but LDRD and LDM cannot be unaligned even there.

The address of a packed field is just as dangerous: uint32_t *p = &w->value; — from then on the compiler no longer knows that the address may be odd, and on M0 reading *p gives a HardFault. gcc warns about this (-Waddress-of-packed-member); pass the field by value or copy it with memcpy.

126.4 Byte order

Cortex-M and x86 are little-endian: the number 0x12345678 lies in memory as 78 56 34 12. Network protocols, Modbus and many sensors send the most significant byte first: 12 34 56 78. You must not mix them up, and swapping bytes by hand is slow and error-prone.

The reliable way is to build and take apart a number byte by byte, stating the order explicitly; then the code is the same on any machine:

p[0] = (uint8_t)(v >> 24);
p[1] = (uint8_t)(v >> 16);
p[2] = (uint8_t)(v >> 8);
p[3] = (uint8_t)v;

The fear that this is slow is unfounded: armclang recognised the whole "most significant first" read on Cortex-M3 and turned it into two instructions — LDR and the byte swap REV. On M0 it came out as 11 instructions, but with no risk of unaligned access.

126.5 Byte swap

__builtin_bswap32 on all three cores is one REV instruction; in CMSIS the same is __REV(). For 16 bits there is REV16. These functions are needed when the data already lies in an aligned word and the order simply has to be reversed.

126.6 Bit fields

Bit fields look like a ready-made description of a register:

uint32_t enable : 1;
uint32_t mode : 2;
uint32_t prescaler : 5;

gcc, clang and the ARM compilers laid them out from the least significant bit: enable = 1, mode = 2, prescaler = 17 gave the word 0x8D. But the order of the fields, their alignment and the access width are left to the implementation by the standard. For peripheral registers this means:

• another compiler may lay them out differently;
• writing one field is a read, modify and write of the whole word, and if next to it there is a "cleared by writing one" bit, it can be cleared by accident;
• the compiler may choose a byte access to a volatile field, while the peripheral only accepts words.

That is why CMSIS headers describe registers with masks and shifts (section 3а), and bit fields are fit for your own data — packing flags in memory.

126.7 Serializing packets

The rule for any data that leaves the device:

• The packet format is described in bytes and their order, not as a structure.
• Writing — byte by byte or with put_be32-style functions.
• Reading — the same way; or memcpy from the buffer into the field if the order matches.
• A packed structure is acceptable as a description, but the fields are read by the compiler, not through a pointer cast.
• The checksum is computed over the bytes of the packet, not over a structure with padding.

uint32_t v;
memcpy(&v, p, sizeof v);
return v;

memcpy on M3 is one LDR, on M0 it is a byte-by-byte read: the compiler chooses itself, and the choice is always correct.

126.8 The data model in the layout

The field sizes come from the data model (section 122). struct { char c; long l; } is 8 bytes on Cortex-M and 16 on 64-bit Linux and macOS: long there is 8 bytes with an alignment of 8. On 64-bit Windows long stays 4 bytes, and the same structure is 8 bytes again: every system has its own data model. A structure that the device writes to a file or sends to a program on a computer can be read wrongly even with fields from stdint.h: it sets the sizes, but not the alignment and not the byte order (126.4). struct { uint32_t a; uint64_t b; } on Cortex-M is 16 bytes, with b at address 8, and in a 32-bit program under Linux on x86 it is 12: uint64_t is aligned there to 4. Only byte-by-byte writing is reliable (126.7), and _Static_assert catches a mismatch at build time. The file for the board starts with static assertions about the layout:

_Static_assert(sizeof(struct plain) == 12 && offsetof(struct plain, value) == 4, "3 байта заполнения после type");
_Static_assert(sizeof(struct wire) == 7 && offsetof(struct wire, value) == 1, "упакованная: value с адреса 1");

126.9 Alignment of DMA buffers and of the stack

• A DMA buffer for words must be aligned to 4 — _Alignas(4) or __attribute__((aligned(4))); a uint8_t array is aligned only to 1.
• By the ARM calling convention the stack is aligned to 8 at function calls: otherwise double and printf with floating-point numbers work incorrectly — the startup code takes care of it (section 11).
• On an M7 with a cache a DMA buffer is also aligned to a cache line, 32 bytes, and the size of the buffer is made a multiple of 32: SCB_InvalidateDCache_by_Addr works on whole lines and would otherwise destroy fresh data of a neighbouring variable in the same line.

126.10 Pitfalls

HardFault on M0 while parsing a packet. *(uint32_t *)(buf + 1) is an unaligned access. Use memcpy or byte-by-byte parsing.

The receiver sees garbage in the packet. A structure with padding was sent. Serialize the fields byte by byte.

Numbers from a sensor are reversed. The sensor sends the most significant byte first, the device reads least significant first. Byte-by-byte parsing with an explicit order, or REV.

Writing a bit field cleared a flag in the neighbouring bit. The field is written by a read-modify-write of the whole word. For registers, use masks.

A file written by the device cannot be read on the computer. A different data model: long is 4 versus 8 on Linux and macOS, uint64_t in a structure is aligned to 8 versus 4 in a 32-bit program on x86. Fields from stdint.h, byte-by-byte writing, the layout checked with _Static_assert.

126.11 Summary

• Padding: type, value, crc take 12 bytes, from large fields to small — 8, packed — 7; the padding holds garbage (01 AA AA AA 02 00 00 00 03 00 AA AA).
• A packed field on M3 is one LDR r0, [r0, #1], on M0 — 11 instructions; a pointer cast on M0 means a HardFault.
• Little-endian: 0x12345678 in memory is 78 56 34 12; the compiler reduced byte-by-byte "most significant first" parsing on M3 to LDR and REV.
• Bit fields were laid out from the least significant bit (0x8D), but that is an agreement between compilers; for registers, use masks.
• The layout is checked with _Static_assert, packets go byte by byte.

Full sources

mem126.c — alignment, packing, bit fields and byte order — the experiment on three machines.

/* Раздел 126: данные в памяти — выравнивание, упаковка, битовые поля, порядок байт.
 * Опыт на трёх машинах (gcc и clang x86-64, gcc для ARM в qemu). Печатает размеры,
 * смещения полей, байты в памяти и результат ручной сериализации. */
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>

struct plain {                 /* как лягут поля по умолчанию */
    uint8_t type;
    uint32_t value;
    uint16_t crc;
};

struct ordered {               /* те же поля, от больших к малым */
    uint32_t value;
    uint16_t crc;
    uint8_t type;
};

struct __attribute__((packed)) wire {    /* без заполнения — как в пакете */
    uint8_t type;
    uint32_t value;
    uint16_t crc;
};

struct with_long {
    char c;
    long l;                    /* long разной ширины у машин */
};

union reg {                    /* битовые поля поверх 32-битного слова */
    uint32_t word;
    struct {
        uint32_t enable : 1;
        uint32_t mode : 2;
        uint32_t prescaler : 5;
        uint32_t : 24;
    } bits;
};

/* Сериализация в порядке «старший байт первым» — одинакова на любой машине. */
static void put_be32(uint8_t *p, uint32_t v)
{
    p[0] = (uint8_t)(v >> 24);
    p[1] = (uint8_t)(v >> 16);
    p[2] = (uint8_t)(v >> 8);
    p[3] = (uint8_t)v;
}

static uint32_t get_le32(const uint8_t *p)
{
    return (uint32_t)p[0] | (uint32_t)p[1] << 8 | (uint32_t)p[2] << 16 | (uint32_t)p[3] << 24;
}

static void hex(const char *key, const void *mem, size_t n)
{
    const uint8_t *b = mem;
    printf("%s=", key);
    for (size_t i = 0; i < n; i++)
        printf("%s%02X", i ? " " : "", b[i]);
    printf("\n");
}

int main(void)
{
    printf("plain=%zu,%zu,%zu,%zu\n", sizeof(struct plain), offsetof(struct plain, type),
           offsetof(struct plain, value), offsetof(struct plain, crc));
    printf("ordered=%zu,%zu,%zu,%zu\n", sizeof(struct ordered), offsetof(struct ordered, value),
           offsetof(struct ordered, crc), offsetof(struct ordered, type));
    printf("wire=%zu,%zu,%zu,%zu\n", sizeof(struct wire), offsetof(struct wire, type),
           offsetof(struct wire, value), offsetof(struct wire, crc));
    printf("with_long=%zu\n", sizeof(struct with_long));

    uint32_t v = 0x12345678u;
    hex("mem_u32", &v, sizeof v);                 /* порядок байт в памяти */
    uint8_t be[4];
    put_be32(be, v);
    hex("be_u32", be, sizeof be);
    static const uint8_t in[4] = { 0x78, 0x56, 0x34, 0x12 };
    printf("le_read=0x%08X\n", get_le32(in));
    printf("bswap=0x%08X\n", __builtin_bswap32(v));

    union reg r = { 0 };
    r.bits.enable = 1;
    r.bits.mode = 2;
    r.bits.prescaler = 17;
    printf("bitfield_word=0x%08X\n", r.word);    /* порядок полей задаёт реализация */

    struct plain p;
    memset(&p, 0xAA, sizeof p);                   /* заполнение между полями — тоже байты */
    p.type = 1;
    p.value = 2;
    p.crc = 3;
    hex("plain_bytes", &p, sizeof p);
    printf("done=1\n");
    return 0;
}

mem126_arm.c — data layout on Cortex-M: sizes, offsets, a packed structure and the byte swap.

/* Раздел 126: раскладка данных на Cortex-M — armclang для F103, F407 и M0.
 * Статические утверждения — размеры и смещения; функции — что делает компилятор
 * с упакованной структурой и с переворотом байтов. */
#include <stddef.h>
#include <stdint.h>
#include <string.h>

struct plain {
    uint8_t type;
    uint32_t value;
    uint16_t crc;
};

struct __attribute__((packed)) wire {
    uint8_t type;
    uint32_t value;
    uint16_t crc;
};

_Static_assert(sizeof(struct plain) == 12 && offsetof(struct plain, value) == 4, "3 байта заполнения после type");
_Static_assert(sizeof(struct wire) == 7 && offsetof(struct wire, value) == 1, "упакованная: value с адреса 1");
_Static_assert(sizeof(struct { char c; long l; }) == 8, "на Cortex-M long — 4 байта");

/* Поле упакованной структуры — невыровненный адрес. */
uint32_t wire_value(const struct wire *w)
{
    return w->value;
}

/* Надёжный разбор пакета: побайтно, порядок байт задан явно. */
uint32_t get_be32(const uint8_t *p)
{
    return (uint32_t)p[0] << 24 | (uint32_t)p[1] << 16 | (uint32_t)p[2] << 8 | p[3];
}

/* Переворот байтов: компилятор знает команду REV. */
uint32_t swap32(uint32_t v)
{
    return __builtin_bswap32(v);
}

/* То же через memcpy — выровнено или нет, компилятор решает сам. */
uint32_t load_u32(const uint8_t *p)
{
    uint32_t v;
    memcpy(&v, p, sizeof v);
    return v;
}

Next: section 127 "The preprocessor as a tool" — X-macros, _Generic and static checks.