144. x86-64 assembly through the eyes of C: registers, calling convention, gcc -S, inline asm, nasm

Cortex-M assembly is covered in section 7б: on a microcontroller you look at it every time you debug. The processor in a computer is different: x86-64, with sixteen registers and CISC instructions that are turned into micro-operations inside. You do not need to write programs in it; you need to read what the compiler produced — to understand why code is slow, what happened at the crash site and where the loop went after -O2. All listings are the output of gcc -S -masm=intel and objdump from stm-sys:1 on funcs.c, and the program without libc and the nasm function were built and run (_verify/sys144/).

What it is called in the documentation

Term	Meaning
general-purpose register	a register for any data and addresses
flags register (RFLAGS)	the condition flags: ZF, CF, SF, OF
calling convention (System V ABI)	the rules for passing arguments and results
caller-saved, callee-saved	registers saved by the calling / the called function
prologue, epilogue	the entry and exit code of a function
red zone	128 bytes below rsp that a leaf function uses without sub rsp
jump table, lookup table	a table of addresses to jump to, a table of values
inline assembly	assembly embedded in C code
constraint ("r", "=a", "memory")	an operand restriction in inline assembly
system call	a request to the kernel
AT&T vs Intel syntax	two ways of writing instructions

144.1 Registers

There are sixteen 64-bit general-purpose registers: rax, rbx, rcx, rdx, rsi, rdi, rbp, rsp and r8–r15. Each has lower halves: eax is 32 bits, ax is 16, al is 8; a write to eax clears the upper half of rax, and that is why listings contain many mov edi, edi — this clears the upper 32 bits. A write to ax or al leaves the upper bits as they were, so a byte is loaded into a register with movzx eax, byte ptr …: it zero-extends to 32 bits, and therefore to 64 as well. Separate from these are the instruction pointer rip, the flags RFLAGS and the registers xmm0–xmm15 for floating-point numbers and vectors.

Almost every arithmetic instruction sets the flags: ZF — the result is zero, CF — a carry out of the top bit (unsigned overflow), SF — the sign, OF — signed overflow. Conditional jumps read the flags: je (ZF), ja (unsigned greater: CF = 0 and ZF = 0), jg (signed greater). This is how an if in C becomes a "cmp + jcc" pair.

Register frequency in the funcs.c listing at -O2: rdi 14 times, rdx 8, eax 8, rax 7, rsp 5 — arguments and the result, everything else less often.

144.2 The System V calling convention

On Linux x86-64 one convention applies to all C code:

A function with two arguments at -O2:

int add(int a, int b) { return a + b; }                       /* аргументы в edi, esi; результат в eax */

add:
    lea eax, [rdi+rsi]
    ret

One instruction: lea computes the address rdi + rsi, but puts it into eax as a number — an addition without flags. With seven arguments the sixth is still in r9, and the seventh comes from the stack:

sum7:
    add rdi, rsi
    add rdi, rdx
    add rdi, rcx
    add rdi, r8
    lea rax, [rdi+r9]
    add rax, QWORD PTR 8[rsp]
    ret

A nasm function must follow the same convention — and then C calls it like one of its own:

asm_add3:
        mov     rax, rdi            ; a
        add     rax, rsi            ; + b
        add     rax, rdx            ; + c
        ret

nasm: asm_add3(1, 2, 3) = 6

Windows has a different convention (rcx, rdx, r8, r9 and 32 bytes of "shadow space" on the stack) — so an assembly function written for Linux will not work under Windows without rewriting.

144.3 Prologue, epilogue and what -O2 does

The same add at -O0:

add:
    push    rbp
    mov rbp, rsp
    mov DWORD PTR -4[rbp], edi
    mov DWORD PTR -8[rbp], esi
    mov edx, DWORD PTR -4[rbp]
    mov eax, DWORD PTR -8[rbp]
    add eax, edx
    pop rbp
    ret

The prologue push rbp; mov rbp, rsp builds a frame, and the arguments are stored on the stack and read back — this way the debugger sees every variable. At -O2 there is no frame and no variables in memory; nine instructions became two. The loop_sum loop is 22 lines at -O0 versus 11 at -O2.

Calling another function: space for the array buf[4] is taken by subtracting from rsp, the address is passed in rdi, the size in esi, then call, then release:

calls_other:
    movd    xmm1, edi
    sub rsp, 24
    mov esi, 4
    pshufd  xmm0, xmm1, 0
    paddd   xmm0, XMMWORD PTR .LC0[rip]
    mov rdi, rsp
    movaps  XMMWORD PTR [rsp], xmm0
    call    loop_sum
    add rsp, 24
    ret

The compiler filled the four elements {x, x+1, x+2, x+3} with a single vector instruction paddd — 16 bytes at a time. sub rsp, 24, not 16: after call the stack also holds the return address, and 24 + 8 = 32 keeps the 16-byte alignment.

144.4 Arithmetic, switch and tables

Multiplying by 10 and dividing by 4 without mul and div:

mul_div:
    lea rdx, [rdi+rdi*4]
    add rdx, rdx
    lea rax, 3[rdx]
    cmovns  rax, rdx
    sar rax, 2
    ret

x·5 via lea with a scale of 4, ·2 via an addition, division by 4 is an arithmetic right shift, and lea rax, 3[rdx] with cmovns is a correction for negative numbers: the shift rounds toward minus infinity, while C requires rounding toward zero. div costs tens of cycles, a shift costs one; so dividing by a constant is cheap, and dividing by a variable is not.

The switch over six values became not a chain of comparisons but a lookup table in .rodata:

by_switch:
    mov eax, -1
    cmp edi, 5
    ja  .L10
    mov edi, edi
    lea rax, CSWTCH.5[rip]
    mov eax, DWORD PTR [rax+rdi*4]
.L10:
    ret

One ja comparison and a read from an array. ja compares the 32-bit edi as an unsigned number: for it −1 is 0xFFFFFFFF, so the "greater than 5" condition cuts off negative values too. The following mov edi, edi clears the upper half of rdi — the index in the address [rax+rdi*4] must be 64-bit. When the branches do not reduce to a value, the compiler builds a jump table — an array of addresses and jmp [rax + rdi*8]. This way a switch over dozens of densely packed case labels costs one comparison and one read. For sparse values the compiler builds a comparison tree: log₂N checks instead of N.

144.5 Reading machine code: objdump

gcc -S shows the assembly before linking; objdump -d disassembles a finished object file or program together with the instruction bytes:

0000000000000000 <add>:
   0:   8d 04 37                lea    eax,[rdi+rsi*1]
   3:   c3                      ret

lea is three bytes, ret is one; x86 instructions have variable length, from 1 to 15 bytes. The -M intel option gives Intel syntax ("destination, source"); without it you get AT&T with % before register names and the reverse operand order. objdump -d -S (for objdump the -S option means "with source", not what it means for gcc), on a build with -g, interleaves machine code with C lines. The code of the six functions in funcs.o takes 395 bytes (size, the text column).

144.6 Inline assembly

Sometimes you need an instruction that C does not have: a time counter, processor identification, the carry flag, a system call without libc.

static inline uint64_t rdtsc(void)
{
    uint32_t lo, hi;
    __asm__ volatile ("rdtsc" : "=a"(lo), "=d"(hi));           /* выходы: eax → lo, edx → hi */
    return ((uint64_t)hi << 32) | lo;
}

The form is: asm(text : outputs : inputs : clobbers). "=a"(lo) is an output in eax, "=d" in edx, "r" is any register, "m" is memory, "+r" is input and output at once. The list after the third colon tells the compiler what the instruction changes: "cc" — the flags, "memory" — memory (a barrier against reordering), register names — specific registers. volatile forbids dropping the instruction as "useless".

    __asm__ ("add %[b], %[a]\n\t"
             "setc %[c]"                                        /* флаг переноса → байт */
             : [a] "+r"(a), [c] "=r"(carry)
             : [b] "r"(b)
             : "cc");                                           /* меняем флаги */

A system call made directly: the number in rax, the arguments in rdi, rsi, rdx, the syscall instruction, the result in rax; the kernel clobbers rcx and r11:

    __asm__ volatile ("syscall"                                 /* номер 1 = write; аргументы rdi, rsi, rdx */
                      : "=a"(ret)
                      : "a"(1), "D"(fd), "S"(buf), "d"(n)
                      : "rcx", "r11", "memory");                /* syscall портит rcx и r11 */

cpuid: GenuineIntel, старший лист 28
rdtsc: 1000 итераций — 4528 тиков TSC
add 0xFFFF…FFFF + 2 = 1, перенос 1
syscall write без libc
syscall вернул 26

add gave 1 with a carry — an unsigned overflow that C makes invisible and the CF flag reveals. On a terminal the lines appear in program order: printf flushes the buffer at every newline. But if the output goes to a pipe (./inline_asm | cat), the line "syscall write without libc" comes out first, before cpuid: in that case printf accumulates everything in the buffer until exit, while write goes to the kernel immediately.

rdtsc returns not core cycles but ticks of the TSC counter. On modern x86 it runs at a constant, nominal processor frequency and ignores both turbo mode and reduced frequency — there ticks and cycles diverge. Besides, the instruction is not ordered: the processor may execute it before or after its neighbours, so a short measurement is wrapped in lfence, or rdtscp is used, which waits for the preceding instructions. Only a performance counter counts core cycles honestly: perf stat -e cycles.

144.7 A program without libc

_start:
        mov     rax, 1              ; write
        mov     rdi, 1              ; stdout
        mov     rsi, msg
        mov     rdx, len
        syscall
        mov     rax, 60             ; exit
        mov     rdi, 7              ; код возврата 7
        syscall

nasm -f elf64 hello.asm && ld -o hello hello.o — 8896 bytes, and ldd says "not a dynamic executable": neither libc nor the dynamic linker. The entry point is _start, not main: main is called by the libc startup code, which is not here. The program prints a line and exits with code 7 — exit is also a system call, number 60. This is exactly what Reset_Handler does in firmware before main (section 11), only on a different processor.

144.8 Pitfalls

144.9 Summary

• Arguments: rdi, rsi, rdx, rcx, r8, r9, then the stack; result in rax; save rbx, rbp, r12–r15; stack aligned to 16.
• -O0: a frame and variables on the stack (9 instructions); -O2: lea + ret.
• Multiplication and division by a constant — lea and shifts; switch — a table.
• gcc -S -masm=intel, objdump -d -M intel.
• Inline assembly: outputs, inputs, clobbers; volatile, "memory", "cc".
• Without libc: _start, syscall 1 and 60, 8896 bytes.

Full sources

inline_asm.c — GCC inline assembly: rdtsc, cpuid, carry, write without libc.

/* Раздел 144: встроенный ассемблер GCC — счётчик тактов rdtsc, cpuid, add с флагом переноса,
   системный вызов write без libc, функция из файла на nasm. */
#include <stdio.h>
#include <stdint.h>
#include <string.h>

long asm_add3(long a, long b, long c);                          /* написана на nasm (add3.asm) */

static inline uint64_t rdtsc(void)
{
    uint32_t lo, hi;
    __asm__ volatile ("rdtsc" : "=a"(lo), "=d"(hi));           /* выходы: eax → lo, edx → hi */
    return ((uint64_t)hi << 32) | lo;
}

static int add_carry(uint64_t a, uint64_t b, uint64_t *sum)
{
    uint8_t carry;
    __asm__ ("add %[b], %[a]\n\t"
             "setc %[c]"                                        /* флаг переноса → байт */
             : [a] "+r"(a), [c] "=r"(carry)
             : [b] "r"(b)
             : "cc");                                           /* меняем флаги */
    *sum = a;
    return carry;
}

static long raw_write(int fd, const void *buf, size_t n)
{
    long ret;
    __asm__ volatile ("syscall"                                 /* номер 1 = write; аргументы rdi, rsi, rdx */
                      : "=a"(ret)
                      : "a"(1), "D"(fd), "S"(buf), "d"(n)
                      : "rcx", "r11", "memory");                /* syscall портит rcx и r11 */
    return ret;
}

int main(void)
{
    char vendor[13] = {0};
    uint32_t eax = 0, ebx, ecx, edx;
    __asm__ ("cpuid" : "+a"(eax), "=b"(ebx), "=c"(ecx), "=d"(edx));
    memcpy(vendor, &ebx, 4); memcpy(vendor + 4, &edx, 4); memcpy(vendor + 8, &ecx, 4);
    printf("cpuid: %s, старший лист %u\n", vendor, eax);
    uint64_t t0 = rdtsc(); volatile int x = 0; for (int i = 0; i < 1000; i++) x += i; uint64_t t1 = rdtsc();
    printf("rdtsc: 1000 итераций — %llu тиков TSC\n", (unsigned long long)(t1 - t0));
    uint64_t s; int c = add_carry(UINT64_MAX, 2, &s);
    printf("add 0xFFFF…FFFF + 2 = %llu, перенос %d\n", (unsigned long long)s, c);
    const char msg[] = "syscall write без libc\n";
    long n = raw_write(1, msg, sizeof msg - 1);
    printf("syscall вернул %ld\n", n);
    printf("nasm: asm_add3(1, 2, 3) = %ld\n", asm_add3(1, 2, 3));
    return 0;
}

Next: section 145 — UNIX as a development environment.