135. The process: startup, environment, memory, limits, termination

Firmware on a microcontroller is a single program that owns everything. On a computer running Linux, a program lives inside a process: the kernel gives it memory, processor time and files, watches its limits and cleans up after it when it exits. This part of the handbook is about how a C program lives under the kernel: processes, threads, communication between them, libraries, the kernel from the inside. It starts with the process itself: what it gets at startup, where its data lives and what happens when it terminates.

All the numbers come from runs in the stm-sys:1 container (Debian 12, gcc 12, kernel 6.6): the program _verify/sys135/proc_layout.c prints the addresses of its own segments, counts page faults and reads its own memory map.

Terms used in the documentation

Term	Meaning
process	a program started by the kernel, with its own memory
exit status	the code the process returns on termination
environment	variables of the form NAME=value
text, data, bss	the segments of code, initialized data and zeroed data
heap, stack	dynamic memory, the call stack
program break	the end of the heap, moved by brk
memory mapping	a region mapped into the address space (mmap)
page fault	an access to a page that is not there yet
resident set size (RSS)	how much physical memory is in use
resource limit	a cap on a resource (ulimit)
nice value	"niceness": the higher it is, the lower the priority
OOM killer	the kernel mechanism that kills processes when memory runs out

135.1 Termination: three functions and the exit code

A process ends in three ways, and they are not equivalent:

The exit code is the low 8 bits: the shell shows it as $?. Death by a signal is shown as $? = 128 + the number: SIGKILL gives 137, SIGSEGV gives 139. EXIT_SUCCESS is 0, EXIT_FAILURE is 1. The program in this section returns its first argument:

./proc_layout 7 >/dev/null || echo "rc=$?"
rc=7

The difference between exit and _exit shows when the output goes not to a terminal but to a pipe: stdout is then fully buffered, and _exit loses everything the program has "printed" so far. The run ./proc_layout _exit | cat does not output a single byte, although a normal run prints 14 lines. A function registered with atexit is not called either. So _exit is for a child after a failed exec and for signal handlers, and on the normal path you use exit.

static void at_exit_hook(void) { puts("atexit: вызвано при exit()"); }

135.2 What a process gets at startup

main takes three arguments: the number of command-line words, the words themselves and the environment. The third argument is not in the C standard, but every Unix has it; the portable way to read the environment is extern char **environ and getenv:

int main(int argc, char **argv, char **envp)

The environment is inherited: a child gets a copy of its parent's environment. The command env -i starts a program with an empty environment, and getenv("PATH") returns NULL:

env -i FOO=bar ./proc_layout
argv0 ./proc_layout argc 1 envp[0] FOO=bar...
PATH через getenv: нет

Variables that almost everything relies on: PATH (where to look for programs), HOME, LANG and LC_* (language and encoding), TERM, PWD. Two are special and belong to the dynamic linker: LD_LIBRARY_PATH adds directories with libraries, LD_PRELOAD loads your own library before all the others (section 143). A program's own settings conventionally go where the XDG rules say: $HOME/.config, $HOME/.cache, $HOME/.local/share.

135.3 Process memory: segments

The program proc_layout declares one variable in each segment and prints their addresses:

char bss_buf[1 << 20];                 /* 1 МБ нулей — в .bss, в файле места не занимает */
char data_buf[1 << 10] = {1};          /* 1 КБ — в .data, лежит в файле */
const char ro_str[] = "только чтение"; /* .rodata */

The size command counts the three segments of the executable:

   text    data     bss     dec     hex filename
   4110    1752 1048608 1054470  101706 proc_layout

bss is 1048608 bytes, yet the proc_layout file is small: the zeros are not stored, the kernel allocates pages for them at startup. This is the same principle as in firmware (section 11): .data is copied from Flash, .bss is zeroed.

Addresses in the running process (one run; on the next one they will be different — the addresses are randomized):

text   main      0x5ce1f313820f
rodata ro_str    0x5ce1f3139010
data   data_buf  0x5ce1f313b0a0
bss    bss_buf   0x5ce1f313b4c0
heap   malloc100 0x5ce1f78482a0
mmap   malloc256K 0x7339f5beb010
stack  local     0x7fff44c88dcc
brk    sbrk(0)   0x5ce1f7869000

The order from the bottom up by address: code, constants, data, zeros, heap — all close together; the stack is far up and grows down; and the 256 KB from malloc lies between them, in a separate region. Why — in the next subsection.

A process can read its memory map itself from /proc/self/maps:

631277405000-631277406000 r--p /v/sys135/proc_layout
631277406000-631277407000 r-xp /v/sys135/proc_layout
631277407000-631277408000 r--p /v/sys135/proc_layout
631277408000-631277409000 r--p /v/sys135/proc_layout
631277409000-63127740a000 rw-p /v/sys135/proc_layout
6312aecb6000-6312aecd7000 rw-p [heap]
7fff76efb000-7fff76f1c000 rw-p [stack]
7fff76f52000-7fff76f56000 r--p [vvar]
7fff76f56000-7fff76f58000 r-xp [vdso]

Five mappings of one file with different permissions: only the code r-xp is executable, only the data rw-p is writable. An attempt to write to ro_str gives SIGSEGV, and that is the kernel's doing, not the compiler's. [vdso] is a piece of kernel code mapped into the process: through it gettimeofday and clock_gettime work without a system call.

The map is shortened: the experiment prints only the lines of its own file, the heap, the stack and the kernel; the lines for libc and the dynamic linker are not in it. The megabyte of bss_buf is not visible either: the file mapping rw-p holds one page with data_buf, and the zeros lie in an anonymous rw-p mapping right after it.

135.4 The heap: brk and mmap

The heap in Linux is not one mechanism but two. malloc from glibc takes small blocks from the region past the end of the data, moving its boundary with the system call brk; sbrk(0) returns the current boundary — in the run it sits right after the heap. Large requests — by default from 128 KB (M_MMAP_THRESHOLD) — malloc serves with a separate mmap: that is why the 256 KB block ended up at address 0x7339…, far from the heap, and free of such a block returns the memory to the kernel at once. Small blocks stay with the process after free: glibc moves the brk boundary down only when the free end of the heap is larger than M_TRIM_THRESHOLD (128 KB by default), and holes in the middle of the heap are released only by malloc_trim(0). In the experiment, 4000 blocks of 1000 bytes took 3960 KB of heap; free of all but the last did not move the boundary by a single byte, malloc_trim(0) returned the pages from the middle (RSS fell by almost 4 MB, the boundary by only 20 KB), and free of the last block lowered the boundary to 136 KB. The mmap threshold, meanwhile, floats: after free of a large block it goes up, and the next block of the same 256 KB came from the heap.

The threshold and the behavior are changed with mallopt, and mallinfo shows statistics. The standard malloc is not the only one: jemalloc, tcmalloc and mimalloc replace it entirely through LD_PRELOAD for the sake of multithreading and lower fragmentation, and dmalloc for finding bugs.

135.5 Pages and page faults

Memory is handed out in pages of 4096 bytes (getconf PAGESIZE), and it is handed out lazily: bss_buf is declared as 1 MB, but there is no physical memory behind it until it is accessed. The first access to each page is a page fault: the kernel finds a page, zeroes it and maps it in. The program counts the faults with getrusage before and after a memset over 1 MB:

minor faults за memset 1 МБ: 256 (страниц по 4 КБ: 256)

Exactly one fault per page. This is a minor fault — the page simply has not been allocated yet. A major fault is when the page has to be read from disk: a file or swap. Their counts are the minflt and majflt columns in getrusage, and RSS in top is how many of the process's pages are physically in memory right now. VIRT there is everything the process has reserved, including untouched zeros and mapped files; a VIRT of gigabytes is nothing to worry about, look at RES.

135.6 Limits

A process has a soft and a hard limit for each resource: getrlimit reads them, setrlimit changes them, and the soft one can be raised only up to the hard one. In the container:

RLIMIT_STACK мягкий 8192 КБ
RLIMIT_NOFILE 1048576

An 8 MB stack — this is where "it crashes on a big array in main" comes from: a 10 MB array on the stack exceeds the limit and gives SIGSEGV. Such an array is made static or taken from the heap. The other limits:

From the shell, the same thing is ulimit -a; ulimit -c unlimited enables core files for post-mortem debugging (section 138).

135.7 Priority, time, /proc

Priority. nice runs from −20 to 19; an ordinary user can only raise the number, that is, yield. From a program — nice(incr) or setpriority. A check: nice -n 10 sh -c 'cat /proc/$$/stat' shows nice=10 in field 19.

Time. times and getrusage split time into user and system time — how much the process computed itself and how much the kernel worked on its behalf. The time command prints the same; /usr/bin/time -v also prints the maximum RSS, page faults and context switches.

/proc. Everything the kernel knows about a process lives in /proc/<pid>/: status — state and memory, maps — the map, fd/ — open files, limits — the limits, stat — one line of fields for programs. /proc/self is a link to the process itself.

OOM. When memory runs out, the kernel kills the process with the highest oom_score; the log line is Out of Memory: Killed process …. The file /proc/<pid>/oom_score_adj, from −1000 to 1000, tells the kernel whom to kill reluctantly and whom first.

Hence a surprise for a C programmer: malloc in Linux almost never returns NULL when memory runs short. By default the kernel promises more than it has (vm.overcommit_memory = 0 refuses only a clearly impossible request), and hands out pages on first touch (135.5). In an experiment in a container with a 128 MB limit, malloc of 512 MB returned an address, and the process was killed while writing the pages — exit code 137, that is, SIGKILL; a check for NULL does not help here. Strict accounting is vm.overcommit_memory = 2; for a single program it is the RLIMIT_AS limit (135.6): with it set to 256 MB, malloc of 1 GB honestly returned NULL.

135.8 Pitfalls

135.9 Summary

• exit calls atexit and flushes the buffers, _exit does not; the exit code is 8 bits.
• main(argc, argv, envp); the environment is inherited, env -i clears it.
• Segments: text, rodata, data, bss and the heap are close together; the stack is at the top; a 1 MB .bss takes no space in the file.
• malloc: small blocks come from brk, large ones (from 128 KB at first, the threshold floats) from a separate mmap; free of small blocks gives memory back only when the end of the heap is free.
• A 4096-byte page is handed out on first access: 256 faults per 1 MB.
• Limits via getrlimit/ulimit: an 8 MB stack; nice can only yield; everything about a process is in /proc/<pid>/.

Full sources

proc_layout.c — addresses of the process's segments, heap and stack, and the mmap threshold.

/* Раздел 135: где что лежит в памяти процесса — адреса сегментов, куча, стек, порог mmap. */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <malloc.h>
#include <sys/resource.h>

char bss_buf[1 << 20];                 /* 1 МБ нулей — в .bss, в файле места не занимает */
char data_buf[1 << 10] = {1};          /* 1 КБ — в .data, лежит в файле */
const char ro_str[] = "только чтение"; /* .rodata */

static void at_exit_hook(void) { puts("atexit: вызвано при exit()"); }

int main(int argc, char **argv, char **envp)
{
    int local = 0;
    char *small = malloc(100);              /* из кучи (brk) */
    char *big = malloc(256 * 1024);         /* больше M_MMAP_THRESHOLD (128 К) — отдельное mmap */
    printf("text   main      %p\n", (void *)main);
    printf("rodata ro_str    %p\n", (void *)ro_str);
    printf("data   data_buf  %p\n", (void *)data_buf);
    printf("bss    bss_buf   %p\n", (void *)bss_buf);
    printf("heap   malloc100 %p\n", (void *)small);
    printf("mmap   malloc256K %p\n", (void *)big);
    printf("stack  local     %p\n", (void *)&local);
    printf("brk    sbrk(0)   %p\n", sbrk(0));
    printf("argv0 %s argc %d envp[0] %.20s...\n", argv[0], argc, envp[0]);
    printf("PATH через getenv: %s\n", getenv("PATH") ? "есть" : "нет");
    struct rlimit rl;
    getrlimit(RLIMIT_STACK, &rl);
    printf("RLIMIT_STACK мягкий %ld КБ\n", (long)(rl.rlim_cur / 1024));
    getrlimit(RLIMIT_NOFILE, &rl);
    printf("RLIMIT_NOFILE %ld\n", (long)rl.rlim_cur);
    atexit(at_exit_hook);
    /* page faults: тронуть 1 МБ bss — каждая страница даёт minor fault */
    struct rusage ru0, ru1;
    getrusage(RUSAGE_SELF, &ru0);
    memset(bss_buf, 1, sizeof bss_buf);
    getrusage(RUSAGE_SELF, &ru1);
    printf("minor faults за memset 1 МБ: %ld (страниц по 4 КБ: %d)\n",
           ru1.ru_minflt - ru0.ru_minflt, (int)(sizeof bss_buf / 4096));
    if (argc > 1 && strcmp(argv[1], "maps") == 0) {     /* карта памяти процесса от ядра */
        FILE *f = fopen("/proc/self/maps", "r"); char line[256];
        while (fgets(line, sizeof line, f))
            if (strstr(line, "proc_layout") || strstr(line, "[heap]") || strstr(line, "[stack]") || strstr(line, "[v"))
                printf("maps %s", line);
        fclose(f);
    }
    free(big); free(small);
    if (argc > 1 && strcmp(argv[1], "_exit") == 0) _exit(3);   /* без atexit и без сброса буферов */
    return argc > 1 ? atoi(argv[1]) : 0;
}

Next: section 136 — fork, exec, wait and signals.