138. A daemon and its log: setsid, syslog, lock file, backtrace, core

A program that must run all the time — a server, a data collector for instruments, a bridge between a serial port and the network — cannot depend on the terminal it was started from. Such a process is called a daemon: it lives in the background, without a terminal, writes to a log instead of the screen, and can reread its settings and stop cleanly on a signal. The second half of the section covers what a daemon is blind without: your own logging library with a call stack on errors, and post-mortem debugging from a core file.

The daemon and the logger in this section were written and run in stm-sys:1 (_verify/sys138/): ps shows that the daemon has no terminal, /proc shows that its descriptors point to /dev/null, and a second instance refuses to start.

Terms used in the documentation

Term	Meaning
daemon	a background process without a terminal
controlling terminal	the terminal a session is attached to
session, session leader	a session and the process that leads it
process group	a group of processes signalled together
double fork	calling fork twice
pid file, lock file	a file with the process number; a file that is locked
advisory lock	a lock that only cooperating processes honour (fcntl)
syslog, facility, priority	the system log, the message source, the severity
reload on SIGHUP	reread the settings on SIGHUP
unit file	a systemd service description file
backtrace	the call stack
core dump, post-mortem debugging	a memory image of the process; debugging after the crash

138.1 Session, group, terminal

Every process has three kinship numbers: pid, the process group pgid and the session sid. The shell keeps all processes of one pipeline in one group, and all its groups in one session, which has a controlling terminal. Ctrl-C sends SIGINT to the whole foreground group; closing the terminal sends SIGHUP to the session leader and on to everyone. A program started from a terminal and simply left there dies together with it.

A daemon must leave this hierarchy: become the leader of its own session, which has no terminal and never will.

138.2 Seven steps of daemonization

static void daemonize(void)
{
    umask(022);                                          /* 1. маска прав — известная и безопасная */
    pid_t pid = fork();                                  /* 2. первый fork: оболочка получает управление */
    if (pid < 0) exit(1);
    if (pid > 0) exit(0);                                /*    родитель уходит — потомок становится сиротой */
    setsid();                                            /* 3. новый сеанс: без управляющего терминала */
    signal(SIGHUP, SIG_IGN);
    pid = fork();                                        /* 4. второй fork: не лидер сеанса — терминал не получить */
    if (pid > 0) exit(0);
    chdir("/");                                          /* 5. корень: не держать смонтированный каталог */
    struct rlimit rl; getrlimit(RLIMIT_NOFILE, &rl);
    if (rl.rlim_max == RLIM_INFINITY) rl.rlim_max = 1024;
    for (int i = 0; i < (int)rl.rlim_max; i++) close(i); /* 6. закрыть всё унаследованное */
    open("/dev/null", O_RDWR); dup(0); dup(0);           /* 7. 0, 1, 2 → /dev/null */
}

Why each step is there:

1. umask(022) — a known and safe permission mask: otherwise the files the daemon creates get permissions inherited from who knows where. umask(0) is also "known", but then fopen creates files with mode 0666, and any user can write to them — and a daemon often runs as root. If others must not even read them, use umask(027).
2. First fork — the parent exits, and the shell thinks the command has finished; the child is not a group leader, which is the condition for setsid.
3. setsid — the process becomes the leader of a new session and a new group and loses its controlling terminal: pid = sid = pgid.
4. Second fork — a session leader can acquire a terminal again by opening a terminal device; a non-leader cannot. The library function daemon(3) in glibc skips this step, and its daemon stays a session leader.
5. chdir("/") — a process sitting in a directory on a removable disk prevents it from being unmounted.
6. Close descriptors — inherited files and sockets hold resources.
7. 0, 1, 2 to /dev/null — so that a forgotten printf does not fail with an error and does not write into someone else's file opened as number 1.

The result is visible in ps: the daemon with pid 52 is a child of init (ppid 1), session 51, no terminal (?), state S:

   52     1    51    51 ?        S    daemon

Its descriptors:

0 -> /dev/null
1 -> /dev/null
2 -> /dev/null
3 -> /tmp/demo-daemon.pid
cwd: /

nohup and setsid from the shell do part of the same work for someone else's program; systemd (138.5) does all of it, and a modern daemon does not need to daemonize itself — it needs a switch to skip doing so (here -f).

138.3 One instance: the lock file

Two daemon instances on one port or one instrument are trouble. The classic approach: a file /var/run/name.pid holding the process number, with an advisory lock on it:

static int already_running(void)
{
    int fd = open(LOCKFILE, O_RDWR | O_CREAT, 0644);
    if (fd < 0) return -1;
    struct flock fl = { .l_type = F_WRLCK, .l_whence = SEEK_SET };
    if (fcntl(fd, F_SETLK, &fl) < 0) {                   /* блокировка снимается ядром при смерти процесса */
        if (errno == EACCES || errno == EAGAIN) { close(fd); return 1; }
        return -1;
    }
    ftruncate(fd, 0);
    dprintf(fd, "%ld\n", (long)getpid());
    return 0;                                            /* дескриптор не закрываем — блокировка живёт с ним */
}

Why a lock and not just "the file exists": after the daemon crashes the file remains, and an existence check will never let a new instance start. The kernel releases an fcntl lock by itself when the process dies. The second instance from the run:

demo-daemon[60]: уже запущен, файл /tmp/demo-daemon.pid занят
rc=1

The lock file descriptor is never closed: a close of any descriptor of this file in the process releases its locks — a quirk of fcntl that flock does not have.

138.4 Logging through syslog and reloading on SIGHUP

A daemon has no stderr, and its log is syslog:

    openlog("demo-daemon", LOG_PID | (foreground ? LOG_PERROR : 0), LOG_DAEMON);

openlog sets the name, the options (LOG_PID — add the process number, LOG_PERROR — duplicate to stderr) and the facility (LOG_DAEMON, LOG_USER, LOG_LOCAL0…LOG_LOCAL7), which the logging service uses to sort records into files. Every record has a priority, from LOG_EMERG to LOG_DEBUG; %m in the format string inserts strerror(errno):

    if (!f) { syslog(LOG_WARNING, "нет файла настроек %s: %m", path); return; }

Records go to the /dev/log socket, where rsyslog, syslog-ng or journald picks them up and stores them in /var/log or in journalctl. In a container there is no logging service — the /dev/log socket does not exist, and the records are lost; that is why the daemon in the run is started with LOG_PERROR.

The daemon rereads its settings on SIGHUP and stops on SIGTERM. The handler only sets a flag (section 136), and the main loop checks it:

    while (!stop) {
        if (reload) { reload = 0; read_config(cfg); }
        syslog(LOG_DEBUG, "тик %d", ++tick);
        sleep(period);
    }
    syslog(LOG_INFO, "остановлен по сигналу после %d тиков", tick);
    unlink(LOCKFILE);

Good daemons take their settings in one order of precedence: command-line arguments override environment variables, those override the user's file, that overrides the system file, and at the bottom are sensible defaults. File formats and libraries for them: INI (iniparser), JSON (jansson, cJSON), YAML (libyaml), and for complex cases, Lua.

138.5 Starting at boot: systemd

Daemons used to be started by /etc/init.d/name scripts with start-stop-daemon; today it is systemd almost everywhere, and a service is described by a unit file:

[Unit]
Description=Demo daemon
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/demo-daemon -f /etc/demo-daemon.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

Type=simple — the program does not daemonize; systemd itself keeps it in the background, tracks its pid, collects stdout and stderr into the journal and restarts it after a crash. Type=forking is for old daemons with a double fork and PIDFile=. So a modern daemon must be able to run in the foreground: the -f switch in the example is exactly the mode for systemd. Commands: systemctl start|stop|reload|status name, the journal — journalctl -u name.

138.6 Your own logging library

syslog is fine for a daemon, but an application often needs its own log: to a file, with levels, with the location in the code, thread-safe and working even when memory has run out. The interface is macros that insert the file, line and function:

#define LOG(lvl, ...) log_write(lvl, __FILE__, __LINE__, __func__, __VA_ARGS__)
#define LOG_D(...) LOG(LOG_DEBUG, __VA_ARGS__)
#define LOG_I(...) LOG(LOG_INFO, __VA_ARGS__)
#define LOG_W(...) LOG(LOG_WARN, __VA_ARGS__)
#define LOG_E(...) LOG(LOG_ERROR, __VA_ARGS__)

Three design decisions inside log_write:

• Nothing from the heap. The line is assembled in a stack buffer char buf[1024] with snprintf/vsnprintf and goes out in a single write. The log must work when malloc has returned NULL — that is exactly when it is needed most.
• A mutex around the write. One line written with one write to a regular file is not split even without it. But an error is a line plus a stack of several write calls, and without a mutex lines from other threads get in between. Four threads of a hundred lines each write to one file, and everything stays in order.
• A call stack on errors. backtrace collects frame addresses, backtrace_symbols_fd writes them straight to the descriptor — also without malloc. With one caveat: on the first call backtrace loads the stack unwinding library libgcc_s, and loading it calls malloc itself — in the experiment five times, while the second call made none. So log_init makes one dummy backtrace call: at the moment of a crash the library is already in place.

    pthread_mutex_lock(&log_mx);                          /* строки разных потоков не перемешиваются */
    write(log_fd, buf, n);
    if (lvl == LOG_ERROR) {                               /* при ошибке — стек вызовов */
        void *frames[32];
        int depth = backtrace(frames, 32);
        backtrace_symbols_fd(frames, depth, log_fd);      /* пишет прямо в дескриптор, без malloc */
    }
    pthread_mutex_unlock(&log_mx);

The run: 412 lines, of which 401 are INFO (one from main and 400 from four threads), and one error with a stack of eight frames (the first five below; after them come two libc frames and _start):

2026-09-13 15:41:57 DEBUG log_test.c:21 main(): отладочная строка
2026-09-13 15:41:57 INFO  log_test.c:22 main(): старт, аргументов 1
2026-09-13 15:41:57 WARN  log_test.c:23 main(): предупреждение
2026-09-13 15:41:57 ERROR log_test.c:14 deep3(): что-то пошло не так: malloc вернул NULL
./log_test(log_write+0x238)[0x5e9e690876da]
./log_test(+0x12b6)[0x5e9e690872b6]
./log_test(+0x12c2)[0x5e9e690872c2]
./log_test(+0x12ce)[0x5e9e690872ce]
./log_test(main+0x12d)[0x5e9e690873fe]

Only functions visible to the linker have names in the stack: log_write and main. The static deep1–deep3 appear as offsets — their names are recovered by addr2line -e log_test 0x12b6 from the debug information (-g). The -rdynamic build option puts the names of external functions into the dynamic symbol table; without it even main would be an offset. With the threshold at LOG_INFO the file contains 0 DEBUG lines.

138.7 Post-mortem debugging: core and gdb

When a daemon crashes at night, the stack from the log is not enough — you need a memory image. ulimit -c unlimited allows the kernel to write core on SIGSEGV, SIGABRT and other signals whose action is "core"; where and under what name is set by /proc/sys/kernel/core_pattern. Then:

gdb -q -batch -ex bt ./log_test core

prints the stack at the moment of the crash, and interactive gdb gives you frame N, print variable, info locals — everything as in live debugging, except the process is already dead. For this the executable is built with -g and the debug information is not stripped: for a release it is split off into a .debug file (objcopy --only-keep-debug) and stored next to the version.

In the container from the run, core_pattern points to the host's crash handler program (|/wsl-capture-crash …), and no core file appears in the current directory — the crash is visible only from exit code 139 (128 + 11). systemd-coredump behaves the same way on ordinary distributions: the dumps are kept in /var/lib/systemd/coredump, and you read them with coredumpctl gdb.

138.8 Pitfalls

138.9 Summary

• Daemonization: umask, fork, setsid, fork, chdir("/"), close descriptors, 0/1/2 to /dev/null; the result is ppid 1, terminal ?.
• One instance: an fcntl lock on the pid file; the kernel releases it when the process dies.
• A daemon's log is syslog with a facility and a priority; SIGHUP — reread, SIGTERM — stop.
• systemd: Type=simple, the program runs in the foreground.
• Your own library: macros with __FILE__/__LINE__, a stack buffer, a mutex, backtrace_symbols_fd on errors; -rdynamic for names.
• ulimit -c, core_pattern, gdb -ex bt — post-mortem debugging.

Full sources

daemon.c — step-by-step daemonization, lock file, syslog and rereading the settings on SIGHUP.

/* Раздел 138: демонизация по шагам, файл блокировки, syslog, перечитывание настроек по SIGHUP.
   Запуск: ./daemon <файл настроек>; ./daemon -f — в переднем плане, без демонизации. */
#define _DEFAULT_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <syslog.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/resource.h>

#define LOCKFILE "/tmp/demo-daemon.pid"
static volatile sig_atomic_t reload, stop;
static int period = 1;

static void on_sig(int s) { if (s == SIGHUP) reload = 1; else stop = 1; }

static void daemonize(void)
{
    umask(022);                                          /* 1. маска прав — известная и безопасная */
    pid_t pid = fork();                                  /* 2. первый fork: оболочка получает управление */
    if (pid < 0) exit(1);
    if (pid > 0) exit(0);                                /*    родитель уходит — потомок становится сиротой */
    setsid();                                            /* 3. новый сеанс: без управляющего терминала */
    signal(SIGHUP, SIG_IGN);
    pid = fork();                                        /* 4. второй fork: не лидер сеанса — терминал не получить */
    if (pid > 0) exit(0);
    chdir("/");                                          /* 5. корень: не держать смонтированный каталог */
    struct rlimit rl; getrlimit(RLIMIT_NOFILE, &rl);
    if (rl.rlim_max == RLIM_INFINITY) rl.rlim_max = 1024;
    for (int i = 0; i < (int)rl.rlim_max; i++) close(i); /* 6. закрыть всё унаследованное */
    open("/dev/null", O_RDWR); dup(0); dup(0);           /* 7. 0, 1, 2 → /dev/null */
}

static int already_running(void)
{
    int fd = open(LOCKFILE, O_RDWR | O_CREAT, 0644);
    if (fd < 0) return -1;
    struct flock fl = { .l_type = F_WRLCK, .l_whence = SEEK_SET };
    if (fcntl(fd, F_SETLK, &fl) < 0) {                   /* блокировка снимается ядром при смерти процесса */
        if (errno == EACCES || errno == EAGAIN) { close(fd); return 1; }
        return -1;
    }
    ftruncate(fd, 0);
    dprintf(fd, "%ld\n", (long)getpid());
    return 0;                                            /* дескриптор не закрываем — блокировка живёт с ним */
}

static void read_config(const char *path)
{
    FILE *f = fopen(path, "r");
    if (!f) { syslog(LOG_WARNING, "нет файла настроек %s: %m", path); return; }
    if (fscanf(f, "period=%d", &period) != 1) period = 1;
    fclose(f);
    syslog(LOG_INFO, "настройки прочитаны: period=%d", period);
}

int main(int argc, char **argv)
{
    int foreground = argc > 1 && strcmp(argv[1], "-f") == 0;
    const char *cfg = argc > 2 ? argv[2] : "/tmp/demo-daemon.conf";
    if (!foreground) daemonize();
    openlog("demo-daemon", LOG_PID | (foreground ? LOG_PERROR : 0), LOG_DAEMON);
    if (already_running()) { syslog(LOG_ERR, "уже запущен, файл %s занят", LOCKFILE); return 1; }
    struct sigaction sa = { .sa_handler = on_sig };
    sigemptyset(&sa.sa_mask);
    sigaction(SIGHUP, &sa, NULL); sigaction(SIGTERM, &sa, NULL); sigaction(SIGINT, &sa, NULL);
    read_config(cfg);
    syslog(LOG_INFO, "старт: pid %ld sid %ld pgid %ld", (long)getpid(), (long)getsid(0), (long)getpgid(0));
    int tick = 0;
    while (!stop) {
        if (reload) { reload = 0; read_config(cfg); }
        syslog(LOG_DEBUG, "тик %d", ++tick);
        sleep(period);
    }
    syslog(LOG_INFO, "остановлен по сигналу после %d тиков", tick);
    unlink(LOCKFILE);
    return 0;
}

logger.h — the interface of the logging library.

/* Раздел 138: интерфейс журнала. Макросы подставляют файл, строку и функцию. */
#ifndef LOGGER_H
#define LOGGER_H
#include <fcntl.h>

enum log_level { LOG_DEBUG, LOG_INFO, LOG_WARN, LOG_ERROR };

int  log_init(const char *path, enum log_level min);      /* path == NULL — в stderr */
void log_write(enum log_level lvl, const char *file, int line, const char *func, const char *fmt, ...)
     __attribute__((format(printf, 5, 6)));
void log_close(void);

#define LOG(lvl, ...) log_write(lvl, __FILE__, __LINE__, __func__, __VA_ARGS__)
#define LOG_D(...) LOG(LOG_DEBUG, __VA_ARGS__)
#define LOG_I(...) LOG(LOG_INFO, __VA_ARGS__)
#define LOG_W(...) LOG(LOG_WARN, __VA_ARGS__)
#define LOG_E(...) LOG(LOG_ERROR, __VA_ARGS__)
#endif

logger.c — the logging library: levels, location in the code, call stack, mutex.

/* Раздел 138: библиотека журналирования — уровни, место в коде, стек вызовов при ошибке,
   потокобезопасность через мьютекс, пригодность при нехватке памяти (без malloc). */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdarg.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <pthread.h>
#include <execinfo.h>
#include "logger.h"

static int log_fd = -1;
static enum log_level log_min = LOG_DEBUG;
static pthread_mutex_t log_mx = PTHREAD_MUTEX_INITIALIZER;
static const char *names[] = { "DEBUG", "INFO", "WARN", "ERROR" };

int log_init(const char *path, enum log_level min)
{
    log_fd = path ? open(path, O_WRONLY | O_CREAT | O_APPEND, 0644) : STDERR_FILENO;
    log_min = min;
    void *warm[1];
    backtrace(warm, 1);                                   /* подгрузить libgcc_s сейчас: первый вызов просит malloc */
    return log_fd < 0 ? -1 : 0;
}

void log_write(enum log_level lvl, const char *file, int line, const char *func, const char *fmt, ...)
{
    if (lvl < log_min || log_fd < 0) return;
    char buf[1024];                                       /* всё на стеке: malloc может быть недоступен */
    time_t now = time(NULL);
    struct tm tm; localtime_r(&now, &tm);
    int n = strftime(buf, sizeof buf, "%Y-%m-%d %H:%M:%S ", &tm);
    n += snprintf(buf + n, sizeof buf - n, "%-5s %s:%d %s(): ", names[lvl], file, line, func);
    va_list ap; va_start(ap, fmt);
    n += vsnprintf(buf + n, sizeof buf - n, fmt, ap);
    va_end(ap);
    if (n > (int)sizeof buf - 2) n = sizeof buf - 2;
    buf[n++] = '\n';
    pthread_mutex_lock(&log_mx);                          /* строки разных потоков не перемешиваются */
    write(log_fd, buf, n);
    if (lvl == LOG_ERROR) {                               /* при ошибке — стек вызовов */
        void *frames[32];
        int depth = backtrace(frames, 32);
        backtrace_symbols_fd(frames, depth, log_fd);      /* пишет прямо в дескриптор, без malloc */
    }
    pthread_mutex_unlock(&log_mx);
}

void log_close(void) { if (log_fd > 2) close(log_fd); log_fd = -1; }

Next: section 139 — POSIX threads and synchronization.