147. The Linux kernel: building, a module, a character device, ioctl

Everything earlier in this part is user-mode programs: they ask the kernel for things through system calls. A driver lives on the other side: inside the kernel, with its memory, without libc, without protection from its own mistakes. This section shows what the kernel is made of as a project, how to build a module for it, and what a driver for the simplest device looks like — a ring buffer available through /dev/ringdev, with read, write and ioctl.

The module is built against the Debian 6.1 kernel headers in stm-sys:1 (_verify/sys147/) — the build, modinfo, the symbols and the sections are real. It cannot be loaded into the container's kernel: the host kernel is a different version (6.6, WSL2) and there are no privileges, so insmod answers Operation not permitted. Loading is done on a real machine or a virtual one with the same kernel.

Terms used in the documentation

Term	Meaning
kernel space, user space	the kernel's address space, the user's address space
loadable kernel module (.ko)	a kernel module loaded at run time
Kbuild, obj-m	the kernel build system, the "module" target
vermagic	a fingerprint of the version and configuration of the kernel the module was built for
character device, major/minor	a byte-stream device, the major/minor number
file_operations	the table of operations on the device file
copy_to_user, copy_from_user	copying between the kernel and the user
ioctl	a command to the device outside read/write
printk, dmesg	kernel printing and its log
tainted kernel	a kernel "tainted" by a non-free or faulty module
ftrace, kprobes	tracing of kernel functions

147.1 The kernel is a program

The kernel is one big executable written in C: vmlinux is an ordinary ELF, bzImage is the same thing compressed for the bootloader. The kernel is always in memory, always running, and there is one for everyone: it schedules processes, shares out memory, talks to the hardware. Everything else — the drivers — is either compiled into it or loaded as .ko modules on demand.

The source tree: arch/ — processors, drivers/ — drivers (more than half of the whole volume), fs/ — file systems, net/ — networking, mm/ — memory, kernel/ — the scheduler and system calls, include/ — headers. include/linux alone holds 1464 headers; fs.h is 3776 lines long. The kernel runs without libc: its own kmalloc instead of malloc, printk instead of printf, mutex_lock instead of pthread.

Building the whole kernel: make defconfig or make localmodconfig (only the modules loaded right now — the build is many times shorter), editing .config with make menuconfig, make -j$(nproc); on Debian, make deb-pkg and dpkg -i. A module is built without building the kernel — against its headers.

147.2 A module: entry, exit, parameters

A module is a .ko file with two mandatory points: what to do on load and on unload:

module_init(ring_init);
module_exit(ring_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Кольцевой буфер как символьное устройство");
MODULE_VERSION("1.0");

A parameter is set at load time — insmod ringdev.ko ring_size=256:

static int ring_size = 64;                              /* параметр: insmod ringdev.ko ring_size=256 */
module_param(ring_size, int, 0444);
MODULE_PARM_DESC(ring_size, "размер кольцевого буфера в байтах");

The build is three lines of Makefile; the kernel's Kbuild does everything else:

obj-m := ringdev.o
KDIR ?= /lib/modules/$(shell uname -r)/build
all:
    $(MAKE) -C $(KDIR) M=$(PWD) modules

The finished module is read with modinfo:

version:        1.0
description:    Кольцевой буфер как символьное устройство
license:        GPL
name:           ringdev
vermagic:       6.1.0-53-amd64 SMP preempt mod_unload modversions 
parm:           ring_size:размер кольцевого буфера в байтах (int)

vermagic is the key: the kernel configuration (SMP preempt mod_unload modversions) must match. With modversions the number 6.1.0-53-amd64 is not compared — instead the kernel checks the checksum of every symbol the module uses. Kernel 6.6 will reject it: the checksums will differ, and dmesg will show disagrees about version of symbol …. That is why modules are built separately for each kernel (DKMS does this on updates). The .ko file is 279672 bytes, of which code and data are 14080 (strip --strip-debug); the rest is debug information.

The module's sections: .init.text — the code of ring_init, discarded after loading; .exit.text — the unload code; .modinfo — what modinfo prints; __param — the parameters. The symbols the module requires from the kernel (nm -u): alloc_chrdev_region, cdev_add, _copy_to_user, _printk, __kmalloc, mutex_lock, device_create… Most of them are exported for any module, and only four — device_create, device_destroy, class_destroy and __class_create (behind it is the class_create macro) — are marked EXPORT_SYMBOL_GPL: for a non-free module they do not exist. A module with a different license will not even build — modpost stops the build with the message "GPL-incompatible module … uses GPL-only symbol". The export type of each symbol is visible in Module.symvers next to the kernel headers.

147.3 A character device and file_operations

A device in Unix is a file in /dev. In a character device the bytes go as a stream (a terminal, a port, our buffer); in a block device they go in blocks with random access (a disk). The file corresponds to a pair of numbers: the major selects the driver, the minor selects the device inside the driver. The driver gives the kernel a table of operations — the same technique as the function table in section 146:

static const struct file_operations ring_fops = {       /* «класс» устройства: таблица функций (раздел 146) */
    .owner = THIS_MODULE,
    .read = ring_read,
    .write = ring_write,
    .unlocked_ioctl = ring_ioctl,
};

In kernel 6.1, struct file_operations is declared in fs.h on line 2174 and starts with owner, llseek, read, write; in total it has several dozen pointers, and the ones left empty mean "operation not supported" (-EINVAL for read on a device without .read).

Registration takes four steps, with rollback on any error:

    rc = alloc_chrdev_region(&devno, 0, 1, "ringdev");   /* ядро выдаёт старший номер */
    if (rc) goto err_buf;
    cdev_init(&cdev, &ring_fops);
    rc = cdev_add(&cdev, devno, 1);
    if (rc) goto err_region;

    device_create(cls, NULL, devno, NULL, "ringdev");     /* udev создаст /dev/ringdev */

goto err_… is the kernel style for releasing in reverse order: the kernel has no exceptions and no second chance. class_create + device_create tell udev about the new device, and the file /dev/ringdev appears by itself; otherwise you would have to make it with mknod using the numbers from /proc/devices.

147.4 read, write, ioctl

The user calls read(fd, buf, n) — the kernel calls ring_read with the same arguments, but buf points into someone else's address space. You cannot write there directly: only through copy_to_user, which checks the address and returns how many bytes were not copied:

static ssize_t ring_read(struct file *f, char __user *ubuf, size_t n, loff_t *off)
{
    size_t done = 0;
    mutex_lock(&lock);
    while (done < n && count > 0) {
        if (copy_to_user(ubuf + done, &buf[tail], 1)) { mutex_unlock(&lock); return -EFAULT; }   /* в пользователя — только copy_to_user */
        tail = (tail + 1) % ring_size; count--; done++;
    }
    mutex_unlock(&lock);
    return done;                                        /* 0 — «конец файла» для cat */
}

The __user annotation is for the static analyzer (sparse): mixing kernel and user pointers is an error. The mutex is there because read and write can come from two processes at once; races in the kernel are the same as in section 139, only the consequence is a panic of the whole machine. It has to be a mutex, not a spinlock: copy_to_user may sleep while the kernel pages in a user page, and you must not sleep under a spinlock. Here an empty buffer returns 0 — "end of file"; real drivers at this point put the reader to sleep with wait_event_interruptible and wake it up from write, like a pipe. The return value: the number of bytes, 0 for the end, a negative errno for an error. write to a full buffer returns -ENOSPC, not 0: with zero, echo would retry the write forever.

ioctl is for commands that do not fit into a stream of bytes: clear the buffer, find out how full it is. The command number encodes the direction, a "magic" number, a sequence number and the data size, and the header with them is shared by the module and the program:

#define RING_IOC_MAGIC 'R'
#define RING_IOC_CLEAR _IO(RING_IOC_MAGIC, 0)              /* без данных */
#define RING_IOC_COUNT _IOR(RING_IOC_MAGIC, 1, int)        /* ядро → пользователь: сколько байт в буфере */
#define RING_IOC_SIZE  _IOR(RING_IOC_MAGIC, 2, int)

RING_IOC_CLEAR = 0x5200, RING_IOC_COUNT = 0x80045201, RING_IOC_SIZE = 0x80045202

0x52 is the code of the letter R; the top bit 0x8000… means "read from the kernel", 04 means four bytes of data. The driver rejects an unknown command with -ENOTTY. The user side is the usual open, write, ioctl(fd, RING_IOC_COUNT, &n), read.

147.5 Loading, the log, debugging

printk prints to the kernel log with a level: pr_err, pr_warn, pr_info, pr_debug are wrappers around printk(KERN_ERR …); the eight levels from KERN_EMERG to KERN_DEBUG are a prefix string before the message. pr_debug prints only if the module is built with -DDEBUG (ccflags-y += -DDEBUG in the Makefile) or the messages are enabled on the fly: echo 'module ringdev +p' > /sys/kernel/debug/dynamic_debug/control. The module's message ringdev: major N, буфер 64 байт (pr_info in ring_init) appears in dmesg on load.

A debugger is attached to a running kernel differently than to a process. The kernel in QEMU is started with the -s -S flags — a debugger server on port 1234, the processor halted until the c command — and with nokaslr on the kernel command line, otherwise the addresses will not match vmlinux. Then gdb vmlinux (the kernel with debug information) after target remote :1234 sets breakpoints both in the kernel and in the module. The module symbols are loaded by the lx-symbols command from scripts/gdb — for it the kernel is built with CONFIG_GDB_SCRIPTS. On hardware, kgdb does the same over a serial port (CONFIG_KGDB). In the Debian kernel both options are disabled, so people usually do without a debugger: printk, ftrace (/sys/kernel/tracing: which kernel function was called how many times and how long it took), kprobes (a breakpoint on a kernel function with a handler), perf. A bug in a module means an Oops with a stack trace in dmesg, and often a hang of the whole machine: modules are debugged in a virtual machine, not on your working computer.

Secure Boot. The Debian kernel is built with CONFIG_LOCK_DOWN_IN_EFI_SECURE_BOOT: if Secure Boot is enabled on the machine, it goes into lockdown mode and refuses an unsigned module even from root — insmod answers with the same Operation not permitted as in the container, and in dmesg the kernel writes that lockdown prohibited loading. A kernel that always requires a signature (module.sig_enforce=1) answers Key was rejected by service. You sign the module with your own key — a pair made with openssl req -x509:

/usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 key.priv key.der ringdev.ko

After that modinfo shows sig_hashalgo: sha256, and the key is registered in the firmware: mokutil --import key.der and a confirmation at the next reboot. For experiments a virtual machine without Secure Boot is simpler.

147.6 Pitfalls

147.7 Summary

• The kernel is an ELF without libc; a .ko module is built by Kbuild against the headers of its kernel, and vermagic ties it to the version.
• module_init/module_exit, module_param, MODULE_LICENSE("GPL").
• A character device: alloc_chrdev_region → cdev_add → class_create → device_create; file_operations is a table of functions.
• User data only through copy_to_user/copy_from_user; read returns 0 at the end, write returns -ENOSPC on a full buffer.
• ioctl: _IO/_IOR/_IOW, a shared header; an unknown command gets -ENOTTY.
• insmod/modprobe/rmmod/dmesg; debugging with printk, ftrace, gdb attached to the kernel in QEMU, in a virtual machine; with Secure Boot, only a signed module.

Full sources

ringdev.h — the ioctl commands — a header shared by the module and the program.

/* Раздел 147: команды ioctl — общий заголовок модуля и пользовательской программы. */
#ifndef RINGDEV_H
#define RINGDEV_H
#include <linux/ioctl.h>
#define RING_IOC_MAGIC 'R'
#define RING_IOC_CLEAR _IO(RING_IOC_MAGIC, 0)              /* без данных */
#define RING_IOC_COUNT _IOR(RING_IOC_MAGIC, 1, int)        /* ядро → пользователь: сколько байт в буфере */
#define RING_IOC_SIZE  _IOR(RING_IOC_MAGIC, 2, int)
#endif

ringdev.c — the kernel module: a character device with a ring buffer.

/* Раздел 147: модуль ядра — символьное устройство /dev/ringdev с кольцевым буфером:
   read/write/ioctl, параметр модуля, регистрация через cdev и класс устройств.
   Собирается против заголовков ядра; загрузка требует настоящего ядра той же версии. */
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/uaccess.h>
#include <linux/mutex.h>
#include <linux/version.h>
#include "ringdev.h"

static int ring_size = 64;                              /* параметр: insmod ringdev.ko ring_size=256 */
module_param(ring_size, int, 0444);
MODULE_PARM_DESC(ring_size, "размер кольцевого буфера в байтах");

static char *buf;
static size_t head, tail, count;
static DEFINE_MUTEX(lock);
static dev_t devno;
static struct cdev cdev;
static struct class *cls;

static ssize_t ring_read(struct file *f, char __user *ubuf, size_t n, loff_t *off)
{
    size_t done = 0;
    mutex_lock(&lock);
    while (done < n && count > 0) {
        if (copy_to_user(ubuf + done, &buf[tail], 1)) { mutex_unlock(&lock); return -EFAULT; }   /* в пользователя — только copy_to_user */
        tail = (tail + 1) % ring_size; count--; done++;
    }
    mutex_unlock(&lock);
    return done;                                        /* 0 — «конец файла» для cat */
}

static ssize_t ring_write(struct file *f, const char __user *ubuf, size_t n, loff_t *off)
{
    size_t done = 0;
    mutex_lock(&lock);
    while (done < n && count < (size_t)ring_size) {
        if (copy_from_user(&buf[head], ubuf + done, 1)) { mutex_unlock(&lock); return -EFAULT; }
        head = (head + 1) % ring_size; count++; done++;
    }
    mutex_unlock(&lock);
    return done ? done : -ENOSPC;                       /* полон — ошибка, а не 0 (иначе echo зациклится) */
}

static long ring_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
    switch (cmd) {
    case RING_IOC_CLEAR: mutex_lock(&lock); head = tail = count = 0; mutex_unlock(&lock); return 0;
    case RING_IOC_COUNT: return put_user((int)count, (int __user *)arg);
    case RING_IOC_SIZE:  return put_user(ring_size, (int __user *)arg);
    }
    return -ENOTTY;                                     /* чужая команда */
}

static const struct file_operations ring_fops = {       /* «класс» устройства: таблица функций (раздел 146) */
    .owner = THIS_MODULE,
    .read = ring_read,
    .write = ring_write,
    .unlocked_ioctl = ring_ioctl,
};

static int __init ring_init(void)
{
    int rc;
    buf = kmalloc(ring_size, GFP_KERNEL);
    if (!buf) return -ENOMEM;
    rc = alloc_chrdev_region(&devno, 0, 1, "ringdev");   /* ядро выдаёт старший номер */
    if (rc) goto err_buf;
    cdev_init(&cdev, &ring_fops);
    rc = cdev_add(&cdev, devno, 1);
    if (rc) goto err_region;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 4, 0)
    cls = class_create("ringdev");                       /* с 6.4 — без владельца */
#else
    cls = class_create(THIS_MODULE, "ringdev");
#endif
    if (IS_ERR(cls)) { rc = PTR_ERR(cls); goto err_cdev; }
    device_create(cls, NULL, devno, NULL, "ringdev");     /* udev создаст /dev/ringdev */
    pr_info("ringdev: major %d, буфер %d байт\n", MAJOR(devno), ring_size);
    return 0;
err_cdev:   cdev_del(&cdev);
err_region: unregister_chrdev_region(devno, 1);
err_buf:    kfree(buf);
    return rc;
}

static void __exit ring_exit(void)
{
    device_destroy(cls, devno);
    class_destroy(cls);
    cdev_del(&cdev);
    unregister_chrdev_region(devno, 1);
    kfree(buf);
    pr_info("ringdev: выгружен\n");
}

module_init(ring_init);
module_exit(ring_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Кольцевой буфер как символьное устройство");
MODULE_VERSION("1.0");

Next: section 148 — databases from C.