141. Non-blocking I/O: select, poll, epoll, event descriptors

The server from section 78 served one connection at a time: read waits for data, and while one client is silent, all the others are stuck. A thread per connection (section 139) solves this for up to a few hundred clients, and beyond that it runs into the memory for stacks and into context switches. The third way is one thread and a kernel that tells you which descriptors are ready: non-blocking sockets and multiplexing. nginx, Redis and Node.js are built on it.

Here one echo server is written three times — on select, poll and epoll — and run under a load of 500 and 2000 connections (_verify/sys141/, stm-sys:1); after that come a timer, an event and a signal as descriptors, and what lies behind the words "reactor", io_uring and "coroutines".

Terms used in the documentation

Term	Meaning
blocking, non-blocking I/O	a call that waits, a call that returns at once
EAGAIN / EWOULDBLOCK	"nothing right now" — try again later
I/O multiplexing	waiting on many descriptors at once
readiness notification	a notice that a descriptor is ready
level-triggered, edge-triggered	reported while the state holds, reported once on a change
FD_SETSIZE	the select limit — 1024 descriptors
interest list, ready list	the watched list and the ready list (epoll)
timerfd, eventfd, signalfd	a timer, an event, a signal as a descriptor
C10K problem	the problem of ten thousand connections
reactor pattern	an event loop with handlers
completion-based I/O	I/O reported on completion (io_uring, IOCP)
coroutine	a function that can suspend and resume

141.1 Non-blocking mode

By default read, write, accept and connect wait. The O_NONBLOCK flag changes the contract: if there is nothing to do, the call returns −1 with errno == EAGAIN immediately. The exception is connect: it returns EINPROGRESS, and you learn the result of the connection when the socket becomes ready for writing, via getsockopt(fd, SOL_SOCKET, SO_ERROR, …):

static int set_nonblocking(int fd)
{
    int fl = fcntl(fd, F_GETFL, 0);
    return fcntl(fd, F_SETFL, fl | O_NONBLOCK);
}

неблокирующий read без данных: -1, errno Resource temporarily unavailable

Non-blocking mode on its own is useless — spinning in a loop polling sockets means burning the CPU. You need a call that sleeps and wakes up when at least one descriptor becomes ready.

141.2 Three calls, one server

The server does the same thing whatever the way of waiting: the listening socket is in the set; when it is ready, accept everyone you can and add them to the set; when a connection is ready, read everything up to EAGAIN and send it back:

static int serve(int fd)
{
    char buf[4096];
    for (;;) {
        ssize_t n = read(fd, buf, sizeof buf);
        if (n < 0) return errno == EAGAIN ? 1 : 0;      /* EAGAIN: данных больше нет — не ошибка */
        if (n == 0) return 0;
        if (write(fd, buf, n) < 0) return 0;
    }
}

This is a simplification: on a non-blocking socket write may write only part of the data or return EAGAIN when the send buffer is full. A real server keeps the unsent tail in the connection's buffer, subscribes to EPOLLOUT and finishes writing when the socket is ready for writing again.

select is the oldest. The descriptor set is an fd_set bit mask of 1024 bits (128 bytes), and that is the limit: a descriptor numbered 1024 or above does not fit into it. The call overwrites the set, so you copy it before every call, and afterwards you go through all numbers up to maxfd:

        fd_set rd = all;                                 /* select портит набор — копия на каждый вызов */
        select(maxfd + 1, &rd, NULL, NULL, NULL);
        for (int fd = 0; fd <= maxfd; fd++) {            /* перебор всех номеров до maxfd */

poll takes an array of struct pollfd of any length, with no 1024 limit, but every call passes the whole array into the kernel and gets the whole array back with revents flags; the work grows with the number of connections, not with the number of ready ones.

epoll — the kernel keeps the watched list itself; epoll_ctl adds and removes one at a time, and epoll_wait returns only the ready ones:

        int n = epoll_wait(ep, evs, 64, -1);             /* ядро отдаёт только готовые дескрипторы */

The load: a client opens connections and sends messages back and forth over each one:

At 500 connections all three cope; the difference in speed comes from scanning: select and poll walk the whole set for every message. At 2000 connections select breaks — the server has to close everything that did not fit into fd_set, and the client gets an echo on only half of them. poll copes, but four times slower than epoll: 2000 structures into the kernel and back for each of the 4000 exchanges.

141.3 epoll: level-triggered and edge-triggered

By default epoll_wait is level-triggered: while the socket has unread data, it returns this descriptor again and again. The EPOLLET flag makes it edge-triggered: one notification when data arrives, and if you do not read all of it, the next one comes only with new data. A client that waits for a reply and stays silent will not send any — the tail stays in the socket for good. Hence the rule for edge-triggered mode: the socket is non-blocking, and you read up to EAGAIN — as in serve above. The server in the experiment accepts connections with accept4(…, SOCK_NONBLOCK) and sets EPOLLIN | EPOLLET:

                    struct epoll_event ce = { .events = EPOLLIN | EPOLLET, .data.fd = c };   /* по фронту */
                    epoll_ctl(ep, EPOLL_CTL_ADD, c, &ce);

The data field can be anything: the descriptor number or a pointer to a connection structure with buffers and state; the kernel returns it as is. EPOLLRDHUP reports that the client has closed its side; EPOLLONESHOT gives one trigger, after which the descriptor must be re-armed (for a thread pool).

epoll exists only in Linux; in BSD and macOS its role is played by kqueue, in Windows by completion ports (IOCP). The libraries libevent, libuv (the core of Node.js) and libev hide this difference behind one event loop and add timers, DNS and HTTP.

141.4 Everything is a descriptor

In one poll, next to the sockets, it is convenient to wait for other things too, if they are also descriptors. Linux provides three of them:

• timerfd — a timer: read returns the number of periods that have elapsed;
• eventfd — a counter for signalling between threads or processes: write adds to it, read takes the value and resets it to zero;
• signalfd — signals arrive as signalfd_siginfo records rather than in a handler; the signal itself is blocked with the mask, otherwise it goes the old way.

    struct pollfd p[3] = { { tfd, POLLIN, 0 }, { efd, POLLIN, 0 }, { sfd, POLLIN, 0 } };

таймер: тик 1
eventfd: 1 от другого потока
таймер: тик 2
signalfd: сигнал 10
таймер: тик 3

This way the server's main loop receives timers, SIGTERM signals and messages from worker threads by the same path as data from sockets — with no races between the signal handler and the loop (section 136).

141.5 Reactor, C10K and io_uring

The loop "wait for readiness → call the handler" is called a reactor; this is how nginx (one reactor process per core), Redis (one thread) and Node.js are built. Ten thousand simultaneous connections is the problem on which servers with a thread or process per client broke down in the late nineties; epoll and kqueue appeared as the answer to it. Today the same scheme holds a million connections: the only cost is memory for buffers.

epoll reports readiness, and the program must read and write itself. io_uring changes the model: the program puts requests (read so many bytes into this buffer) into a ring in memory shared with the kernel, the kernel executes them and puts the results into another ring — with no system call per operation. This is completion-based I/O, like IOCP in Windows, and it also covers files, for which epoll is useless (a regular file is always "ready"). The old POSIX AIO in glibc is threads with pread inside, and it gives no gain.

141.6 Coroutines: sequential code on an event loop

Reactor handlers tear the logic of a connection into pieces between events. A coroutine hides this: a function "waits for data" and suspends, and the event loop resumes it when the data has arrived; the code reads as sequential. C has no coroutines in the language; they are provided by libraries: context switching via makecontext/swapcontext (libtask) or a short piece of assembly (libco), macros on a switch by line number (Duff's device trick — with no stack of its own), or ready-made libmill/libdill in the Go style. Each coroutine has its own stack on the heap (usually 8–64 KB) instead of megabytes for a thread. A switch in assembly does without the kernel — a few nanoseconds; swapcontext from glibc also saves the signal mask with a system call on every switch and comes out many times slower — in the experiment, more than a hundred nanoseconds. The downside: one coroutine that goes off into a long computation stops all of them.

141.7 Pitfalls

141.8 Summary

• O_NONBLOCK: instead of waiting — EAGAIN; read up to it.
• select — 1024 descriptors and an overwritten set; poll — no limit, but the whole array goes into the kernel; epoll — only the ready ones: 154 ms versus 698 for poll at 2000 connections.
• EPOLLET requires reading up to EAGAIN; data holds your own pointer.
• timerfd, eventfd, signalfd — a timer, an event and a signal in one poll.
• A reactor is one thread per core; io_uring is completion-based; coroutines are sequential code on top of an event loop.

Full sources

echo_epoll.c — an echo server on epoll, select or poll.

/* Раздел 141: эхо-сервер на epoll — один поток обслуживает любое число соединений.
   Ключ -s — select, -p — poll, по умолчанию epoll. Порт — первый аргумент. */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <unistd.h>
#include <poll.h>
#include <sys/select.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/in.h>

static int set_nonblocking(int fd)
{
    int fl = fcntl(fd, F_GETFL, 0);
    return fcntl(fd, F_SETFL, fl | O_NONBLOCK);
}

static int listen_on(int port)
{
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    int one = 1; setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);
    struct sockaddr_in a = { .sin_family = AF_INET, .sin_port = htons(port), .sin_addr.s_addr = htonl(INADDR_ANY) };
    if (bind(fd, (struct sockaddr *)&a, sizeof a) < 0 || listen(fd, 128) < 0) { perror("bind/listen"); exit(1); }
    set_nonblocking(fd);
    return fd;
}

/* обслужить готовый дескриптор: читать всё, что есть, и вернуть; 0 — соединение закрыто */
static int serve(int fd)
{
    char buf[4096];
    for (;;) {
        ssize_t n = read(fd, buf, sizeof buf);
        if (n < 0) return errno == EAGAIN ? 1 : 0;      /* EAGAIN: данных больше нет — не ошибка */
        if (n == 0) return 0;
        if (write(fd, buf, n) < 0) return 0;
    }
}

static void run_epoll(int lfd)
{
    int ep = epoll_create1(0);
    struct epoll_event ev = { .events = EPOLLIN, .data.fd = lfd }, evs[64];
    epoll_ctl(ep, EPOLL_CTL_ADD, lfd, &ev);
    for (;;) {
        int n = epoll_wait(ep, evs, 64, -1);             /* ядро отдаёт только готовые дескрипторы */
        for (int i = 0; i < n; i++) {
            int fd = evs[i].data.fd;
            if (fd == lfd) {
                int c;
                while ((c = accept4(lfd, NULL, NULL, SOCK_NONBLOCK)) >= 0) {
                    struct epoll_event ce = { .events = EPOLLIN | EPOLLET, .data.fd = c };   /* по фронту */
                    epoll_ctl(ep, EPOLL_CTL_ADD, c, &ce);
                }
            } else if (!serve(fd)) {
                epoll_ctl(ep, EPOLL_CTL_DEL, fd, NULL); close(fd);
            }
        }
    }
}

static void run_poll(int lfd)
{
    static struct pollfd fds[8192]; int n = 1;      /* poll: массив любого размера, но весь идёт в ядро */
    fds[0] = (struct pollfd){ lfd, POLLIN, 0 };
    for (;;) {
        poll(fds, n, -1);                                /* каждый вызов — весь массив в ядро и обратно */
        for (int i = 0; i < n; i++) {
            if (!(fds[i].revents & (POLLIN | POLLHUP))) continue;
            if (fds[i].fd == lfd) {
                int c;
                while ((c = accept4(lfd, NULL, NULL, SOCK_NONBLOCK)) >= 0 && n < 8192)
                    fds[n++] = (struct pollfd){ c, POLLIN, 0 };
            } else if (!serve(fds[i].fd)) {
                close(fds[i].fd); fds[i] = fds[--n]; i--;
            }
        }
    }
}

static void run_select(int lfd)
{
    fd_set all; FD_ZERO(&all); FD_SET(lfd, &all); int maxfd = lfd;
    for (;;) {
        fd_set rd = all;                                 /* select портит набор — копия на каждый вызов */
        select(maxfd + 1, &rd, NULL, NULL, NULL);
        for (int fd = 0; fd <= maxfd; fd++) {            /* перебор всех номеров до maxfd */
            if (!FD_ISSET(fd, &rd)) continue;
            if (fd == lfd) {
                int c;
                while ((c = accept4(lfd, NULL, NULL, SOCK_NONBLOCK)) >= 0) {
                    if (c >= FD_SETSIZE) { close(c); continue; }     /* select не умеет больше 1024 */
                    FD_SET(c, &all); if (c > maxfd) maxfd = c;
                }
            } else if (!serve(fd)) { close(fd); FD_CLR(fd, &all); }
        }
    }
}

int main(int argc, char **argv)
{
    signal(SIGPIPE, SIG_IGN);
    int lfd = listen_on(argc > 1 ? atoi(argv[1]) : 7000);
    const char *mode = argc > 2 ? argv[2] : "-e";
    if (!strcmp(mode, "-s")) run_select(lfd);
    else if (!strcmp(mode, "-p")) run_poll(lfd);
    else run_epoll(lfd);
    return 0;
}

Next: section 142 — web services in C: HTTP, CGI, FastCGI, sendfile.