152. FreeRTOS on two cores: task pinning, spinlocks, tick, stack, esp_timer, watchdog

On the ESP32 you do not choose FreeRTOS from part IV — it is already inside ESP-IDF and is running before app_main. But it is not the same FreeRTOS: the ESP32 has two cores, and the scheduler keeps a current task on each of them. Everything new follows from that: a task can be pinned to a core, taskENTER_CRITICAL no longer just disables interrupts but takes a lock — the other core, reaching for the same lock, waits; the stack is given in bytes, not in words; and a watchdog looks after each core. This section walks through these differences with one program in QEMU (_verify/esp152/), and where the emulator lies, it says so plainly.

Terms used in the documentation

Term	Meaning
SMP (symmetric multiprocessing)	two identical cores under one scheduler
PRO_CPU, APP_CPU	core 0 and core 1
core affinity, pinned task	binding a task to a core
tskNO_AFFINITY	a task without binding: the scheduler picks the core itself
spinlock, portMUX_TYPE	a lock on which the other core spins while waiting
critical section	interrupts are off and the lock is taken
tick, tick rate	the tick and its frequency (CONFIG_FREERTOS_HZ)
stack high water mark	the minimum remaining stack over the whole run
idle task	the idle task, one per core
task watchdog (TWDT)	waits for check-ins from the idle tasks
esp_timer	high-resolution timers on one hardware timer
IPC task	a service task for calling a function on the other core

152.1 Tick, tasks, cores

The first thing the program prints is the scheduler parameters:

тик 100 Гц, pdMS_TO_TICKS(5) = 0, pdMS_TO_TICKS(10) = 1, portTICK_PERIOD_MS = 10
app_main: ядро 0, приоритет 1, стек задан 3584 байт

The default tick is 100 Hz, that is 10 ms: vTaskDelay(pdMS_TO_TICKS(5)) turns into vTaskDelay(0) and does not wait at all, it only yields the processor. Whoever needs milliseconds sets CONFIG_FREERTOS_HZ=1000 or uses esp_timer. app_main is an ordinary task with priority 1 on core 0; its stack is 3584 bytes — on the ESP32 the stack size in xTaskCreate is always in bytes, whereas in vanilla FreeRTOS it is in words.

A task is created with or without pinning:

    xTaskCreatePinnedToCore(whoami, "w0", 2048, "к ядру 0", 4, NULL, 0);
    vTaskDelay(pdMS_TO_TICKS(50));
    xTaskCreatePinnedToCore(whoami, "w1", 2048, "к ядру 1", 4, NULL, 1);
    vTaskDelay(pdMS_TO_TICKS(50));
    xTaskCreate(whoami, "wany", 2048, "любое", 4, NULL);   /* tskNO_AFFINITY */

задача к ядру 0 ядро 0 приоритет 4
задача к ядру 1 ядро 1 приоритет 4
задача любое ядро 0 приоритет 4

An unpinned task can end up on either core and even move between them from one call to the next — so anything tied to a core (interrupt handlers, which are allocated to the core they were installed on) is pinned explicitly. ESP-IDF keeps Wi-Fi and Bluetooth on core 0, and applications usually get core 1.

152.2 A race on two cores and three ways to cure it

In section 139 two Linux threads lost increments of a shared counter; on the ESP32 two cores lose them. The tricks of single-core firmware do not help: disabling interrupts (portDISABLE_INTERRUPTS) and suspending the scheduler (vTaskSuspendAll) act only on their own core, and a high priority does not stop the other core from running a lower-priority task. A FreeRTOS mutex does protect across cores (152.6), but it is heavy for a single ++. For short sections there are three ways:

        if (mode == 0) {
            plain++;
        } else if (mode == 1) {
            taskENTER_CRITICAL(&lock);              /* спин-блокировка: держит оба ядра порознь */
            guarded++;
            taskEXIT_CRITICAL(&lock);
        } else {
            __atomic_fetch_add(&atomic_cnt, 1, __ATOMIC_RELAXED);
        }

On the ESP32 taskENTER_CRITICAL always takes a portMUX_TYPE spinlock: the core disables its own interrupts and spins until the other core releases the lock. The compiler implements an atomic addition with the Xtensa instruction s32c1i — "compare and store", the same idea as the CAS from section 140. It is visible in the machine code of the task:

      2 s32i.n a8,
      2 memw 
      2 addi.n a8,
      1 wsr.scompare1 a8
      1 s32c1i a11,

plain++ is three instructions, l32i, addi, s32i, between which the other core can manage to do the same; the atomic variant is wsr.scompare1 (the expected value into a special register) and s32c1i in a loop until it succeeds. The result in the emulator:

без защиты 4000000 из 4000000, потеряно 0, 49 мс
спин-блокировка 4000000 из 4000000, потеряно 0, 1119 мс
atomic           4000000 из 4000000, потеряно 0, 59 мс

The most important thing here is what is not there: losses. QEMU executes the two cores in turn in large chunks, and the three instructions of plain++ are never interleaved with the other core's — a race that loses increments on silicon is invisible in the emulator. This is a general property of emulators: they show the logic correctly, but not simultaneity — and not the timing of the silicon. In the emulator the critical section came out 23 times more expensive than the unprotected addition, and the atomic instruction almost free, but QEMU translates Xtensa instructions into host instructions, and those have their own cost. The ratio of 23 is a rough guide; the real figure is measured on the board with the cycle counter esp_cpu_get_cycle_count(). The rule for code: short counters — __atomic_* or <stdatomic.h>, structures — a spinlock, and never hold it longer than a few microseconds: on its own core interrupts are disabled all that time, and the other core, if it reaches for the same lock, spins idle.

152.3 The stack and the task list

стек app_main: свободно 1188 байт из 3584

uxTaskGetStackHighWaterMark returns the smallest remainder over the whole run, also in bytes. vTaskList prints all tasks with the core in the last column:

main            X   1   1188    4   0
IDLE1           R   0   1036    6   1
IDLE0           R   0   1024    5   0
ipc1            S   24  496 2   1
esp_timer       S   22  3576    3   0
ipc0            S   24  504 1   0

Besides our main there is one idle task per core (IDLE0, IDLE1, priority 0, stack 1536 bytes), two service tasks ipc0/ipc1 with priority 24 for calling functions on the neighbouring core, and the esp_timer task with priority 22, in which all callbacks of the software timers run. States: X — running, R — ready, B — waiting for an event or a timeout (vTaskDelay, queue, semaphore), S — suspended; a wait for a notification without a timeout (ulTaskNotifyTake(…, portMAX_DELAY)) also lands here — this is how ipc and esp_timer wait. D — deleted, but the memory is not freed yet.

152.4 esp_timer instead of software timers

FreeRTOS software timers work in ticks; esp_timer gives microseconds on one hardware timer shared by all:

    esp_timer_create(&ta, &th);
    esp_timer_start_periodic(th, 10000);
    int64_t t0 = esp_timer_get_time();
    vTaskDelay(pdMS_TO_TICKS(1000));
    int64_t dt = esp_timer_get_time() - t0;
    esp_timer_stop(th);

esp_timer: за vTaskDelay(100 тиков) прошло 999592 мкс, срабатываний по 10 мс: 99

A hundred ticks is one second by the esp_timer_get_time clock, and in that second the periodic timer with a 10 ms step fired 99 or 100 times (from run to run): the first firing comes 10 ms after the start, and the hundredth falls exactly on the one-second boundary and sometimes makes it, sometimes not. The callback runs in the esp_timer task, not in an interrupt, so almost anything is allowed in it, but you must not delay it — the whole timer queue will be delayed.

152.5 The task watchdog

A task that never yields the processor hangs everything of lower priority on a single-core STM32. On the ESP32 the other core keeps running, but the task watchdog (TWDT) will notice:

static void spinner(void *arg)                      /* никогда не уступает — морит задачу простоя ядра 1 */
{
    (void)arg;
    for (;;) { }
}

E (8882) task_wdt: Task watchdog got triggered. The following tasks/users did not reset the watchdog in time:
E (8882) task_wdt:  - IDLE1 (CPU 1)
E (8882) task_wdt: Tasks currently running:
E (8882) task_wdt: CPU 0: IDLE0
E (8882) task_wdt: CPU 1: spin

By default the watchdog watches the idle tasks of both cores (CONFIG_ESP_TASK_WDT_CHECK_IDLE_TASK_CPU0/1=y) and fires after 5 s (CONFIG_ESP_TASK_WDT_TIMEOUT_S=5): if a core's idle task has not had the processor for 5 seconds, someone on that core is spinning without vTaskDelay. The watchdog printed the culprit (CPU 1: spin) and a backtrace, but did not reboot: CONFIG_ESP_TASK_WDT_PANIC is off. In a production device it is turned on, and tasks that are supposed to run for a long time subscribe to the watchdog themselves (esp_task_wdt_add) and check in with esp_task_wdt_reset. The second watchdog, the interrupt watchdog, fires after 300 ms if interrupts are disabled for too long — for example, because of a drawn-out critical section from 152.2.

152.6 What else differs from part IV

Queues, semaphores, mutexes, event groups, notifications are the same as in part IV, and they work across cores without caveats: inside they are already protected by spinlocks.

152.7 Pitfalls

152.8 Summary

• Tick 100 Hz, pdMS_TO_TICKS(5) = 0; stack in bytes; app_main is a task with priority 1 on core 0.
• xTaskCreatePinnedToCore(..., 0/1) or tskNO_AFFINITY; the radio lives on core 0.
• Shared data between cores: a spinlock (23 times more expensive) or the atomic instruction s32c1i; disabling interrupts and vTaskSuspendAll act only on their own core, a FreeRTOS mutex protects across cores as well.
• The emulator does not show the race: logic yes, simultaneity no.
• vTaskList: IDLE0/1, ipc0/1 (24), esp_timer (22); esp_timer is microseconds, 99–100 firings per second.
• Task watchdog: 5 s on the idle tasks of both cores; interrupt watchdog: 300 ms.

Full sources

main.c — FreeRTOS on the two ESP32 cores: task pinning, a race, esp_timer, the watchdog.

/* Раздел 152: FreeRTOS на двух ядрах ESP32 — закрепление задач, гонка и её лечение,
 * тик и pdMS_TO_TICKS, стек в байтах, esp_timer, сторож задач. */
#include <stdio.h>
#include <string.h>
#include "esp_log.h"
#include "esp_task_wdt.h"
#include "esp_timer.h"
#include "freertos/FreeRTOS.h"
#include "freertos/semphr.h"
#include "freertos/task.h"

#define N 2000000

static volatile int plain;                          /* без защиты */
static int guarded;                                 /* под спин-блокировкой */
static int atomic_cnt;                              /* атомарная команда */
static portMUX_TYPE lock = portMUX_INITIALIZER_UNLOCKED;
static SemaphoreHandle_t done_sem;
static volatile int timer_hits;

static void whoami(void *arg)                       /* печатает, на каком ядре оказалась */
{
    printf("задача %-8s ядро %d приоритет %u\n", (const char *)arg, xPortGetCoreID(), (unsigned)uxTaskPriorityGet(NULL));
    vTaskDelete(NULL);
}

static void racer(void *arg)
{
    int mode = (int)arg;
    for (int i = 0; i < N; i++) {
        if (mode == 0) {
            plain++;
        } else if (mode == 1) {
            taskENTER_CRITICAL(&lock);              /* спин-блокировка: держит оба ядра порознь */
            guarded++;
            taskEXIT_CRITICAL(&lock);
        } else {
            __atomic_fetch_add(&atomic_cnt, 1, __ATOMIC_RELAXED);
        }
    }
    xSemaphoreGive(done_sem);
    vTaskDelete(NULL);
}

static void run_race(int mode, const char *name)
{
    int64_t t0 = esp_timer_get_time();
    xTaskCreatePinnedToCore(racer, "r0", 2048, (void *)mode, 5, NULL, 0);
    xTaskCreatePinnedToCore(racer, "r1", 2048, (void *)mode, 5, NULL, 1);
    xSemaphoreTake(done_sem, portMAX_DELAY);
    xSemaphoreTake(done_sem, portMAX_DELAY);
    int got = mode == 0 ? plain : mode == 1 ? guarded : atomic_cnt;
    printf("%-16s %d из %d, потеряно %d, %lld мс\n", name, got, 2 * N, 2 * N - got, (long long)((esp_timer_get_time() - t0) / 1000));
}

static void on_timer(void *arg) { (void)arg; timer_hits++; }

static void spinner(void *arg)                      /* никогда не уступает — морит задачу простоя ядра 1 */
{
    (void)arg;
    for (;;) { }
}

void app_main(void)
{
    done_sem = xSemaphoreCreateCounting(2, 0);
    printf("тик %d Гц, pdMS_TO_TICKS(5) = %u, pdMS_TO_TICKS(10) = %u, portTICK_PERIOD_MS = %u\n",
           CONFIG_FREERTOS_HZ, (unsigned)pdMS_TO_TICKS(5), (unsigned)pdMS_TO_TICKS(10), (unsigned)portTICK_PERIOD_MS);
    printf("app_main: ядро %d, приоритет %u, стек задан %d байт\n", xPortGetCoreID(), (unsigned)uxTaskPriorityGet(NULL), CONFIG_ESP_MAIN_TASK_STACK_SIZE);

    xTaskCreatePinnedToCore(whoami, "w0", 2048, "к ядру 0", 4, NULL, 0);
    vTaskDelay(pdMS_TO_TICKS(50));
    xTaskCreatePinnedToCore(whoami, "w1", 2048, "к ядру 1", 4, NULL, 1);
    vTaskDelay(pdMS_TO_TICKS(50));
    xTaskCreate(whoami, "wany", 2048, "любое", 4, NULL);   /* tskNO_AFFINITY */
    vTaskDelay(pdMS_TO_TICKS(50));

    run_race(0, "без защиты");
    run_race(1, "спин-блокировка");
    run_race(2, "atomic");

    /* стек задачи считается в байтах, не в словах */
    printf("стек app_main: свободно %u байт из %d\n", (unsigned)uxTaskGetStackHighWaterMark(NULL), CONFIG_ESP_MAIN_TASK_STACK_SIZE);

    /* esp_timer: периодический вызов раз в 10 мс, считаем за секунду по тикам FreeRTOS */
    esp_timer_handle_t th;
    const esp_timer_create_args_t ta = { .callback = on_timer, .name = "t10ms" };
    esp_timer_create(&ta, &th);
    esp_timer_start_periodic(th, 10000);
    int64_t t0 = esp_timer_get_time();
    vTaskDelay(pdMS_TO_TICKS(1000));
    int64_t dt = esp_timer_get_time() - t0;
    esp_timer_stop(th);
    printf("esp_timer: за vTaskDelay(100 тиков) прошло %lld мкс, срабатываний по 10 мс: %d\n", (long long)dt, timer_hits);

    char buf[1024];
    vTaskList(buf);
    printf("-- задачи (имя, состояние, приоритет, остаток стека, номер, ядро)\n%s", buf);

    /* сторож задач: задача, которая не уступает процессор, лишает ядро 1 задачи простоя */
    printf("запускаю бесконечный цикл на ядре 1, сторож — %d с\n", CONFIG_ESP_TASK_WDT_TIMEOUT_S);
    xTaskCreatePinnedToCore(spinner, "spin", 2048, NULL, 1, NULL, 1);
    vTaskDelay(pdMS_TO_TICKS(7000));
    printf("done\n");
}

Next: section 153 — GPIO and PWM: the pin matrix, LEDC, MCPWM.