121а. OpenMV: a machine-vision camera on STM32H7 — DCMI, MicroPython and object search

Sections 116–121 worked with numbers from sensors: vibration, current, voltage. A camera gives a different volume: one 320×240 colour frame is 153600 bytes, more than the entire on-chip SRAM of many STM32 parts. Section 22б showed how DCMI and DMA put a frame into memory. Here is a ready-made open system that goes further: it receives the frame, searches it for colour blobs, AprilTag markers and QR codes, and runs a neural network, all controlled by a MicroPython script. This is the OpenMV project.

This section takes apart its sources: which boards and chips, how the frame travels from the sensor to the buffer, how many frames fit in memory, how colour is converted to LAB and how find_blobs finds blobs. The blob search function was built on a PC from the project sources and run on a synthetic frame with known answers.

References: the OpenMV sources (github.com/openmv/openmv, commit 62369938a94b). License: MIT — the line SPDX-License-Identifier: MIT in the imlib headers (blob.c, framebuffer.c; the lab_tab.c table is generated by the MIT script tools/gen_rgb2lab.py), stm_dma.c and py_image.c. But not everywhere. The camera layer (common/omv_csi.c, ports/stm32/omv_csi.c, modules/py_csi_ng.c), the neural network module modules/py_ml.c and the boot/ bootloader are under a different license: you may copy them, but "solely for personal benefit and not for any commercial purpose"; commercial use needs a separate agreement with OpenMV. The barcode decoder lib/imlib/zbar.c is LGPL 2.1, the ST library in cubeai/AI is under the ST license SLA0044. The DCMI registers and splitting the frame into parts for DMA are in section 22б, the H7 cache and memory domains in section 11б, quantization and TensorFlow Lite Micro in section 119.

What it is called in the documentation

Original	Meaning
machine vision	a program finding objects in a frame
frame buffer	memory holding one frame
line buffer	buffer for one or two lines of the frame
MDMA (master DMA)	the H7 D1-domain DMA: copies blocks between memories
UMA pool	the OpenMV memory pool that frames and models are taken from
blob	a connected area of pixels that passed the threshold
threshold	six bounds on L, A, B
CIELAB (L*a*b*)	colour space: lightness L and two colour axes A and B
flood fill	walking a connected area
roundness, elongation	how round or stretched a blob is
AprilTag	a square coded marker for finding the camera pose

121а.1 Boards and chips

Each board in OpenMV is a directory boards/<name> with two files: board_config.mk (chip, clock, which modules to build) and board_config.h (pins, memory, sensors). From them:

From here on we talk about OPENMV4 — the STM32H743. Its Flash is 2 MB: 128 KB is taken by the bootloader, 128 KB by the file system, 1664 KB is left for the firmware, and the last 128 KB hold the ROMFS image. The stack is 64 KB in ITCM. Frames, neural network models and temporary arrays live in the UMA pool: 480 KB of the 512 KB of SRAM at 0x2400 0000.

There are 20 driver files in drivers/sensors; OPENMV4 includes ten: OV2640, OV5640, OV7725, MT9V0xx, OV9650, MT9M114, the Lepton thermal camera, PAG7920, PAJ6100 and FrogEye2020. The XCLK clock for the sensor comes from timer TIM1 on pin PA8: 12 MHz on H7, 6 MHz on OPENMV2, 9 MHz on OPENMV3 (OMV_CSI_CLK_FREQUENCY). The hardware JPEG codec is present in the H7 and N6 builds (OMV_JPEG_CODEC_ENABLE (1)), F4 and F7 do not have it.

The user program is a MicroPython script. A frame is one line, blob search is one more (example from scripts/examples/02-Image-Processing/02-Color-Tracking/single_color_rgb565_blob_tracking.py, MIT):

thresholds = [
    (30, 100, 15, 127, 15, 127),  # generic_red_thresholds
    (30, 100, -64, -8, -32, 32),  # generic_green_thresholds
    (0, 30, 0, 64, -128, 0),  # generic_blue_thresholds
]

A threshold is six numbers: the bounds of L, A and B. Under Python runs the imlib library in C: the script calls img.find_blobs(...), and the work is done by the function imlib_find_blobs from lib/imlib/blob.c.

121а.2 The frame path: DCMI, a line and MDMA

Section 22б drove DMA over the whole frame, in parts of up to 65535 words. OpenMV on H7 does it differently: DMA receives one line in circular mode, and a second controller — MDMA — lays the lines out into the frame. The camera driver for STM32 (ports/stm32/omv_csi.c) enables circular mode on the DMA stream, starts DCMI in continuous capture and gives DMA a length of exactly one line, in words, with the address of the line buffer. DMA writes the line, reaches the end of the buffer and starts again from the beginning — in that time MDMA must take the finished line.

The DMA stream is DMA2_Stream1, the DCMI request. The line buffer is defined in boards/OPENMV4/board_config.h:

#define OMV_LINE_BUF_SIZE                     (3 * 1024) // Image line buffer round(640 * 2BPP * 2 buffers).

A check before starting requires the line to be no longer than half the buffer: 1536 bytes, that is 768 RGB565 pixels. A VGA line is 1280 bytes, it fits. On the transfer-complete interrupt the handler starts MDMA: channels 0 and 1 are reserved for MDMA (OMV_MDMA_CHANNEL_DCMI_0, _1), and on AXI MDMA has the highest priority, 15, for reads and writes.

What this gives:

• Horizontal cropping without DCMI. The vertical window is set by DCMI (HAL_DCMI_ConfigCrop, section 22б), and the horizontal one by the MDMA source address: the copy starts from the needed pixel of the line. DCMI only cuts lines, the extra pixels on the left and right are dropped by the copy.
• Grayscale from a colour sensor. Many sensors output grayscale as YUV 4:2:2 — two bytes per pixel. On H7, MDMA takes only the luma bytes from the line; 2 bytes still travel over the DCMI bus. On N6 DCMI itself does the same by byte decimation (the BSM field, section 22б).
• Frame rotation. In transpose mode a line goes into the frame as a column — by copying with a stride.

The data rate on the DCMI bus is computed as in section 22б: 8 bits per PIXCLK cycle, 2 bytes per pixel. The PIXCLK frequency is set by the sensor configuration; for the board only XCLK of 12 MHz is known; here is what an example at 24 and 48 MHz gives:

This is the upper bound without line and frame blanking and without processing: the real frame rate is lower.

121а.3 Frame buffers and JPEG for the IDE

Frame size:

A buffer in the pool is larger than the frame by the vbuffer_t header — offset and flags, with the data starting on a cache line boundary: 32 bytes, and the alignment is also 32 (lib/imlib/framebuffer.h). Buffers move between two queues: free and filled. The number of buffers is chosen by the function omv_csi_set_framebuffers in common/omv_csi.c: if the script has not set the number, it asks framebuffer_resize for three buffers, on failure for two, then one; if even one does not work — an error.

It computes the frame size as window width × height × 2 — two bytes per pixel for any format. framebuffer_resize adds two queues of 28 bytes for three buffers (24 for two, 20 for one). How many buffers fit in the pool if it is empty:

Three QVGA buffers are 460952 bytes out of 491520 on H7. VGA at 16 bits needs 614472 bytes for one buffer — impossible on H7; on H7 Plus three fit in SDRAM (section 41б).

Three buffers mean triple buffering: DMA writes into one, the program reads another, the third holds the last finished frame. When only one free buffer is left, framebuffer_release with three buffers swaps the old finished frame for the new one, and with two it does not hand out the new frame: the next one is written into the same buffer.

Preview in the IDE. The frame goes to OpenMV IDE over USB compressed as JPEG (lib/imlib/framebuffer.c). The quality adapts: the initial value is the middle between 50 and 90, that is 70. If the compressed frame did not fit into the transmit buffer, the quality is halved, the frame is skipped, and for the next 60 frames the quality does not grow: 70 → 35 → 17 → 8 → 4 on overflows in a row. Without overflows it grows by 1 per frame up to a ceiling: 90 for frames up to 153600 bytes inclusive (QVGA RGB565 is exactly that much) and 50 for larger ones. The ceiling only stops the growth: for VGA RGB565 the quality will not rise above 50, but it does not lower the starting 70 by itself — only an overflow does.

121а.4 Colour: RGB565 to LAB by a table

A colour threshold in RGB copes badly with lighting: a shadow changes all three channels. In LAB the lightness is a separate axis L (0…100), and the colour is on the axes A (green — red) and B (blue — yellow). That is why OpenMV thresholds are in LAB.

Computing LAB for every pixel is expensive: a power of 2.4 and a cube root. imlib takes ready values from the lab_table table in lib/imlib/lab_tab.c, three bytes L, A, B per entry. There are not 65536 entries but 32768 — the index is pixel >> 1, the lowest bit of blue is dropped:

#define COLOR_RGB565_TO_L(pixel)                lab_table[((pixel >> 1) * 3) + 0]

The table takes 98304 bytes instead of 196608. The generator is in the project — tools/gen_rgb2lab.py: it converts each of the 65536 pixels with the sRGB and CIELAB formulas (D65 white point), rounds and averages neighbouring pairs. Running the generator on a PC:

• the table matched lab_tab.c in all 98304 bytes, the linearization table xyz_table in all 256 numbers;
• the difference between the table and the exact formula for an individual pixel is up to 1 in L, up to 3 in A and in B; 1761 pixels out of 65536 match exactly;
• values in the table: L from 0 to 100, A from −86 to 97, B from −106 to 94.

The colours of the test frame from this table:

The red passes the example's "generic red" threshold (L 30…100, A and B from 15), the blue passes "generic blue" (L up to 30, A 0…64, B up to 0). But the green does not: its B = 54, and the "generic green" threshold limits B to 32. This green with a yellow tint will not be found in the run below.

121а.5 find_blobs: flood fill by lines

imlib_find_blobs walks the frame line by line with a step of x_stride in X (by default 2) and y_stride in Y (1). A pixel that passed the threshold and is not yet taken becomes the start of a blob, and the blob is filled line by line: the segment is extended left and right while the pixels pass the threshold, they are marked in a frame-sized bitmap, and a continuation is searched for in the line above and below. Instead of recursion — a LIFO stack of xylr_t records (six 16-bit numbers, 12 bytes); for the stack lifo_alloc_all takes everything that is free in the UMA pool.

During the fill each segment adds its pixel count and the sums of X, Y, X², X·Y and Y² to the blob — from these moments come the centre, the rotation angle and the roundness without a second pass. Then blobs with fewer than pixels_threshold pixels or less than area_threshold bounding-box area are dropped (by default both are 10), and with merge=True overlapping boxes are merged; margin expands the box before the check.

The run. The files blob.c, collections.c, fmath.c, lab_tab.c, sincos_tab.c and three functions from imlib.c were built with gcc on a PC with stubs for MicroPython and the UMA pool. A QVGA RGB565 frame: a gray background with ±12 noise in each channel, rectangles 60×40 (red), 50×80 (green), 70×30 (blue), a red 60×10 strip 4 lines below the red rectangle, a red circle of radius 20 with 1257 pixels and 40 single red pixels.

Areas and centres matched the constructed ones exactly: 60 × 40 = 2400, 70 × 30 = 2100, the centre of the 60×40 rectangle with its corner at 20, 30 is 49.5; 49.5. The merged blob has a weighted centre: (2400 · 49.5 + 600 · 78.5) / 3000 = 55.3. The roundness of the circle is 1.000, of the 60×40 rectangle 0.445, of the 60×10 strip 0.028. No memory leaks: all pool blocks are freed.

With the default threshold (10 pixels) all single pixels are dropped — this is how noise is filtered. With step 2 the function does not see 21 pixels out of 40 at all: the start of a blob is searched for only at every second pixel of a line.

Other searches in imlib. find_apriltags is a wrapper around the AprilTag 3.4.5 library (the lib/apriltag submodule); on H7 the TAG36H11 family is enabled. find_qrcodes is the quirc decoder (MIT), embedded in lib/imlib/qrcode.c. Haar cascades (lib/imlib/haar.c, face and eye cascades in lib/haar) compute the integral image not over the whole frame but in a sliding window of lines (imlib_integral_mw_*) — the memory needed is for the window, not for the frame.

121а.6 Neural networks: TensorFlow Lite Micro and ST Edge AI

The ml module (modules/py_ml.c) loads a .tflite model into the UMA pool — into fast memory and for good, until the model is unloaded — and runs it through the build's engine: on OPENMV4, OPENMV4P, Nicla Vision and Portenta it is TensorFlow Lite Micro (MICROPY_PY_ML_TFLM = 1), on N6 it is ST Edge AI for the neural accelerator (MICROPY_PY_ML_STAI = 1). The input scale and zero point are taken from the model itself, and the input is quantized by the formula of section 119: the value is divided by the scale (in the code it is multiplied by the precomputed reciprocal), the zero point is added, and the result is written as int8. The output is converted back: (q − zero_point) · scale.

Ready models in lib/models and their sizes:

person_detect is 300568 bytes; together with three QVGA buffers (460952) this is more than the H7 pool of 491520 bytes, so the model and the frames have to share memory: fewer buffers, a smaller frame. YOLOv8n at 3.2 MB does not fit into the H7 pool at all — that is a job for H7 Plus with SDRAM or N6.

The cubeai/ directory is a build variant with STM32Cube.AI models for H7 and H7 Plus: the wrappers nn_st.c, py_st_nn.c are under MIT, and the library in cubeai/AI is under the ST license SLA0044.

Building the firmware. For this section the submodules were not fetched: lib/micropython, lib/tflm/libtflm, lib/apriltag, lib/tinyusb, modules/ulab are empty (git submodule status shows them with "−"). Without MicroPython the firmware does not build — so the firmware size is not given here. Only a test file of the frame buffer structures was built for ARM: sizeof(vbuffer_t) = 32 and the queue sizes 28/24/20 bytes are confirmed by _Static_assert in arm-none-eabi-gcc for Cortex-M7.

121а.7 Pitfalls

• The threshold drifts with auto gain. Auto gain and auto white balance shift L, A and B from frame to frame. The OpenMV example turns both off before a colour search: auto_gain(False) and auto_whitebal(False) with the note "must be turned off for color tracking".
• A "generic" threshold does not catch your colour. Green 30, 170, 40 has B = 54, the example threshold is B up to 32: zero blobs. Tune the threshold for your object and lighting — the IDE has a threshold editor.
• Merging does not work when the gap equals the margin. The boxes are compared strictly: with a 4-line gap margin=4 leaves two blobs, you need 5.
• Step 2 loses small things. With x_stride=2, 19 of 40 single pixels were found. For thin lines and small objects use x_stride=1.
• Thresholds near the table's limits. The LAB table is off for individual pixels by up to 3 units in A and B: if the threshold bound is closer than 3 units to the object's colour, the blob will "flicker".
• A grayscale frame takes as much space as a colour one. omv_csi_set_framebuffers reserves 2 bytes per pixel for any format: a grayscale VGA frame on H7 does not fit in the pool, just like a colour one.
• The model and the frames share one pool. person_detect at 300568 bytes and three QVGA buffers do not fit together in the 480 KB of H7.
• The picture in the IDE is worse than the frame. The preview is JPEG; after overflows the quality drops to single digits; processing works on the uncompressed frame. Do not judge a threshold by the IDE picture under heavy compression.
• The H7 cache. Frame buffers are in cacheable SRAM; framebuffer_acquire with the FB_FLAG_INVALIDATE flag invalidates the cache lines with SCB_InvalidateDCache_by_Addr — without it the CPU reads a stale frame (section 11б).
• The licenses differ. imlib is MIT, but the camera driver (omv_csi.c) and the ml module are for non-commercial use only, zbar.c (barcodes) is LGPL 2.1, the Cube.AI library is SLA0044. For a device for sale you can freely take imlib from OpenMV, and write the frame capture yourself (section 22б) or make an agreement with OpenMV.

121а.8 Summary

• OpenMV is an open machine vision system: OPENMV4 on the STM32H743 at 480 MHz, H7 Plus with 32 MB SDRAM, Arduino boards on H747, N6 on the STM32N657; imlib is MIT, the camera driver and the ml module are under a non-commercial license, zbar is LGPL, Cube.AI is SLA0044.
• The frame goes by lines: DMA2 stream 1 in circular mode into a 3 KB line buffer, MDMA on channels 0 and 1 into the frame buffer; a line is no longer than 1536 bytes.
• QVGA RGB565 is 153600 bytes; the H7 with a 480 KB pool holds three buffers (460952 bytes), VGA RGB565 not a single one, H7 Plus three.
• JPEG for the IDE: quality 70 at the start, growth up to 90 for frames up to 153600 bytes and up to 50 for larger ones, on overflow — halved and 60 frames without growth.
• LAB by a table of 98304 bytes for 32768 entries; the project's generator reproduces it byte for byte, the error against the exact formula is up to 3 units.
• find_blobs on a synthetic frame: areas 2400, 1257, 2100 and centres matched exactly; margin=5 merges blobs across a 4-line gap; x_stride=2 finds 19 single pixels out of 40.
• int8 models: person_detect 300568 bytes through TensorFlow Lite Micro; the firmware cannot be built without the submodules — its size is not given.

Next: section 121б "Audio on a microcontroller: from a PDM microphone to MFCC and command recognition" — the path from a PDM bit to a command: CIC and FIR, MFCC in integers, an int8 network and a decision on the stream.