snake_differ

Architecture overview — Puara module for the Seeed XIAO ESP32-S3 Sense

Purpose

A motion-detection firmware that watches a camera feed, computes frame-by-frame pixel differences, and broadcasts the results as OSC messages over WiFi — so a DAW, synthesizer, or generative environment can react to the presence, position, and shape of moving objects in the frame. It was built for tracking dark shapes (kinetic coil sculptures) on a light background, but works for any dark-on-light motion source.

Hardware

No wiring is required for the camera or mic; motor outputs need an external driver for anything beyond a small signal.

System overview

The camera and microphone feed the firmware, which either streams an OSC bundle out to receiving software, or serves a small web interface (with a live MJPEG preview) in a browser for calibrating the camera before a show. Both paths share the same WiFi connection and the same on-device settings store.

Firmware task structure

Internally, three concurrent pieces of code — two FreeRTOS tasks plus the Arduino main loop — share state through mutex-protected structs:

1. Camera task (~30 fps) — for every frame, a single pixel-by-pixel pass computes two independent sets of metrics at once:
   • Diff metrics: compares the current frame to the previous one; pixels changed by more than diffThreshold count as "in motion." From those pixels it derives a bounding box, centroid, frame-to-frame velocity, spread (standard deviation around the centroid), fill ratio (motion density inside the bounding box), and overall magnitude (fraction of the frame moving).
   • Pixel metrics: independent of motion, any pixel darker than pixThreshold counts as "dark coil material." This gives a global coverage fraction and centroid of dark mass, plus three values per zone across a 6×4 grid of 24 zones: dark-pixel coverage, edge/gradient density (a proxy for how tightly the coils are wound), and local brightness variance (texture richness).
2. Mic task (~100 Hz) — reads 512 PDM samples at 16 kHz, computes RMS, and applies a linear micGain clamped to 1.0.
3. Main loop (~30 Hz) — reads both metric sets under their locks, applies latch behaviour (most spatial values hold their last known state when the scene goes still, rather than snapping to zero; velocity, motion flag, and microphone RMS are always live), then either:
   • builds and sends the outgoing OSC bundle over UDP, or
   • parses any incoming OSC message and updates the diff threshold or drives the motor PWM outputs accordingly.

The web server (port 80, from the Puara module) and the optional MJPEG snapshot server (port 81) run independently of this loop, reading the latest raw frame and current settings as needed.

Settings reference

All settings live on the Settings web page, apply immediately without reflashing, and persist across reboots.

OSC reference

Outgoing bundle — ~30 Hz

Sent as a single OSC bundle to oscIP:oscPORT; always contains all messages below. Device name comes from config.json (default snake_differ_0).

Incoming messages

First-time setup

1. Flash the firmware (PlatformIO)

pio run --target upload
pio run --target uploadfs

Requires the pioarduino platform (ESP32 Arduino 3.x) — the standard PlatformIO ESP32 2.x platform has a known crash in UDP receive; platformio.ini already points at the correct platform.

2. Connect to the device's WiFi

On first boot (or with no router configured), the device creates its own access point (SSID snake_differ_0, password garnetwillis).

3. Open the web interface

http://snake_differ_0.local (fallback: the AP's default IP, 192.168.4.1, or whatever the serial monitor shows). Config / Scan / Settings pages are all there.

4. Optionally join a studio router

On the Config page, enter router SSID/password and Save; the device connects on next reboot and the AP stays active as a fallback.

Calibration workflow

1. Position the device so the camera sees the full area of interest.
2. Set videoStream = 1, open Config, check framing.
3. Adjust camContrast until the tracked shape reads clearly darker than the background.
4. With the scene still, watch the serial monitor (pio device monitor) for continuous [diff] no motion.
5. Move the shape — [diff] mag=... lines should appear immediately.
6. Tune diffThreshold until detection is clean with minimal noise.
7. Set videoStream = 0, enter oscIP/oscPORT, Save.
8. Confirm the bundle arrives at ~30 Hz in Protokol or the intended receiver.

Receiving OSC

• Protokol — fastest way to confirm the device is sending: add a UDP listener on port 8000 and watch the bundle arrive.
• Max/MSP — udpreceive 8000 → OSC-route /<device>/motion /<device>/bbox ... → unpack per message.
• Logic Pro (no native OSC) — bridge via Max/MSP → IAC Driver → Logic (enable IAC Driver, map OSC values to MIDI CC in Max, then MIDI-learn them in Logic's Smart Controls or a plugin), or use TouchOSC to forward OSC as MIDI CC without needing Max.

Camera preview

When videoStream = 1, an MJPEG stream is served at http://<device_ip>:81/stream (also embedded directly in the Config page) using a single persistent multipart HTTP connection — no polling, no per-frame round-trip. Preview is the same grayscale frame used for diff computation, and camContrast affects both. Turn videoStream off before a show — JPEG encoding runs continuously while a client is connected.

Technical notes

• Diff + pixel analysis + OSC rate: ~30 fps / ~30 Hz
• Frame format: QVGA 320×240 grayscale — 76,800 pixels processed per frame, single pass
• Analysis grid: 6×4 = 24 zones (~80×60 px each); zone index = row × 6 + col, row 0 = top, col 0 = left
• Mic: PDM 16 kHz, RMS over 512 samples, ~100 Hz update
• Motor PWM: 1 kHz, 10-bit (0–1023 range; 0–1000 input maps directly)
• PSRAM: 8 MB used for frame buffers; firmware logs an error and skips camera init if unavailable
• Port 80: Puara web interface; port 81: MJPEG stream (/stream), only active when videoStream = 1
• Platform: pioarduino (ESP32 Arduino 3.x) required

Built with

• Puara Module — WiFi, webserver, filesystem, settings management
• CNMAT OSC — OSC encode/decode
• ESP32 Arduino / ESP-IDF — camera driver, I2S PDM, LEDC PWM