A $5 chip runs open-vocabulary speech recognition: 13M-parameter model stays in flash, with 1.8–2.8 fewer errors than the vendor recognizer on the same chip

Open-vocabulary English recognition now runs entirely on a $5 microcontroller, with no cloud round trip and no neural accelerator. On such chips, speech recognition has in practice been limited to keyword spotting or a predefined command list, and even the chip vendor's most capable recognizer accepts only fixed command lists. On LibriSpeech test-clean/test-other the on-device int8 arithmetic reaches 3.70%/8.23% word error rate against 3.68%/8.11% for the full-precision reference, a gap of 0.02 and 0.12 points; on test-clean every on-chip configuration makes 1.8–2.8 fewer errors than the vendor's MultiNet7 on the same chip.

The model fits and runs because weights stay in flash and are read on a bandwidth-aware schedule, rather than being moved wholesale into memory. At int8 the 13M-parameter model occupies 14.0 MB, which does not fit in the 512 KB of on-chip SRAM and fits only partially in PSRAM, since PSRAM must also hold working buffers. The engine keeps the model in a flash partition mapped into the address space, copies the most frequently read matrices to PSRAM and keeps activations in SRAM; processing each layer in blocks of 64 frames fetches every weight byte once per block, cutting modelled weight traffic from about 18 MB/s to 7.5 MB/s.

Quantization costs almost no accuracy and needs no retraining. Earlier integer Conformer work generally relied on quantization-aware training and targeted accelerators or mobile CPUs. Weights use symmetric int8 with one scale per output channel, activations are quantized dynamically with one scale per frame just before each matrix product, and accumulation is exact in int32; the int4 variant instead needs 8,000 steps of quantization-aware fine-tuning and reaches 4.61%/9.98% greedy.

A streaming variant returns text sooner after the speaker stops, at a cost in accuracy. Utterance mode must wait for the speaker to finish and then run the whole encoder, so text appears only after the pause. Fed at the rate of speech, clips of 3.6–10.6 s deliver final text 4.1–11.1 s after the last sample, against 7.2–22.4 s for the same clips in utterance mode; with the language model, streaming reaches 4.86%/11.03% on LibriSpeech against 3.33%/7.15% in utterance mode.

The reported results cover one ESP32-S3-WROOM-1-N16R8 module on one board, running on a single core at about twice the audio duration, so they suit short commands where several seconds of delay are acceptable rather than live transcription or conversation. The streaming variant roughly halves the wait after the speaker stops, at a cost in accuracy. Language coverage is English and Spanish.

The two-core schedule is correct in the emulator but produces wrong transcripts and occasional crashes on silicon at 240 MHz, and the cause is not known, so every measurement uses one core. Chunked streaming loses about 1.5 points on test-clean against utterance mode, and the final text still arrives roughly in proportion to utterance length. Board speed figures cover 27 clips for CTC int8 and 10–12 clips for the other models, all at most 10.6 s; the 2.4 MB PSRAM peak for a 20 s utterance comes from the emulator.