# Connectionist Temporal Classification

Across days, neural drift degrades a fixed intracortical speech decoder. One study confined adaptation to a 256×256 affine transform and bias at the neural input, updating 65,792 parameters, under 0.05% of the full model, while the recurrent encoder and phoneme classifier stayed frozen, using 40 or 80 labeled trials for supervised calibration. On the public T12 dataset, macro phoneme error rate fell 11.37% and 15.51% across six chronologically held-out sessions, with all six improving under all three seeds; on the independent T15 dataset the same frozen rule gave 2.72% and 10.05% reductions, with more session-to-session variability and neither budget below the nominal 0.05 reference threshold.

Across days, neural drift degrades a fixed intracortical speech decoder. One study confined adaptation to a 256×256 affine transform and bias at the neural input, updating 65,792 parameters, under 0.05% of the full model, while the recurrent encoder and phoneme classifier stayed frozen, using 40 or 80 labeled trials for supervised calibration. On the public T12 dataset, macro phoneme error rate fell 11.37% and 15.51% across six chronologically held-out sessions, with all six improving under all three seeds; on the independent T15 dataset the same frozen rule gave 2.72% and 10.05% reductions, with more session-to-session variability and neither budget below the nominal 0.05 reference threshold.

Oído runs open-vocabulary English speech recognition entirely on a $5 ESP32-S3, a chip with no neural accelerator. Such microcontrollers previously managed only keyword spotting or a fixed command list, while a 13M-parameter Conformer-CTC occupies 14.0 MB at int8, too large for the 512 KB of on-chip SRAM and only partly placeable in PSRAM. The engine schedules execution around flash and PSRAM bandwidth, reaching 3.70%/8.23% word error rate on LibriSpeech test-clean/test-other, within 0.12 points of full precision and 1.8–2.8 fewer errors than the vendor recognizer on the same chip; measured on a board it runs at 1.97 times the audio duration on one core, not yet real time.

Oído runs open-vocabulary English speech recognition entirely on a $5 ESP32-S3, a chip with no neural accelerator. Such microcontrollers previously managed only keyword spotting or a fixed command list, while a 13M-parameter Conformer-CTC occupies 14.0 MB at int8, too large for the 512 KB of on-chip SRAM and only partly placeable in PSRAM. The engine schedules execution around flash and PSRAM bandwidth, reaching 3.70%/8.23% word error rate on LibriSpeech test-clean/test-other, within 0.12 points of full precision and 1.8–2.8 fewer errors than the vendor recognizer on the same chip; measured on a board it runs at 1.97 times the audio duration on one core, not yet real time.

The work introduces Align Then Reason (ATR), in which a monotonic CTC alignment scorer between candidate-text phonetic units and frame-level lip representations supplies per-unit soft tokens and a calibrated global alignment score to an LLM reasoner that outputs a Yes/No judgment with a short explanation, raising Qwen3.5-9B mean AUC from 0.610 to 0.920 on a seven-language benchmark and improving on three unseen MuAViC languages and two real dubbing downstream tasks.

The work introduces Align Then Reason (ATR), in which a monotonic CTC alignment scorer between candidate-text phonetic units and frame-level lip representations supplies per-unit soft tokens and a calibrated global alignment score to an LLM reasoner that outputs a Yes/No judgment with a short explanation, raising Qwen3.5-9B mean AUC from 0.610 to 0.920 on a seven-language benchmark and improving on three unseen MuAViC languages and two real dubbing downstream tasks.