← Projects
Neuromorphic × Edge AI

HyVox: Rethinking Low-Power Voice Intelligence towards Neuromorphic-von Neumann Edge Computing

Low-power speaker identification that enrolls new users and turns away strangers, for about one and a half millijoules per inference.

Snehasis Addy Eduardo Calle Ortiz Prashant Shenoy VP Nguyen
Manning College of Information & Computer Sciences, University of Massachusetts Amherst
Paper · soon BibTeX
1.63mJ
dynamic energy per inference, measured on the chip
13.5×
less dynamic energy than the most efficient embedded ANN
42%
lower latency than the fastest embedded ANN baseline
96.7%
on-chip top-1 in cross-dataset evaluation with 20 speakers in the gallery
In one paragraph

A smart speaker, hearable, or appliance should know who is talking: it should enroll new household members after it ships and treat strangers as guests. But today's TinyML pipelines run a power-hungry ANN on a microcontroller for every window of audio, which makes always-on, on-device recognition hard to fit into a wearable's battery. So HyVox splits the job at the speaker embedding. A spiking network on a neuromorphic chip embeds every window of the audio stream with sparse, event-driven compute, and a microcontroller holds the gallery and makes every decision. The two talk through just 32 spike counts per window, and together they use 13.5× less dynamic energy and 5.8× less total energy per inference than the most efficient embedded ANN.

See it in action

HyVox inside everyday devices

This step: 0 inferences · 0.0 mJ

Illustrative animation. The HyVox prototype runs on a Xylo Audio 3 board with an STM32L476 host; the devices show where the same chip-plus-MCU split could live. Energy uses the measured 1.63 mJ of dynamic energy per inference.

Motivation

Always-on devices need to know who is speaking

Always-on audio devices are moving beyond a fixed list of commands. A shared smart speaker adds household members after it is installed, a hearable recognizes its owner, and an appliance tailors its responses to whoever is speaking and treats strangers as guests. In every case, the speakers that matter are only known after the device ships.

Audio path sets the energy budget

The device listens continuously on a battery, so the network that turns every window of speech into speaker evidence runs at the highest duty cycle in the system.

Von Neumann bottleneck makes MCUs costly

TinyML pipelines run the audio path as an ANN on a von Neumann MCU, where memory and compute are separate: every multiply-accumulate fetches its weight and activation from memory. Repeated on every window, this data movement dominates the device's energy.

Neuromorphic-only solution isn't feasible

Event-driven chips remove that traffic, but they only run the network mapped onto them. They have no general-purpose compute to hold a changing gallery or enroll new speakers, and edge neuromorphic devices have at most 1,000 neurons.

Open question 1How do we learn a speaker embedding that generalizes to unseen speakers within a fixed, tiny hardware budget?
Open question 2How do we combine a neuromorphic chip's low-energy inference with the general-purpose compute that open-set recognition needs?
HyVox concept: a neuromorphic core, a hybrid manager and a von Neumann core inside earbuds, smartphones and smartwatches
HyVox's answer: a hybrid platform. The neuromorphic core turns the incoming signal into spikes and runs the SNN, the von Neumann core holds memory and makes decisions, and a hybrid manager handles readout and scheduling between them. The same split fits earbuds, smartphones, and smartwatches.
How it works

Two processors, one embedding between them

A neuromorphic chip has no general-purpose compute, so it can't hold a gallery that changes at runtime. A microcontroller can hold one, but running the neural network there is costly, since every weight must be fetched from memory for every window of audio. HyVox gives each processor the part it does best.

Mic
16 kHz
Neuromorphic core
Xylo Audio 3
Audio front endsound → spikes
16 channels
HTA-SNNmulti-timescale backbone
832 LIF · 29.0k syn
SEEspiking embedding extractor
32 neurons
→
32 spike counts
70 B per inference
von Neumann core
STM32L476 MCU
Scorecentre, normalize, cosine vs. gallery
Decideaccept the best match or reject as guest
Enrollwrite a 64-byte prototype
No retraining, ever. Enrolling a user is a 64-byte memory write. The network on the chip never changes, and it never trains on a user's audio.
One network for every device. Two households run the same chip configuration and differ only in the gallery on their host.
Cheap decisions. Each decision averages the latest three windows and takes fewer than 10³ operations on the MCU at 20 speakers.
HyVox system overview: inference pipeline across the neuromorphic core, hybrid manager and von Neumann core, and the offline training and mapping pipeline
The full system. Left: the inference pipeline on the device. The microphone feeds the audio front end, the mapped SNN emits 32-d spike embeddings, the hybrid manager reads them out over UART, and the von Neumann core centres, normalizes, scores, enrolls and decides. Right: the offline pipeline that trains the network and fits it to the chip (dense training → pruning → quantization-aware training → CAST, then SEE training on the frozen backbone). Click to enlarge.
What makes it work

Four pieces that fit speaker recognition onto 1,000 neurons

The chip holds at most 1,000 spiking neurons and 31,744 int8 synapses, orders of magnitude below standard speaker encoders. Moving the embedding onto it raises four challenges, from designing the network to running it on silicon. Each piece of HyVox answers one, and each builds on the last.

01 · ARCHITECTURE

HTA-SNN backbone

ChallengeSpeaker identity lives on many timescales, from millisecond formant transitions to pitch and speaking rate over hundreds of milliseconds. In a small SNN, each neuron's decay is its only memory, so the timescales given to so few neurons decide what it can represent.

Our answer: a hierarchical temporal abstraction backbone of 832 neurons that learns one synaptic time constant per neuron. Training decides how many neurons listen fast and how many listen slow, and cross-scale skips carry fast evidence to the slow stages.

+4.1 top-1 vs. SynNet at the same budget
02 · EMBEDDING

Spiking embedding extractor (SEE)

ChallengeThe backbone's state never leaves the chip; only 32 spike counts reach the MCU. Those counts must identify speakers never seen in training, enroll a new one with no weight update, and keep outsiders away, which closed-set training never practises.

Our answer: a 32-neuron spiking readout on the frozen backbone, pretrained on episodes that each enroll new speakers from a few windows and reject outsiders, exactly what the device does after deployment. Its spike counts are the embedding.

+5.7 top-1, −6.0 EER vs. closed-set training
03 · COMPRESSION

Dense-to-sparse pruning

ChallengeA network with enough neurons to cover these timescales needs several times the chip's synapse budget, while a dense network that fits would have under half the neurons. The chip also allows only int8 weights and power-of-two decays.

Our answer: train the backbone dense at 188.4k synapses, then prune it 6.5× to 29.0k with a per-synapse Fisher spike score ranked across all stages and skips. Meanwhile, the time constants settle onto the chip's bit-shift grid, and bitshift QAT recovers the accuracy.

6.5× fewer synapses
04 · DEPLOYMENT

Chip-accurate spiking training (CAST)

ChallengeEven compressed and quantized, the network computes differently on the chip: neurons fire at most 31 events per timestep, every hop adds a timestep of delay, and state saturates. Mapped directly, it loses half of its accuracy.

Our answer: a differentiable PyTorch replica of the chip's integer neuron update, derived from its cycle-accurate simulator. Gradient descent then tunes the exact int8 weights and integer thresholds the chip runs, under all of its limits at once.

Doubles on-chip top-1: 31% → 61%

The HTA-SNN backbone learns its own timescales

Bar height is stage width (neurons). Colour is the median learned synaptic time constant τsyn (timesteps).
Training spreads the timescales from about 2.8 timesteps in the local stage to about 58 in the readout, a range no single shared value covers. Removing the slow global stage costs 3.4 top-1 points even when its neurons are kept elsewhere; removing both skips costs 6.2.
Results

Recognizing households of speakers it has never heard

We evaluate on 20 accent-homogeneous Common Voice households whose speakers appear in neither training set. Each member enrolls from 60 s of speech; every 3 s a new decision is made from the latest three windows, and guests from outside the household must be rejected.

Accuracy as the household grows

Top-1 identificationRejection EER (lower is better)
Mean over 20 households (PyTorch model). Top-1 falls only 4.7 points from 2 to 20 members, and the decline flattens as the household grows. Rejection gets harder with size because a guest is compared against every prototype, so the host picks its threshold by gallery size (0.78 at n = 2, 0.83 at n = 20).
t-SNE of HyVox embeddings for one household with 2, 10 and 20 enrolled speakers
What the embedding space looks like. t-SNE of one household (United States English) with 2, 10, and 20 members enrolled; the guest sample (grey) is the same in every panel. Each member’s decisions form a tight cluster around their prototype (star). As the household grows, prototypes cover more of the region where guests fall, which is why rejection gets harder with size.
System cost

Near-baseline accuracy at a fraction of the energy

Baselines run as int8 models on an ultra-low-power STM32U575 MCU; HyVox runs on Xylo Audio 3 with its STM32L476 host. Both at 50 MHz, per inference.

Total energy per decision

Right column: top-1 at household size n = 6, where the paper's comparison focuses (shared devices typically enroll 2–7 people). MCUs idle at 0.05 mW against the chip's 2.30 mW, but the chip's dynamic power is 7.5–7.9× lower, so HyVox still uses 5.8–11.4× less total energy per decision.
At n = 6, HyVox is within 1.8 top-1 points of the strongest baseline, which spends 7.6× its total energy. Each baseline that is more accurate costs more: from 1.7× the latency to 5.8× the total energy, for 1.0 extra top-1 point. The dense MLP with HyVox's input and hidden widths is both costlier and 9.3 points less accurate at n = 20.
Results by component

What each piece contributes

Each piece against a simpler alternative, with the same budget, data and training.

01 · HTA-SNN BACKBONE
Does learning each neuron's timescale help at the same 29.0k synapses? Top-1 on held-out VoxCeleb2 (%)

HTA beats SynNet, the standard SNN for this chip, by 4.1 top-1 and 1.6 EER points. Giving every neuron one shared timescale costs 12 points, and removing the skip connections costs 6.

02 · SPIKING EMBEDDING EXTRACTOR
Does training the readout to enroll and reject help on new speakers? Top-1 on unseen VoxCeleb1 speakers (%), 20-way

Training on episodes that enroll and reject, the way the device is used, beats standard closed-set training by 5.7 top-1 and 6.0 EER points. It also beats closed-form PCA and LDA readouts, and centring the spike counts adds 3.9 more.

03 · COMPRESSION + QAT
188.4k → 29.0k synapses 6.5× fewer, within the chip's 31.7k budget
Then: can the pruned backbone survive the chip's integer arithmetic? Top-1 on unseen VoxCeleb1 speakers (%)

Simply switching the trained network to the chip's integer bit-shift dynamics costs 37 points. Training for those dynamics with bitshift QAT recovers all of it.

04 · CAST
Does the QAT backbone keep its accuracy on the real chip? Top-1 on unseen VoxCeleb1 speakers (%)

The chip's event caps, hop delays, int8 grid and integer thresholds halve top-1 when the model is mapped directly. CAST recovers 29.5 of the 31.1 lost points. Retraining with the event caps alone reaches only 30.0.

Deploying the HTA-SNN backbone: synapse compression, synapses kept per component, bitshift QAT and CAST fine-tuning curves
From dense model to chip, step by step. (A) Pruning from 188.4k to 29.0k synapses (6.5×). (B) Global ranking lets each component keep what it needs: skip A and the local stage keep nearly twice the average. (C) Bitshift QAT recovers integer-dynamics accuracy within a few epochs. (D) CAST fine-tuning takes on-chip top-1 from 31% to 62% on held-out VoxCeleb1 speakers.
On-device evaluation

HyVox running on real hardware

HyVox hardware: the dense model is pruned, quantized and CAST-trained into a hardware-aware model, mapped to Xylo Audio 3, stacked on an STM32L476 host
The prototype. The dense trained model is pruned, quantized, and fine-tuned with CAST into a hardware-aware model, then written to the Xylo Audio 3 board, which is stacked on an STM32L476 Nucleo host. The audio stream enters the chip directly. We tested it on 20 new households with no speakers in common with the earlier ones.
Recurrent synapses on chip
27,259 / 31,744
Hidden + readout neurons
832 + 32
Latency per window
961.9 ms
Gallery cost
64 B / speaker

PyTorch model vs. on-device system

PyTorchOn-device (Xylo Audio 3)
Mean over 20 households with 95% confidence intervals. At n = 20 the chip is 1.4 points higher in top-1 and 2.4 points lower in EER. Training through the chip's quantization and event limits seems to act as a regularizer, helping the backbone generalize from VoxCeleb interviews to read speech recorded at home.
Conclusion

HyVox shows that a neuromorphic chip can serve as the low-power embedding engine for open-set speaker recognition at the edge. The chip embeds every window, a small microcontroller enrolls, identifies and rejects, and users are added with a 64-byte write instead of retraining. Deployed on real hardware, it matches its PyTorch model while using 13.5× less dynamic energy than the most efficient embedded ANN.

Cite

BibTeX

@article{addy2026hyvox,
  title  = {HyVox: Rethinking Low-Power Voice Intelligence towards
            Neuromorphic-von Neumann Edge Computing},
  author = {Addy, Snehasis and Calle Ortiz, Eduardo and
            Shenoy, Prashant and Nguyen, VP},
  year   = {2026}
}