Neuromorphic In-Ear LLM Conversation Assistant
Research project at LASS, UMass Amherst
Developing an energy-efficient in-ear conversation assistant that integrates LLMs with neuromorphic audio sensing for seamless, always-on operation on low-power IoT devices. The system targets 0.54 mW energy efficiency through event-driven audio processing, replacing traditional 16 kHz continuous sampling to minimize power usage.
Some highlights of this project are:
- Designed the full system architecture for an always-on neuromorphic in-ear assistant.
- Optimized the LLM inference pipeline for low-latency response and low-power operation.
- Implemented adaptive event-driven thresholds for context-aware wake-word detection.
- Deployed and benchmarked the prototype on a neuromorphic HDK, demonstrating measurable efficiency gains and latency improvements over the LlamaPIE baseline model.
Advisor: Prof. Prashant Shenoy and Prof. VP Nguyen, Laboratory for Advanced System Software, UMass Amherst.
Figure: Colored neural network by Glosser.ca, CC BY-SA 3.0, via Wikimedia Commons.