arXiv cs.LG + cs.RO + cs.HC· Julian Silva, Jaliya Kodithuwakku, Navindu Sirimanna, Shaveen Herath, Wageesha N. Manamperi, Joshua P. Kulasingham·· 6 小时前AI 评分55
NeuroCodec：用 EEG 引导潜空间生成模型提取目标说话人语音
NeuroCodec: EEG-Guided Latent-Domain Generative Model for High-Fidelity Target Speaker Extraction
AI 导读
研究团队提出 NeuroCodec，一个在神经音频编解码器潜空间中运行的 EEG 引导目标说话人提取框架，通过图卷积建模 EEG 空间依赖，并用注意力机制与 Mamba 时序建模融合脑电和语音表征。
正文 · 原文
Abstract:Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.
| Comments: | Supplementary results, audio files and the spectrograms are available at the following link: this https URL |
| Subjects: | Audio and Speech Processing (eess.AS); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.29815 [eess.AS] |
| (or arXiv:2610.29815v1 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29815 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joshua Kulasingham [view email]
[v1]
Wed, 21 Oct 2026 14:38:19 UTC (6,431 KB)
来源：arXiv cs.LG + cs.RO + cs.HC · arxiv.org