arXiv q-bio.NC· Riki Shimizu, Richard J. Antonello, Chandan Singh, Nima Mesgarani·· 6 小时前AI 评分42
可解释特征嵌入提升并解释音频模型的脑编码表现
Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models
AI 导读
研究用 mel 频谱、Gabor 滤波器组、语音存在、音素、句法、语义六类可解释特征及 Whisper、HuBERT、WavLM 的上下文嵌入构建线性编码模型，量化 ECoG 响应方差。方差分解显示，SFM 与大脑的对齐大部分可由其编码简单可解释语音特征的能力解释，且各层在低层与高层脑相关特征间存在系统性权衡。SFM 还学到低层特征无法解释的脑相关语义，该能力随模型规模和上下文长度提升。
正文 · 原文
Abstract:Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs' alignment with the brain can be mostly explained by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be explained by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.
| Comments: | Accepted into PLOS Computational Biology |
| Subjects: | Neurons and Cognition (q-bio.NC); Sound (cs.SD); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2507.16080 [q-bio.NC] |
| (or arXiv:2507.16080v3 [q-bio.NC] for this version) | |
| https://doi.org/10.48550/arXiv.2507.16080 arXiv-issued DOI via DataCite |
Submission history
From: Riki Shimizu [view email]
[v1]
Mon, 21 Jul 2025 21:33:36 UTC (6,587 KB)
[v2]
Wed, 24 Sep 2025 22:04:30 UTC (7,252 KB)
[v3]
Tue, 20 Oct 2026 20:55:37 UTC (8,408 KB)
来源：arXiv q-bio.NC · arxiv.org