UCSF 团队在 Nature Neuroscience 发表研究，用单块皮层植入体同时解码语音与手势

Experimental technology translates brain signals into speech and gestures through a virtual avatar.

Researchers at the University of California, San Francisco (UCSF), have demonstrated an experimental brain-computer interface that enables people with severe paralysis to communicate through a virtual avatar using both speech and body gestures.

The system translates neural activity into digital commands, allowing users to display text representing attempted speech while a personalized avatar performs corresponding gestures. The findings, published online in Nature Neuroscience on September 14, 2026, could help advance assistive technologies designed to make communication more expressive for people who have lost the ability to speak or move.

The research team received funding from the US National Institutes of Health (NIH), including its National Institute on Deafness and Other Communication Disorders.

How the technology works

The brain-computer interface uses a surgically implanted, high-density electrode array positioned over the sensorimotor cortex, a brain region involved in controlling movement, including movements associated with speech.

The array records electrical signals from the brain. Machine-learning models then interpret these signals and translate them into commands that control a personalized, full-body virtual avatar.

When a participant attempts to communicate, the system can display decoded speech as text while animating gestures such as waving or giving a thumbs-up. The speech and gesture outputs can be generated simultaneously or separately.

Previous brain-computer interface research has demonstrated ways to decode speech or movement individually. This study investigated whether a single cortical implant could support both forms of communication, including when they are attempted at the same time.

What the study found

The researchers examined neural recordings from three participants with severe paralysis affecting their vocal tracts. Two also had severe limb paralysis, while the third had moderate limb paralysis.

The team first tested whether neural signals associated with different movements could be decoded reliably. It then developed parallel speech and gesture decoders that enabled two participants to control a virtual avatar during structured tasks and conversational interactions.

The findings showed that training the models with examples of speech and gestures attempted together improved their ability to interpret simultaneous communication. Models trained only on isolated speech or gestures did not automatically perform as well when participants attempted both at once.

During conversational testing, the participant with amyotrophic lateral sclerosis (ALS) achieved average decoding accuracies of 85 per cent for gestures and 75 per cent for speech across five test blocks. A second participant, who had experienced a stroke, achieved median decoding accuracies of 100 per cent for both speech and gestures across three test blocks.

These figures reflect results from specific participants in a controlled experiment, not the expected accuracy for all users or every communication situation. The system was tested using a restricted set of phrases and gestures rather than unrestricted conversation.

Why the findings matter

Human communication involves more than spoken words. Hand movements, nods and other gestures can reinforce meaning, express reactions and help people participate in conversations.

Conditions such as ALS and the effects of a stroke can impair both speech and body movement, making communication more difficult. Existing assistive technologies can help people express themselves, but some provide fewer ways to convey gestures and other non-verbal cues.

A system that combines speech and gestures could eventually offer additional ways for people with severe paralysis to communicate. A virtual avatar may represent intended movements that a person cannot physically perform, allowing visible gestures to accompany words.

However, the study does not show that the implant restores natural speech or physical movement. Instead, it demonstrates that recorded brain activity can be translated into digital representations of intended communication.

Challenges remain before wider use

Despite the promising findings, the technology remains experimental and is not yet an established clinical communication device.

The avatar-based communication experiments involved two participants, while the broader study examined neural signals from three. The small sample means that further research is needed to determine how well the approach would work for people with different conditions and degrees of paralysis.

The system also relies on implanted electrodes, external computing equipment and machine-learning models tailored to individual users. Its performance remains limited by the phrases and gestures included in the training and testing process.

Further studies will need to examine larger groups of participants, expand the range of recognised phrases and gestures, improve decoding performance and assess reliability over longer periods.

The research team has identified a fully implantable, wireless version as a future development goal. Such a system could reduce reliance on external wired equipment, but its feasibility, safety and performance would need to be evaluated.

A step towards more expressive communication

The findings provide proof of concept that a single cortical implant can support the decoding of speech and upper-body gestures, with the results represented through a personalized virtual avatar.

Although substantial technical and clinical challenges remain, the research marks a step towards brain-computer interfaces that combine different forms of communication rather than treating speech and movement separately.

Further development could eventually help people with severe paralysis communicate in more flexible and expressive ways. For now, the results demonstrate a promising experimental approach rather than a ready-to-use clinical solution.