Brain implant decodes speech and gestures simultaneously for the first time

Researchers at UCSF built an AI-powered brain-computer interface that reads a paralyzed person's attempts to speak and gesture at the same time, converting them into text and digital avatar movements. The system was tested on two participants with limited vocabularies and is not yet fully accurate.

• The implant is a high-density electrocorticography array placed on the brain surface
• A model trained on both speech and gestures outperformed single-modality models
• The system distinguished 43–45 movements, except left-hand and leg movements
• Decoded signals drove an Unreal Engine avatar while speech appeared as on-screen text