At USC, Dr. Sook-Lei Liew is testing whether watching a virtual avatar that moves in response to brain commands can activate portions of the brain damaged by stroke.
Photo: Nate Jensen
I am hooked up to a 16-channel brain machine interface with 12 channels of EEG on my head and ears and four channels of electromyography (EMG) on my arms. An Oculus Rift occludes my vision.
Two inertial measurement units (IMU) are stuck to my wrists and forearms, tracking the orientation of my arms, while the EMG monitors my electrical impulses and peripheral nerve activity.
Dr. Sook-Lei Liew, Director of USC’s Neural Plasticity and Neurorehabilitation Laboratory, and Julia Anglin, Research Lab Supervisor and Technician, wait to record my baseline activity and observe a monitor with a representation of my real arm and a virtual limb. I see the same image from inside the Rift.
“Ready?” asks Dr. Liew. “Don’t move—or think.”
I stay still, close my eyes, and let my mind go blank. Anglin records my baseline activity, allowing the brain-machine interface to take signals from the EEG and EMG, alongside the IMU, and use that data to inform an algorithm that drives the virtual avatar hand.
“Now just think about moving your arm to the avatar’s position,” says Dr. Liew.
I don’t move a muscle, but think about movement while looking at the two arms on the screen. Suddenly, my virtual arm moves toward the avatar appendage inside the VR world.
Something happened just because I thought about it! I’ve read tons of data on how this works, even seen other people do it, especially inside gaming environments, but it’s something else to experience it for yourself.
“Very weird isn’t it?” says David Karchem, one of Dr. Liew’s trial patients. Karchem suffered a stroke while driving his car eight years ago, and has shown remarkable recovery using her system.
“My stroke came out of the blue and it was terrifying, because I suddenly couldn’t function. I managed to get my car through an intersection and call the paramedics. I don’t know how,” Karchem says.
He gets around with a walking stick today, and has relatively normal function on the right side of his body. However, his left side is clearly damaged from the stroke. While talking, he unwraps surgical bandages and a splint from his left hand, crooked into his chest, to show Dr. Liew the progress since his last VR session.
As a former software engineer, Karchem isn’t fazed by using advanced technology to aid the clinical process. “I quickly learned, in fact, that the more intellectual and physical stimulation you get, the faster you can recover, as the brain starts to fire. I’m something of a lab rat now and I love it,” he says.
Karchem is participating in Dr. Liew’s REINVENT (Rehabilitation Environment using the Integration of Neuromuscular-based Virtual Enhancements for Neural Training) project, funded by the American Heart Association, under a National Innovative Research Grant. It’s designed to help patients who have suffered strokes reconnect their brains to their bodies.
“My PhD in Occupational Science, with a concentration in Cognitive Neuroscience, focused on how experience changes brain networks,” explains Dr. Liew. “I continued this work as a Postdoctoral Fellow at the National Institute of Neurological Disorders and Stroke at the National Institutes of Health, before joining USC, in my current role, in 2015.
“Our main goal here is to enhance neural plasticity or neural recovery in individuals using noninvasive brain stimulation, brain-computer interfaces and novel learning paradigms to improve patients’ quality of life and engagement in meaningful activities,” she says.
Here’s the science bit: the human putative mirror neuron system (MNS) is a key motor network in the brain that is active both when you perform an action, like moving your arm, and when you simply watch someone else—like a virtual avatar—perform that same action. Dr. Liew hypothesizes that, for stroke patients who can’t move their arm, simply watching a virtual avatar that moves in response to their brain commands will activate the MNS and retrain damaged or neighboring motor regions of the brain to take over the role of motor performance. This should lead to improved motor function.
“In previous occupational therapy sessions, we found many people with severe strokes got frustrated because they didn’t know if they were activating the right neural networks when we asked them to ‘think about moving’ while we physically helped them to do so,” Dr. Liew says. “If they can’t move at all, even if the right neurological signals are happening, they have no biological feedback to reinforce the learning and help them continue the physical therapy to recover.”
For many people, the knowledge that there’s “intent before movement”—in that the brain has to “think” about moving before the body will do so, is news. We also contain a “body map” inside our heads that predicts our spacetime presence in the world (so we don’t bash into things all the time and know when something is wrong). Both of these brain-body elements face massive disruption after a stroke. The brain literally doesn’t know how to help the body move.
What Dr. Liew’s VR platform has done is show patients how this causal link works and aid speedier, and less frustrating, recovery in real life.
From the Conference Hall to the Lab
She got the idea while geeking out in Northern California one day.
“I went to the Experiential Technology Conference in San Francisco in 2015, and saw demos of intersections of neuroscience and technology, including EEG-based experiments, wearables, and so on. I could see the potential to help our clinical population by building a sensory-visual motor contingency between your own body and an avatar that you’re told is ‘you,’ which provides rewarding sensory feedback to reestablish brain-body signals.
“Inside VR you start to map the two together, it’s astonishing. It becomes an automatic process. We have seen that people who have had a stroke are able to ’embody’ an avatar that does move, even though their own body, right now, cannot,” she says.
Dr. Liew’s system is somewhat hacked together, in the best possible Maker Movement style; she built what didn’t exist and modified what did to her requirements.
“We wanted to keep costs low and build a working device that patients could actually afford to buy. We use Oculus for the [head-mounted display]. Then, while most EEG systems are $10,000 or more, we used an OpenBCI system to build our own, with EMG, for under $1,000.
“We needed an EEG cap, but most EEG manufacturers wanted to charge us $200 or more. So, we decided to hack the rest of the system together, ordering a swim cap from Amazon, taking a mallet and bashing holes in it to match up where the 12 positions on the head electrodes needed to be placed (within the 10-10 international EEG system). We also 3D print the EEG clips and IMU holders here at the lab.
“For the EMG, we use off-the-shelf disposable sensors. This allows us to track the electromyography, if they do have trace muscular activity. In terms of the software platform, we coded custom elements in C#, from Microsoft, and implemented them in the Unity3D game engine.”
Dr. Liew is very keen to bridge the gap between academia and the tech industry; she just submitted a new academic paper with the latest successful trial results from her work for publication. Last year, she spoke at SXSW 2017 about how VR affects the brain, and debuted REINVENT at the conference’s VR Film Festival. It received a “Special Jury Recognition for Innovative Use of Virtual Reality in the Field of Health.”
Going forward, Dr. Liew would like to bring her research to a wider audience.
“I feel the future of brain-computer interfaces splits into adaptive, as with implanted electrodes, and rehabilitative, which is what we work on. What we hope to do with REINVENT is allow patients to use our system to re-train their neural pathways, [so they] eventually won’t need it, as they’ll have recovered.
“We’re talking now about a commercial spin-off potential. We’re able to license the technology right now, but, as researchers, our focus, for the moment, is in furthering this field and delivering more trial results in published peer-reviewed papers. Once we have enough data we can use machine learning to tailor the system precisely for each patient and share our results around the world.”
If you’re in L.A., Dr. Liew and her team will be participating in the Creating Reality VR Hackathon from March 12-15 at USC. Details here.
via How Virtual Avatars Help Stroke Patients Improve Motor Function | News & Opinion | PCMag.com