This work presents the design, implementation, and evaluation of a P300-based brain-machine interface (BMI) developed to control a robotic hand-orthosis. The purpose of this system is to assist patients with amyotrophic lateral sclerosis (ALS) who cannot open and close their hands by themselves. The user of this interface can select one of six targets, which represent the flexion-extension of one finger independently or the movement of the five fingers simultaneously. We tested offline and online our BMI on eighteen healthy subjects (HS) and eight ALS patients. In the offline test, we used the calibration data of each participant recorded in the experimental sessions to estimate the accuracy of the BMI to classify correctly single epochs as target or non-target trials. On average, the system accuracy was 78.7% for target epochs and 85.7% for non-target trials. Additionally, we observed significant P300 responses in the calibration recordings of all the participants, including the ALS patients. For the BMI online test, each subject performed from 6 to 36 attempts of target selections using the interface. In this case, around 46% of the participants obtained 100% of accuracy, and the average online accuracy was 89.83%. The maximum information transfer rate (ITR) observed in the experiments was 52.83 bit/min, whereas that the average ITR was 18.13 bit/min. The contributions of this work are the following. First, we report the development and evaluation of a mind-controlled robotic hand-orthosis for patients with ALS. To our knowledge, this BMI is one of the first P300-based assistive robotic devices with multiple targets evaluated on people with ALS. Second, we provide a database with calibration data and online EEG recordings obtained in the evaluation of our BMI. This data is useful to develop and compare other BMI systems and test the processing pipelines of similar applications.
1. Introduction
Since the early developments of BMIs, one of the most promising applications of this technology is the use of neuroprosthetic devices to assist people with reduced mobility. There is a consensus among researchers of this area that BMIs may significantly improve the lives of patients who suffer neuromuscular disorders such as ALS. Even so, despite all the efforts in the last three decades to design and implement reliable BMI systems, the goal of developing functional neuroprostheses has not been reached yet. Researchers and engineers must solve many technical and practical problems before bringing this technology into everyday life. Some open issues concerning the development of robust brain-controlled applications are the ability of the system to interpret the user’s intentions accurately, the time to process and analyze brain signals, and the stability of performance over time (Murphy et al., 2016).
A BMI is a system that translates cerebral activity into commands to communicate with an external device, bypassing the normal neuromuscular pathways (Wolpaw et al., 2002; Aydin et al., 2018). There are various techniques to register brain signals, but the non-invasive neuroimage modality most widely used in BMI applications is electroencephalography (EEG) because of its high temporal resolution, low cost, and mobility (Flores et al., 2018; Xiao et al., 2019). Among EEG-based BMIs, the P300 paradigm is one of the most popular techniques for building applications with multiple options because it allows achieving high accuracies without the need for long calibration sessions (Hwang et al., 2013; De Venuto et al., 2018). Compared with other paradigms, P300-based BMIs have higher bit rates than motor imagery interfaces, while the stimulation technique for evoking P300 potentials is less visually fatiguing than the method used to elicit steady-state visually evoked potentials (Cattan et al., 2019).
The P300 signal is an event-related potential (ERP) component observed in the electroencephalogram elicited about 300 ms after the perception of an oddball or relevant auditory, visual, or somatosensory stimulus (Cattan et al., 2019). Typically, in a P300-based BMI, characters, syllables, or icons presented on a computer screen flash randomly one at a time while the user focuses attention on one particular graphical element (target stimulus). Each flashing stimulus represents an option, action, or command that the system can execute. The user selects one option of the interface by counting or performing a cognitive task every time the target stimulus is highlighted. Because the target option flashes randomly, this stimulus produces a P300 evoked potential synchronized with the flickering event in the timeline. In this way, a P300-based BMI identifies which option is evoking an ERP to decode the user’s intentions and perform the desired action.
Numerous published works have reported examples of P300-based BMIs for communication and control, including spellers (Kleih et al., 2016; Poletti et al., 2016; Okahara et al., 2017; Flores et al., 2018; Guy et al., 2018; Deligani et al., 2019; Shahriari et al., 2019), authentication systems (Yu et al., 2016; Gondesen et al., 2019), assistive robots (Arrichiello et al., 2017), smart home environments (Achanccaray et al., 2017; Masud et al., 2017; Aydin et al., 2018), neurogames (Venuto et al., 2016), remote vehicles (De Venuto et al., 2017; Nurseitov et al., 2017), wheelchairs (De Venuto et al., 2018), and robotic arms (Tang et al., 2017; Garakani et al., 2019). Because the development of assistive technologies for motor-impaired people is one of the major purposes of BMI research, some groups have evaluated similar applications in clinical environments on people with neurological disorders or reduced mobility. Regarding medical applications, we can find P300-based BMIs for ALS (Liberati et al., 2015; Schettini et al., 2015; Poletti et al., 2016; Guy et al., 2018; Deligani et al., 2019; Shahriari et al., 2019; McFarland, 2020), Alzheimer’s (Venuto et al., 2016), spinocerebellar ataxia (Okahara et al., 2017), and post-stroke paralysis (Kleih et al., 2016; Achanccaray et al., 2017; Flores et al., 2018). Recently, P300-based BMIs have also been proposed for rehabilitation contexts (Kleih et al., 2016), and diagnosis/evaluation purposes (Poletti et al., 2016; Venuto et al., 2016; Deligani et al., 2019; Shahriari et al., 2019).
Some studies have stated the benefits of orthoses for ALS patients (Tanaka et al., 2013; Ivy et al., 2014); however, the implementation of BMI-controlled robotic hand-orthoses for this target population remains underexplored in comparison to the application of these systems for other neuromotor disorders. Moreover, most of the recent published BMIs for ALS are designed for communication purposes (Vaughan, 2020). Similarly, while the employment of BMI-controlled hand-orthoses is well-known in other neuromotor conditions (e.g., stroke recovery), the effect of the use of these systems in ALS patients remains poorly investigated. A critical step toward the development of practical robotic neuroprostheses for people with ALS is the evaluation of this technology in different scenarios. It is essential to determine if ALS patients can operate this particular mind-controlled application and evaluate the possible effect of a hand-orthosis on the user’s experience and performance.
This work presents the development and evaluation of a P300-based BMI coupled with a robotic hand-orthosis device. The purpose of this system is to assist people with ALS to perform movements of individual fingers of one hand, or more complex tasks that involve a sequence of actions of one or more fingers. Eighteen healthy participants and eight ALS patients conducted an experiment in which they tested the proposed BMI selecting a sequence of actions that the robotic hand-orthosis executed. In the evaluation of this BMI, we considered six types of operations: the flexion-extension of individual fingers, and the flexion-extension of the five fingers simultaneously.
In the experiments, we recorded the data used in the training phase of the BMI, and the EEG signals measured during the online tests. The training data was used to evaluate offline the performance of the classification model implemented in the BMI to discriminate between target and non-target epochs. Additionally, we analyzed the P300 responses of the participants to determine if there are subjects without clear evoked potentials. In the online tests, we calculated the classification accuracy and the selection times of the BMI. It is important to say that some selections were made without connecting the hand-orthosis to the system to evaluate the effect of the robotic device in the online accuracy of the BMI.
To our knowledge, our system is the first P300-based BMI that allows ALS patients to perform sequences of movements of individual or two or more digits simultaneously; it is important to consider the advantage of our system to allow the individual movement of the digits since ALS is associated with the degeneration of the corticospinal tract (Sarica et al., 2017) that allows to perform the fine finger motor tasks (Levine et al., 2012). Besides, being a P300-based system, the calibration precises a minimum time consuming calibration, reducing the fatigue of patients in comparison with other systems.
Another contribution of this work is the dataset obtained in the experimental sessions of the proposed BMI. This dataset contains the training data and the online recordings of 26 participants. The calibration samples are useful to evaluate different machine learning models of P300-based BMIs, whereas the online signals can be used to test practical systems without the need for real participants. The relevance of this database resides in the importance of providing high-quality EEG observations that represent both control and ALS groups. Any researcher may evaluate other P300-based BMIs and verify if their proposals can correctly identify the user’s intentions in online conditions.
The remainder of this paper is divided into three sections. Section 2 describes the hardware and software components of the mind-controlled hand-orthosis, and the experimental setup under which we tested the BMI. Section 3 shows the results obtained from the system evaluation, while section 4 discusses the implications of the results and the conclusions derived from this work.
2. Materials and Methods
2.1. Brain-Machine Interface
The proposed system consists of a P300-based BMI coupled with a Hand Of Hope robotic arm (Rehab-Robotics Company, China). This hand-orthosis is a therapeutic device with five DC linear motors designed initially for the rehabilitation of post-stroke patients (Aggogeri et al., 2019). There is a detailed description of the Hand of Hope and its functionality in Ho et al. (2011). To communicate the orthosis with the BMI, we enabled a wireless communication channel to send the position of each motor during the execution of one movement or sequence of movements. In this way, the user selects one action to perform with the hand-orthosis using the P300-based interface.
Figure 1 sketches the components of the mind-controlled hand-orthosis, and how the users interact with them. The main hardware components of the interface are:
• An EEG recording system (a g.GAMMASYS active wet electrode arrangement and a g.USBamp amplifier provided by g.tec medical engineering GmbH, Austria). For this study, the sampling rate was 256 Hz, and we used eight monopolar electrodes, placed according to the 10–20 international system at positions Fz, Cz, P3, Pz, P4, PO7, PO8, and Oz. The ground electrode was located at AFz, and the reference electrode on the right earlobe.
• A Hand of Hope robotic arm. The users can wear the robotic device on any hand.
• A monitor that displays the graphical user interface (GUI) of the BMI.
• A computer that processes the EEG signals, synchronizes the stimulus presentation, and sends the control commands to the hand-orthosis.
FIGURE 1
[…]