The proportion of rehabilitation doctors and patients mismatch is very grim in the context of social aging. The Family Rehabilitation System captures the profound information of the trainer’s movements through the kinect bone tracing technique, allowing the doctor to remotely master the patient’s training progress. With the help of computers and the Internet, the patient can consult a physician, while the physician can remotely guide and launch the training “prescription” through the Internet according to the training effect. Patients can have rehabilitated training at home. The results of the test showed that the system has a positive effect on the rehabilitation of the patient.
Posts Tagged Kinect
Interactive technologies are beneficial to stroke recovery as rehabilitation interventions; however, they lack evidence for use as assessment tools. Mystic Isleis a multi-planar full-body rehabilitation game developed using the Microsoft Kinect® V2. It aims to help stroke patients improve their motor function and daily activity performance and to assess the motions of the players. It is important that the assessment results generated from Mystic Isle are accurate. The Kinect V2 has been validated for tracking lower limbs and calculating gait-specific parameters. However, few studies have validated the accuracy of the Kinect® V2 skeleton model in upper-body movements. In this paper, we evaluated the spatial accuracy and measurement validity of a Kinect-based game Mystic Isle in comparison to a gold-standard optical motion capture system, the Vicon system. Thirty participants completed six trials in sitting and standing. Game data from the Kinect sensor and the Vicon system were recorded simultaneously, then filtered and sample rate synchronized. The spatial accuracy was evaluated using Pearson’s r correlation coefficient, signal to noise ratio (SNR) and 3D distance difference. Each arm-joint signal had an average correlation coefficient above 0.9 and a SNR above 5. The hip joints data had less stability and a large variation in SNR. Also, the mean 3D distance difference of joints were less than 10 centimeters. For measurement validity, the accuracy was evaluated using mean and standard error of the difference, percentage error, Pearson’s r correlation coefficient and intra-class correlation (ICC). Average errors of maximum hand extent of reach were less than 5% and the average errors of mean and maximum velocities were about 10% and less than 5%, respectively. We have demonstrated that Mystic Isle provides accurate measurement and assessment of movement relative to the Vicon system.
In the past decade and quite rapidly in the past five years, Natural User Interfaces (NUIs) and video games have grown in popularity in both consumer applications and in healthcare [1–3]. Specifically, physical rehabilitation (e.g., physical and occupational therapy) has embraced novel NUI applications in clinics, hospitals, nursing homes, and the community [4–6]. Robotic systems have long included game-based and NUI-based user interfaces and most robotic devices provide some form of physical assistance to the patient and/or haptic feedback [7, 8]. With the release of the Nintendo Wii in 2008, many NUI applications for healthcare moved away from bulky, expensive robotics and embraced the portable nature of movement and gesture recognition devices and systems. One of the biggest breakthroughs for this field came in 2010 when Microsoft released the Kinect sensor to accompany its Xbox console system. Within days and weeks of the Kinect’s release, hackers, universities, and companies began to exploit its markerless movement sensing abilities for educational and healthcare use. Since then, there has been an exponential increase in the number of studies that report the use of the Kinect as the input device for a NUI-based rehabilitation game or feedback application [9, 10].
In 2014, Jintronix was the first company to receive FDA approval for its rehabilitation game system that uses the Microsoft Kinect. There are a number of similar companies that utilize the Kinect sensor including SeeMee , VirtualRehab , Reflexion Health , MIRA , MotionCare360 , and 5Plus Therapy . Many of these systems are marketed for delivering rehabilitation therapy in the home setting. This type of delivery is termed “tele-rehabilitation” and can involve remote monitoring by the therapist or virtual sessions over teleconferencing software [17, 18]. For telerehabilitation or remote sessions, it is imperative that the data the therapist receives from the system or movement-sensing device (such as the Microsoft Kinect) are accurate and reliable. If the therapist plans to use the data for documentation or for reimbursement from a health insurance company, the data ought to be as accurate as current clinical tools (e.g., goniometers).
Only one of the listed companies has validated the measurement capabilities of their systems and of the Microsoft Kinect. Kurillo and colleagues evaluated their system used in 5Plus Therapy against the Impulse motion-capture system (PhaseSpace Inc., San Leandro, CA) and found that it had good accuracy of joint positions and small to large percentage errors in joint angle measurements . However, this study had a small sample size of only 10 subjects and used the first version of the Kinect sensor in its validation. Additionally, the movements used in the assessment were only within a single plane for each movement and all participants were seated during data collection.
Other researchers have validated the Kinect’s measurement and tracking capabilities for both general and specific applications. Hondori and Khademi  provide an excellent summary of the work completed prior to 2014. It should be noted that all of these studies evaluated the first version of the Kinect. Following the release of the Kinect V2 sensor, most researchers have focused their validation efforts on gait and posture applications [21–24]. The Kinect V2 has good-to-excellent tracking and measurement capabilities for gait-specific parameters and clinical outcomes. However, many of these studies tracked only the lower limbs. Furthermore, gait is a relatively consistent, rhythmic motion that is consistent across participants, even in rehabilitation populations (i.e., one foot in front of the other). The full-body movements that participants are not limited to specific planes and could choose to use either hand have not been studied in current and prior comparisons of the Microsoft Kinect and optical marker-based motion capture systems.
We have developed software called Mystic Isle that utilizes the Microsoft Kinect V2 sensor as the input device . Mystic Isle is designed as a rehabilitation game and has shown good results in improving motor function and daily activity performance in persons with chronic stroke . The software initially used the first version (V1) of the Microsoft Kinect as the input device and we completed a study that compared it to the OptiTrack optical system . Based on a visual analysis, we demonstrated that for the hand and elbow, the Kinect V1 has good accuracy in calculating trajectory of movement. For the shoulder, the Kinect V1 tracking abilities limit its validity. Although these findings are promising, the types and number of movements used in the study were limited to those in a seated position and mostly in one plane of movement (e.g., sagittal). Furthermore, the tracking capabilities of the Kinect V2 have substantially improved in the past 7 years and include more data points (joints) for comparison.
The current Mystic Isle game involves multi-planar, full body movements. Designed for individuals with diverse abilities, games can be played in a sitting or standing position, depending on the therapy treatment plan. In standing, the player is able to move around in the 3-dimensional space, akin to real-world rehabilitation. Few studies have evaluated the tracking and measurement capabilities of the Microsoft Kinect V2 for full-body, multi-planar movements in both sitting and standing. The purpose of this study was to determine the spatial accuracy and measurement validity of the Microsoft Kinect V2 sensor in a NUI rehabilitation game in comparison to a gold-standard marker-based motion capture system (Vicon™).
Materials and methods
Participants were recruited via convenience sample at the University of Missouri- Columbia campus. Participants were included if they: 1) were over the age of 18, 2) could understand conversational English, and 3) had no medical conditions which prevented them from playing video games. The study has been approved by the Health Sciences Institutional Review Board at the University of Missouri with the approval number IRB 2005896 HS. All potential participants were screened and all subjects provided written informed consent before beginning the study.
Mystic Isle is a platform for rehabilitation that allows a user to interact with a virtual environment by using their body (Fig 1). The Mystic Isle software was created in Unity 3D and Mystic Isle allows the tracked user to interact with virtual environments and objects in a 3-D world. Using Mystic Isle, specific movements, distances, and locations of objects can be tailored to the abilities and requirements of the user. The system uses the Microsoft Kinect V2 camera to track participant movements. The Kinect V2 tracks 20 discrete points/joints on the body of the user. Both gross motor (stepping, jumping, squatting) and fine motor (waving the hand, turning the palm facing up, open/close hand) movements can be tracked. The Kinect V2 tracks the user in 3-dimensional space and then inputs the data in real time to the associated software, Mystic Isle. The Kinect V2 tracks and records the x, y, and z coordinates (and confidence) of each discrete joint at either 15 or 30 frames per second.
[ARTICLE] Development of a Novel Home Based Multiscene Upper Limb Rehabilitation Training and Evaluation System for Post-stroke Patients – Full Text PDF
Upper limb rehabilitation requires long-term, repetitive rehabilitation training and assessment, and many patients cannot pay for expensive medical fees in the hospital for so long time. It is necessary to design an effective, low cost, and reasonable home rehabilitation and evaluation system. In this paper, we developed a novel home based multi-scene upper limb rehabilitation training and evaluation system (HomeRehabMaster) for post-stroke patients. Based on the Kinect sensor and posture sensor, multi-sensors fusion method was used to track the motion of the patients. Multiple virtual scenes were designed to encourage rehabilitation training of upper limbs and trunk. A rehabilitation evaluation method integrating Fugl-meyer assessment (FMA) scale and upper limb reachable workspace relative surface area (RSA) was
proposed, and a FMA-RSA assessment model was established to assess upper limb motor function.
Correlation based dynamic time warping (CBDTW) was used to solve the problem of inconsistent upper limb movement path in different patients. Two clinical trials were conducted. The experimental results show that the system is very friendly to the subjects, the rehabilitation assessment results by this system are highly correlated with the therapist’s (the highest forecast accuracy was 92.7% in the 13th item), and longterm rehabilitation training can improve the upper limb motor function of the patients statistically significant (p=0.02<0.05). The system has the potential to become an effective home rehabilitation training and evaluation system.[…]
Full Text PDF —> IEEE Xplore Full-Text PDF:
[Abstract] Towards Bilateral Upper-Limb Rehabilitation after Stroke using Kinect Game – IEEE Conference Publication
[ARTICLE] Development of a 3D, networked multi-user virtual reality environment for home therapy after stroke – Full Text
Impairment of upper extremity function is a common outcome following stroke, to the detriment of lifestyle and employment opportunities. Yet, access to treatment may be limited due to geographical and transportation constraints, especially for those living in rural areas. While stroke rates are higher in these areas, stroke survivors in these regions of the country have substantially less access to clinical therapy. Home therapy could offer an important alternative to clinical treatment, but the inherent isolation and the monotony of self-directed training can greatly reduce compliance.
We developed a 3D, networked multi-user Virtual Environment for Rehabilitative Gaming Exercises (VERGE) system for home therapy. Within this environment, stroke survivors can interact with therapists and/or fellow stroke survivors in the same virtual space even though they may be physically remote. Each user’s own movement controls an avatar through kinematic measurements made with a low-cost, Kinect™ device. The system was explicitly designed to train movements important to rehabilitation and to provide real-time feedback of performance to users and clinicians. To obtain user feedback about the system, 15 stroke survivors with chronic upper extremity hemiparesis participated in a multisession pilot evaluation study, consisting of a three-week intervention in a laboratory setting. For each week, the participant performed three one-hour training sessions with one of three modalities: 1) VERGE system, 2) an existing virtual reality environment based on Alice in Wonderland (AWVR), or 3) a home exercise program (HEP).
Over 85% of the subjects found the VERGE system to be an effective means of promoting repetitive practice of arm movement. Arm displacement averaged 350 m for each VERGE training session. Arm displacement was not significantly less when using VERGE than when using AWVR or HEP. Participants were split on preference for VERGE, AWVR or HEP. Importantly, almost all subjects indicated a willingness to perform the training for at least 2–3 days per week at home.
Multi-user VR environments hold promise for home therapy, although the importance of reducing complexity of operation for the user in the VR system must be emphasized. A modified version of the VERGE system is currently being used in a home therapy study.
Chronic upper extremity impairment is all too common among the more than 7 million stroke survivors in the U.S. . These impairments have disabling effects on all facets of life, including self-care, employment, and leisure activities. Repetitive practice of movement, such as arm movement, is thought to improve outcomes for stroke survivors [2, 3, 4], but access to the clinic for therapy is often limited by geography or lack of transportation. While almost 50 million Americans live in rural areas, 90% of physical and occupational therapists live in major urban areas . Per capita ratios of therapists to overall population are 50% larger in urban as compared to rural regions of the country . Rates of stroke in these rural areas, however, exceed those of major urban areas [7, 8, 9]. Thus, a large number of stroke survivors have limited access to skilled treatment. Data from 21 states found that only 30% of stroke survivors received outpatient rehabilitation, a much lower percentage than that recommended by clinical practice guidelines . Declines seen following discharge from inpatient rehabilitation are undoubtedly exacerbated by limited access to clinical therapy .
Disparity in quality of care has been recognized in the acute treatment of stroke for a number of years. This situation has led to the development of telemedicine to extend expert care to individuals during the initial hours and days following the stroke, advance site-independent treatment, and create models of care in rural areas [12, 13, 14]. Therapy options after this acute period, however, generally remain limited for stroke survivors in rural areas. Akin to the telemedicine efforts, telerehabilitation treatments have been proposed. However, telerehabilitation interactions are typically limited to off-line monitoring by the therapist [8, 9, 15], phone calls between a therapist and client [16, 17], or videoconferencing [18, 19, 20]. While systems allowing more direct interaction have been proposed, the hardware cost and complexity limit applicability for home-based therapy [21, 22, 23]. Hence, the therapist is relegated to the role of observer and the intimacy of a clinical therapy session is lost. Therapy options are substantially restricted, as is the available feedback.
Recently, multiple investigators have been exploring means of improving home-based therapy through the development of systems or serious games which permit multiple, simultaneous users [24, 25, 26, 27, 28, 29, 30]. These efforts have proposed the inclusion of multiple users as a means to overcome resistance to home-based therapy that may result due to isolation or lack of engagement. Indeed, studies have observed a preference for multi-user vs, single-user therapy when utilizing these systems [26, 29]. However, these systems have largely been limited to control of a one-dimensional or two-dimensional space and both users remain in the same physical location (e.g., side by side). One team of researchers did develop a framework for supporting distant users (such as a therapist in the hospital and a stroke survivor in their home), but game control was limited to one or two dimensions [31, 32].
Here, we describe the development of a fully three-dimensional (3D) virtual reality environment (VRE) for home-based therapy in which multiple, remote users can interact in real time. In this Virtual Environment for Rehabilitative Gaming Exercises (VERGE) system , movement of the user is mapped to corresponding movement of an avatar to foster a sense of presence in and engagement with the VRE. The 3D environment encompasses aspects of clinical therapy, such as transport of objects or movement of the hand into specified regions of the upper extremity workspace. Although the importance of 3D movements in VR environments is a topic of debate [34, 35], movements tested in environments with lesser degrees-of-freedom (DOF) are often very limited and dictated by a one DOF robot. These movements differ substantially from the types of movements normally seen in 3D reaching movements [4, 36]. The network architecture of the system allows users to be located remotely from each other, such as a stroke survivor in their home, a therapist in a clinic, or a stroke survivor’s friend or relative living in another city or state. The virtual nature of the environment allows even very limited movements in the physical world to have successful functional outcomes in the virtual world, thereby offering a sense of accomplishment and motivation for successive attempts. Additionally, task difficulty can easily be modified in order to maintain the proper level of challenge, which is important for motor learning in general  and rehabilitation in particular .
We developed and performed preliminary testing of the VERGE system to gauge user response in comparison to two other therapy modalities that could be used for home therapy: an existing virtual reality system based on the Alice in Wonderland story (AWVR)  and a home exercise program (HEP). Fifteen stroke survivors completed three, one-hour therapy sessions per week with each of the three therapy modalities (9 sessions total). We hypothesized that the use of the VERGE system would not decrease the amount of arm movement promoted, in comparison with the AWVR and HEP modalities. We further expected that users’ self-described engagement would be greatest for the VERGE system due to the presence of a partner.
At its core, VERGE consists of a 3D VRE in which avatars interact with virtual objects. To date, we have created two such VREs, one depicting a dining room and the other a kitchen. The scenes were created in Maya (Autodesk Inc., San Rafael, CA) and imported into Unity 3D (Unity 4.5, Unity Technologies, San Francisco, CA), the software platform controlling VERGE. The VREs are rich in detail in order to provide depth cues . Thus, depth can be conveyed without the need for stereovision, such as that provided by head mounted displays (HMDs). We have found that HMDs can be difficult for stroke survivors to use due to the limited field-of-view and, especially, involuntary coupling between neck and arm motion [41, 42]. The latter may lead to complications with moving the arm while keeping the head steady.
[Abstract] Design and Development of a Robot Guided Rehabilitation Scheme for Upper Extremity Rehabilitation
To rehabilitate individuals with impaired upper-limb function, we have designed and developed a robot guided rehabilitation scheme. A humanoid robot, NAO was used for this purpose. NAO has 25 degrees of freedom. With its sensors and actuators, it can walk forward and backward, can sit down and stand up, can wave his hand, can speak to the audience, can feel the touch sensation, and can recognize the person he is meeting. All these qualities have made NAO a perfect coach to guide the subjects to perform rehabilitation exercises. To demonstrate rehabilitation exercises with NAO, a library of recommended rehabilitation exercises involving shoulder (i.e., abduction/adduction, vertical flexion/extension, and internal/external rotation), and elbow (i.e., flexion/extension) joint movements was formed in Choregraphe (graphical programming interface). In experiments, NAO was maneuvered to instruct and demonstrate the exercises from the NRL. A complex ‘touch and play’ game was also developed where NAO plays with the subject that represents a multi-joint movement’s exercise. To develop the proposed tele-rehabilitation scheme, kinematic model of human upper-extremity was developed based modified Denavit-Hartenberg notations. A complete geometric solution was developed to find a unique inverse kinematic solution of human upper-extremity from the Kinect data. In tele-rehabilitation scheme, a therapist can remotely tele-operate the NAO in real-time to instruct and demonstrate subjects different arm movement exercises. Kinect sensor was used in this scheme to get tele-operator’s kinematics data. Experiments results reveals that NAO can be tele-operated successfully to instruct and demonstrate subjects to perform different arm movement exercises. A control algorithm was developed in MATLAB for the proposed robot guided supervised rehabilitation scheme. Experimental results show that the NAO and Kinect sensor can effectively be used to supervise and guide the subjects in performing active rehabilitation exercises for shoulder and elbow joint movements.
Assad-Uz-Zaman, Md, “Design and Development of a Robot Guided Rehabilitation Scheme for Upper Extremity Rehabilitation” (2017). Theses and Dissertations. 1578.
[Abstract + Related Articles] Adaptive gameplay and difficulty adjustment in a gamified upper-limb rehabilitation – IEEE Conference Publication
[Abstract] A low cost kinect-based virtual rehabilitation system for inpatient rehabilitation of the upper limb in patients with subacute stroke: A randomized, double-blind, sham-controlled pilot trial.
We designed this study to prove the efficacy of the low-cost Kinect-based virtual rehabilitation (VR) system for upper limb recovery among patients with subacute stroke.
A double-blind, randomized, sham-controlled trial was performed. A total of 23 subjects with subacute stroke (<3 months) were allocated to sham (n = 11) and real VR group (n = 12). Both groups participated in a daily 30-minute occupational therapy for upper limb recovery for 10 consecutive weekdays. Subjects received an additional daily 30-minute Kinect-based or sham VR. Assessment was performed before the VR, immediately and 1 month after the last session of VR. Fugl-Meyer Assessment (FMA) (primary outcome) and other secondary functional outcomes were measured. Accelerometers were used to measure hemiparetic upper limb movements during the therapy.
FMA immediately after last VR session was not different between the sham (46.8 ± 16.0) and the real VR group (49.4 ± 14.2) (P = .937 in intention to treat analysis). Significant differences of total activity counts (TAC) were found in hemiparetic upper limb during the therapy between groups (F2,26 = 4.43; P = .22). Real VR group (107,926 ± 68,874) showed significantly more TACs compared with the sham VR group (46,686 ± 25,814) but there was no statistical significance between real VR and control (64,575 ± 27,533).
Low-cost Kinect-based upper limb rehabilitation system was not more efficacious compared with sham VR. However, the compliance in VR was good and VR system induced more arm motion than control and similar activity compared with the conventional therapy, which suggests its utility as an adjuvant additional therapy during inpatient stroke rehabilitation.
- PMID:29924029 DOI:10.1097/MD.0000000000011173
[Abstract] Effects of Kinect-based virtual reality game training on upper extremity motor recovery in chronic stroke.
Therapeutic benefits of Kinect-based virtual reality (VR) game training in rehabilitation encourage its use to improve motor function.
To assess the effects of Kinect-based VR training on motor recovery of the upper extremity and functional outcomes in patients with chronic stroke.
In this randomized controlled trial, group A received 20 sessions of physical therapy (PT) + 20 sessions of Kinect-based VR training and group B received only 20 sessions of PT. Clinical outcome measures were assessed at baseline and at the end of the treatments. Primary outcome measures that assess stroke patients’ motor function included upper extremity (UE) Fugl-Meyer Assessment (FMA). Secondary outcome measures were Brunnstrom Recovery Stages (BRS), Modified Ashworth Scale (MAS), Box and Block test (BBT), Motricity index (MI), and active range of motion (AROM) measurement.
Statistically significant improvements in game scores (p < 0.05) were observed in group A. In within-group analysis, there were statistically significant improvements in all clinical outcome measures except for the BRS-hand, MAS-distal, and MAS-hand in group A; MAS-(proximal, distal, hand) and BRS-(UE, hand) in group B compared with baseline values. Differences from baseline of FMA, MI, and AROM (except adduction of shoulder and extension of elbow) were greater in group A (p < 0.05).
To conclude, our results suggest that the adjunct use of Kinect-based VR training may contribute to the improvement of UE motor function and AROM in chronic stroke patients. Further studies with a larger number of subjects with longer follow-up periods are needed to establish its effectiveness in neurorehabilitation.
[Conference Paper] Design and Implementation study of Remote Home Rehabilitation Training Operating System based on Internet – Full Text PDF
The number of patients with motor dysfunction caused by hemiplegia and stroke increased. In order to promote better recovery of their body muscles, patients are still required to perform rehabilitation exercises in the community or family after the treatment of discharge. However, there are still some difficulties in community rehabilitation for patients with motor dysfunction:
- (1) The number of therapists on-site services is scarce and expensive;
- (2) In the absence of standard and systematic action guidance, the patients ‘ own training is not only the science is not high and the effect is limited.
- (3) Patients need to be trained in special environments such as rehabilitation centers, and wearing complex training equipment is inconvenient for them.
The family rehabilitation system collects the depth information of the trainer’s movements through the Kinect skeletal tracking technique; With the help of computers and the Internet, patients can consult physicians, and the doctor through the Internet remote guidance and open training action “prescription” according to the training effect so that patients at home can be rehabilitation training. This liberated the physician’s labour force and formed a network community that was closely linked to the hospital and regularly received “training prescriptions” to improve patient rehabilitation. […]
[BOOK Chapter] The “Arm” Line of Devices for Neurological Rehabilitation: Engineering Book Chapter – Abstract
In the modern scenario of neurological rehabilitation, which requires affordable solutions oriented toward promoting home training, the Institute of Industrial Technologies and Automation (ITIA) of the Italian National Research Council (CNR) developed a line of prototypal devices for the rehabilitation of the upper limb, called “Arm.” Arm devices were conceived to promote rehabilitation at affordable prices by capturing all the main features of the state-of-the-art devices. In fact, Arm devices focus on the main features requested by a robot therapist: mechanical adaptation to the patient, ranging from passive motion to high transparency, assist-as-needed and resistive modalities; proper use of sensors for performance monitoring; easy-to-use, modular, and adaptable design. These desirable features are combined with low-cost, additive manufacturing procedures, with the purpose of meeting the requirements coming from research on neuro-motor rehabilitation and motor control and coupling them with the recent breakthrough innovations in design and manufacturing.
The Arm prototypes are:
- • LINarm: linear device, freely orientable in space, suitable for functional movements. It features a variable stiffness actuation, allowing to adapt the mechanical behavior of the device to patients’ needs. Functional Electrical Stimulation, simple Virtual Environments and a Patient Model, gathering data from integrated sensors and modulating the level of assistance, are integrated in the set-up. The LINarm++ Echord++ Project ended in October 2016 and guided the development of a second, more refined prototype, enhancing the original concept.
- • PLANarm: planar device, freely orientable in space, suitable for planar functional movements. The state of the art planar robots used in literature for motor control and motor learning research inspired PLANarm. It features a variable stiffness actuation, allowing adapting the mechanical behavior of the device depending on patients’ needs.
- • DUALarm: Low-Cost device for bimanual rehabilitation, exploiting the capability of the less affected limb to provide rehabilitation to the more affected limb. DUALarm is completely realized in 3D printing technology and aims at being an easy-to-use, low-cost, open-source project. Currently, reaching movements can be trained, but the device is conceived to be suitable for training of other functional gestures.
- • LIGHTarm: Exoskeleton for the rehabilitation of the upper-limb, designed in two versions: LIGHTarm, not actuated, and conceived to support the weight of the impaired limb. The mechanical design includes high backdrivability, focusing on shoulder rhythm and elbow singular configurations.
- • VIRTUALarm: Kinect One-based platform for motor monitoring, including body and limb tracking and a biomechanical evaluation of the performance in relation to databases of healthy subjects. Assessments include range of motion, motion dynamics, effort, motor control indexes, body segments barycenter tracking.