Robotic arm control using hybrid brain-machine interface and augmented reality feedback

Brain-machine interface (BMI) can be used to control robotic arm to assist paralysis people improving their quality of life. However process control of objects grasping is still a complex task for BMI users. High efficiency and accuracy is hard to achieve in objects grasping process even after extensive training. An important reason is lack of sufficient feedback information for performing the closed-loop control. In this study, we describe a method of augmented reality (AR) guiding assistance to provide extra feedback information to the user for closed-loop control. A hybrid BMI based system with AR feedback is proposed to evaluate the performance of our method in objects grasping task using robotic arm. Reaching and releasing tasks are completed by the robotic arm automatically. For the grasping task controlled by the user, AR is used to enrich the normal visual information during the grasping process to provide the BMI user augmented feedback information about the gripper status in real time. The feasibility of the proposed system both in open-loop (visual inspection) and closed-loop (AR feedback) are compared. According to our experimental results obtained from 5 subjects, the time used for controlling the robotic arm to grasp objects with AR feedback reduces more than 5s and the error rate of the gripper aperture decreases approximately 20% compared to those of grasping with normal visual inspection only. The results reveal that the BMI user can benefit from the information provided by AR interface in the grasping task.