Publications

Brain-machine interface (BMI) can be used to control the robotic arm to assist paralysis people for performing activities of daily living. However, it is still a complex task for the BMI users to control the process of objects grasping and lifting with the robotic arm. It is hard to achieve high efficiency and accuracy even after extensive trainings. One important reason is lacking of sufficient feedback information for the user to perform the closed-loop control. In this study, we proposed a method of augmented reality (AR) guiding assistance to provide the enhanced visual feedback to the user for a closed-loop control with a hybrid Gaze-BMI, which combines the electroencephalography (EEG) signals based BMI and the eye tracking for an intuitive and effective control of the robotic arm. Experiments for the objects manipulation tasks while avoiding the obstacle in the workspace are designed to evaluate the performance of our method for controlling the robotic arm. According to the experimental results obtained from 8 subjects, the advantages of the proposed closed-loop system (with AR feedback) over the open-loop system (with visual inspection only) have been verified. The number of trigger commands used for controlling the robotic arm to grasp and lift the objects with AR feedback has reduced significantly and the height gaps of the gripper in the lifting process have decreased more than 50% compared to those trials with normal visual inspection only. The results reveal that the hybrid Gaze-BMI user can benefit from the information provided by the AR interface, improving the efficiency and reducing the cognitive load during the grasping and lifting processes.

We propose a novel approach for enabling comic-style conversation in mixed reality to assist face-to-face conversation on-site or remotely. Our approach brings word balloons of comic-style conversation to the real world. The word balloons can adapt to mixed reality scenes, such as the 3-D head motion of the speaker, the comic styles, and the speech. During the conversation, our approach updates the word balloons continuously in the object space and discretely in the image space, guided by a field learned from comics. Quantitative experiments and perceptual studies were conducted to evaluate and compare our approach with alternatives. The results from the user study and ablation study demonstrated that our approach turns out to be practical for assisting face-to-face conversation.

Electromyography (EMG) generated by human hand movements is usually used to decode different action types with high accuracy. However, the classifications of the gestures rarely consider the impact of force, and the estimation of the grasp force when performing natural grasping movements is so far overlooked. Decoding natural grasping movements and estimating the force generated by the associated movements can help patients to improve the accuracy of prosthesis control. This study mainly focused on two aspects: the classification of four natural grasping movements and the force estimation of these actions. For this purpose, we designed an experimental platform where subjects could perform four common natural grasping movements in daily life, including pinch, palmar, twist, and plug grasp, to complete target profiles. On the one hand, the results showed that, for natural grasping movements with different levels of force (three levels at 20, 50, and 80%), the average accuracy could reach from 91.43 to 97.33% under five classification schemes. On the other hand, the feasibility of force estimation for natural grasping movements was demonstrated. Furthermore, in the process of force estimation, we confirmed that the regression performance about plug grasp was the best, and the average R 2 could reach 0.9082. Besides, we found that the regression results were affected by the speed of force application. These findings contribute to the natural control of myoelectric prosthesis and the EMG-based rehabilitation training system, improving the user's experience and acceptance.

This paper presents a 10 cm × 5 cm × 5 cm, 52 g one-motor driven robot. One DC motor with a driving gear drives two driven gears to implement the functions of jumping and walking. Two one-way bearings mounted on the inner races of the two driven gears are used to switch between jumping and walking when the motor rotates clockwise and anticlockwise respectively. The jumping energy is obtained by compressing and releasing two torsion springs using a cylindrical cam with quick return characteristics. Two disk cams drive two forelegs with elastic joints to step forward one after another to implement the walking locomotion pattern. Two connecting rods link the forelegs and the rear legs on the left and right sides of the robot to transmit motions from forelegs to rear legs. The jumping and walking performances of the robot are tested. Experimental results show that the proposed robot can jump more than 33 cm high at a takeoff angle of 71.2° and it can walk forward at 1.43 mm/s.

Recent developments in the non-muscular human-robot interface (HRI) and shared control strategies have shown potentials for controlling the assistive robotic arm by the people with no residual movement or muscular activity in upper limbs. However, most non-muscular HRIs only produce discrete-valued commands, resulting in non-intuitive and less effective control of dexterous assistive robotic arm. Furthermore, the user commands and the robot autonomy commands usually switch in the shared control strategies of such applications. This characteristic has been found to yield reduced sense of agency and frustration for the user according to previous user studies. In this study, we firstly propose an intuitive and easy-to-learn-and-use hybrid HRI by combing the Brain-machine interface (BMI) and the gaze-tracking interface. For the proposed hybrid gaze-BMI, the continuous modulation of the movement speed via the motor intention is simultaneous to the unconstrained movement direction control with the gaze signals, in a seamless way. We then propose a shared control paradigm that always combines user input and the autonomy with the dynamical combination regulation. The proposed hybrid gaze-BMI and shared control paradigm were validated for a robotic arm reaching task performed with healthy subjects. All the users were able to employ the hybrid gaze-BMI for moving the end-effector sequentially to reach the target across the horizontal plane, while avoiding collisions with obstacles. The shared control paradigm maintained as much volitional control as possible, while providing the assistance for the most difficult parts of the task. The presented semi-autonomous robotic system yielded continuous, smooth and collision-free motion trajectories for the end-effector approaching the target. Compared to the system without assistances from robot autonomy, it significantly reduces the failures, efforts and time for the user to complete the tasks.

Using a long, flexible test cable connected with a one-wire voltage feedback circuit, a resistive tactile sensor in a shared row-column fashion exhibited flexibility in robotic operations but suffered from crosstalk caused by the connected cable due to its wire resistances and its contacted resistances. Firstly, we designed a new non-scanned driving-electrode (VF-NSDE) circuit using two wires for every row line and every column line to reduce the crosstalk caused by the connected cables in the circuit. Then, an equivalent resistance expression of the element being tested (EBT) for the two-wire VF-NSDE circuit was analytically derived. Following this, the one-wire VF-NSDE circuit and the two-wire VF-NSDE circuit were evaluated by simulation experiments. Finally, positive features of the proposed method were verified with the experiments of a two-wire VF-NSDE prototype circuit. The experiment results show that the two-wire VF-NSDE circuit can greatly reduce the crosstalk error caused by the cables in the 2-D networked resistive sensor array.

Automated sleep staging is essential to assess sleep quality and treat sleep disorders, so the issue of electroencephalography (EEG)-based sleep staging has gained extensive research interests. However, the following difficulties exist in this issue: 1) how to effectively learn the intrinsic features of salient waves from single-channel EEG signals; 2) how to learn and capture the useful information of sleep stage transition rules; 3) how to address the class imbalance problem of sleep stages. To handle these problems in sleep staging, we propose a novel method named SleepFC. This method comprises convolutional feature pyramid network (CFPN), cross-scale temporal context learning (CSTCL), and class adaptive fine-tuning loss function (CAFTLF) based classification network. CFPN learns the multi-scale features from salient waves of EEG signals. CSTCL extracts the informative multi-scale transition rules between sleep stages. CAFTLF-based classification network handles the class imbalance problem. Extensive experiments on three public benchmark datasets demonstrate the superiority of SleepFC over the state-of-the-art approaches. Particularly, SleepFC has a significant performance advantage in recognizing the N1 sleep stage, which is challenging to distinguish.