Publications

Iterative GDHP-based approximate optimal tracking control for a class of discrete-time nonlinear systems

An Instrumented Wheel-On-Limb System of Planetary Rovers for Wheel-Terrain Interactions: System Conception and Preliminary Design

Understanding the wheel-terrain interaction is of great importance to improve the maneuverability and traversability of the rovers. A well-developed sensing device carried by the rover would greatly facilitate the complex risk-reducing operations on sandy terrains. In this paper, an instrumented wheel-on-limb (WOL) system of planetary rovers for wheel-terrain interaction characterization is presented. Assuming the function of a passive suspension of the wheel, the WOL system allows itself to follow the terrain contour, and keep the wheel remain lowered onto the ground during rover motion including climbing and descending, as well as deploy and place the wheel on the ground before a drive commanding. The system concept, functional requirements, and pre-design work, as well as the system integration, are presented.

HANDSON Hand: Strategies and Approaches for Competitive Success at CYBATHLON 2024

A significant number of people with disabilities rely on assistive devices, yet these technologies often face limitations, including restricted functionality, inadequate user-centered design, and a lack of standardized evaluation metrics. While upper-limb prosthetics remain a key research focus, existing commercial solutions still fall short of meeting daily reliability and usability needs, leading to high abandonment rates. CYBATHLON integrates assistive technologies into daily living tasks, driving innovation and prioritizing user needs. In CYBATHLON 2024, the HANDSON hand secured first place in the arm prosthesis race, showcasing breakthroughs in human–robot integration. This paper presents the HANDSON hand’s design, core technologies, training strategies, and competition performance, offering insights for advancing multifunctional prosthetic hands to tackle real-world challenges.

Point Cloud Perception in Human-to-Robot Handover based on Semantic Segmentation and Shape Completion

Effective object handovers from humans-to-robots (H2R) in multi-scenario environments are a key challenge in human-robot collaboration. Numerous studies have explored various perception strategies. However, robust handover in complex settings remains challenging due to frequent lighting interference to RGB-based semantic segmentation, or inherent blindspot issue for limited perceptual views. Under these difficulties, this paper proposes a new perception framework for human-robot handover under eye-on-scene setting. Firstly, this framework uses a single-pixel RGB tracking for initial localization, a point cloud segmentation with regulated extraction, and a learning-based completion for camera blind spots. It works reliably encountering RGB overexposure and shows clear improvement in human handover postures at blindspot. Then, the method introduces a KNN-based loss to improve RandLA-Net segmentation, and an object feature classification strategy to enhance ODGNet-based completion. These boost recognition efficiency and generalization in real handover tasks. Finally, in handover experiments on 15 novel objects, our system achieved an overall success rate of 88.5%. In the quantitative evaluation involving four objects and five representative human handover postures, our method demonstrates improvements over the No Completion baseline in terms of both the overall success rate and the number of predicted valid grasp poses. When encountering RGB overexposure, our method exhibits a more significant advantage in perceptual robustness and grasping effectiveness compared to traditional RGB-based segmentation methods.

Design 2D nonlinear system for information storage

A Walking Assistant Robotic System for the Visually Impaired Based on Computer Vision and Tactile Perception

Dual-Modal Magnetic Skin for Robust Tactile Sensing

Decoding Different Reach-and-Grasp Movements Using Noninvasive Electroencephalogram

Grasping is one of the most indispensable functions of humans. Decoding reach-and-grasp actions from electroencephalograms (EEGs) is of great significance for the realization of intuitive and natural neuroprosthesis control, and the recovery or reconstruction of hand functions of patients with motor disorders. In this paper, we investigated decoding five different reach-and-grasp movements closely related to daily life using movement-related cortical potentials (MRCPs). In the experiment, nine healthy subjects were asked to naturally execute five different reach-and-grasp movements on the designed experimental platform, namely palmar, pinch, push, twist, and plug grasp. A total of 480 trials per subject (80 trials per condition) were recorded. The MRCPs amplitude from low-frequency (0.3-3 Hz) EEG signals were used as decoding features for further offline analysis. Average binary classification accuracy for grasping vs. the no-movement condition peaked at 75.06 ± 6.8%. Peak average accuracy for grasping vs. grasping conditions of 64.95 ± 7.4% could be reached. Grand average peak accuracy of multiclassification for five grasping conditions reached 36.7 ± 6.8% at 1.45 s after the movement onset. The analysis of MRCPs indicated that all the grasping conditions are more pronounced than the no-movement condition, and there are also significant differences between the grasping conditions. These findings clearly proved the feasibility of decoding multiple reach-and-grasp actions from noninvasive EEG signals. This work is significant for the natural and intuitive BCI application, particularly for neuroprosthesis control or developing an active human-machine interaction system, such as rehabilitation robot.

Single-trial Classification of Fixation-related Potentials in Guided Visual Search Tasks using A Riemannian Network

Brain responses to visual stimulus can provide information about cognitive process or intentions. Several studies show that it is feasible to use stimulus-dependent modulation of the evoked brain responses after gaze movements (i.e., Fixation Related Potential, FRP) to predict the interested object of human. However, the performance of the state-of the-art shallow models for FRP classification is still far from satisfactory. Recent years, Riemannian geometry based on deep learning has gained its popularity in many image and video processing tasks, thanks to their ability to learn appropriate statistical representations while respecting Riemannian geometry of the data in such fields. In this paper, we have investigated a Riemannian network for classifying FRP in guided visual search task. Experiment results showed that the Riemannian network improved classification performance significantly in comparison to the shallow methods.

Visualizing fuzzy sets using opacity-varying freeform diagrams

As an extension of classical sets, fuzzy sets allow their containing elements to have continuous membership, which broadens its applications in various domains. In order to visualize the uncertain owner–member relationship, we design an intuitive diagram to visualize fuzzy sets. The diagram has a freeform boundary, which allows a clear layout of the data. In addition, the opacity of the diagram reveals the uncertainty of the membership. Physical simulation and bi-harmonic interpolation are introduced to generate the proposed diagram. We test our method with different fuzzy sets. The results show that our method is efficient to generate intuitive visualization of fuzzy sets.

Design and Impedance Control of a Traveling Wave Ultrasonic Motor with a Spherical Rotor

A traveling wave ultrasonic motor (TWUM) with a spherical rotor was designed and fabricated for a haptic interface. The mechanical characteristics testbed was set up to test the performance of the motor under the modulation of frequency and phase difference, as well as for conducting haptic interaction, and the phase difference modulation based on certain frequency was adopted as the driving mode for the proposed motor. Due to the complexity and uncertainty of the contact mechanics of the stator and rotor, the direct torque control effect of the TWUM is not ideal for the force feedback. In this paper, the impedance control method was adopted to realize the force interaction based on positional proportional–integral–derivative (PID) control. The virtual linear spring and virtual wall experiments are conducted to verify the effect of the proposed method. Results show that the position-based impedance control method has a better effect than direct torque control for the haptic interaction based on an ultrasonic motor (USM).

Robot-Assisted Retraction for Transoral Surgery

Tissue retraction is an important task in head and neck surgery to leave space for surgical operations. Because the contact between the retractor and soft tissue is not trivial to model, the retraction operation has not been well addressed by robots in modern robot-assisted surgery. We propose a human-robot collaboration approach to assist the retraction for transoral surgery. The surgeons only need to roughly place the retractors into the oral cavity and specify the recommended retraction force. Robot manipulators will automatically retract the tissues in a safe way. In order to keep the touching force safe, we employ a force-sensing system at the distal end of the retractor. By analyzing the real-time force sensor data, we propose a control strategy which combines active retraction angle compensation and passive torque compensation to adjust the retractor, further reducing potential slippage during the retraction. The proposed method ensures the retraction is adaptive to unknown perturbations of the human anatomy. Our system requires no extra recognition or calibration of the surgical scene or the human tissue models. The approach is validated with two robot manipulators and force-sensing retractors on a physical head phantom as well as a cadaveric experiment. We show that the robots stably retract the mouth under various configurations, where the relative error of the stable retraction force does not exceed 11.8%.

Electroencephalogram Source Imaging and Brain Network Based Natural Grasps Decoding

Studying the decoding process of complex grasping movement is of great significance to the field of motor rehabilitation. This study aims to decode five natural reach-and-grasp types using sources of movement-related cortical potential (MRCP) and investigate their difference in cortical signal characteristics and network structures. Electroencephalogram signals were gathered from 40 channels of eight healthy subjects. In an audio cue-based experiment, subjects were instructed to keep no-movement condition or perform five natural reach-and-grasp movements: palmar, pinch, push, twist and plug. We projected MRCP into source space and used average source amplitudes in 24 regions of interest as classification features. Besides, functional connectivity was calculated using phase locking value. Six-class classification results showed that a similar grand average peak performance of 49.35% can be achieved using source features, with only two-thirds of the number of channel features. Besides, source imaging maps and brain networks presented different patterns between each condition. Grasping pattern analysis indicated that the modules in the execution stage focus more on internal communication than in the planning stage. The former stage was related to the parietal lobe, whereas the latter was associated with the frontal lobe. This study demonstrates the superiority and effectiveness of source imaging technology and reveals the spread mechanism and network structure of five natural reach-and-grasp movements. We believe that our work will contribute to the understanding of the generation mechanism of grasping movement and promote a natural and intuitive control of brain-computer interface.

Bio-inspired vision mimetics toward next-generation collision-avoidance automation

The current "deep learning + large-scale data + strong supervised labeling" technology framework of collision avoidance for ground robots and aerial drones is becoming saturated. Its development gradually faces challenges from real open-scene applications, including small data, weak annotation, and cross-scene. Inspired by the neural structure and processes underlying human cognition (eg, human visual, auditory, and tactile systems) and the knowledge learned from daily driving tasks, a high-level cognitive system is developed for integrating collision sensing and collision avoidance. This bio-inspired cognitive approach has the advantages of good robustness, high self-adaptability, and low computation consumption in practical driving scenes. Vision is the primary sense for motion perception. It provides key information for navigation, course control, gaze stabilization, and other motion-related activities.1Fu Q. Hu C. Peng J. et al.A robust collision perception visual neural network with specific selectivity to darker objects.IEEE Trans. Cybern. 2020; 50: 5074-5088Crossref PubMed Scopus (18) Google Scholar Biological visual neural networks have evolved over millions of years and are working efficiently in nature. These visual neural networks can be ideal models for designing artificial vision systems, especially for collision avoidance. As a visual neural structure, the lobula giant movement detector (LGMD) neuron in locusts shows a strong ability to detect moving objects.2Yue S. Rind F.C. Collision detection in complex dynamic scenes using an LGMD-based visual neural network with feature enhancement.IEEE Trans. Neural Network. 2006; 17: 705-716Crossref PubMed Scopus (98) Google Scholar This LGMD neuron consists of a four-layer neural network in which the first layer detects the changes in luminance, the second and third layers process the interaction between excitation and inhibition responses, and the last layer outputs a spike (peak response). When the locust eyes perceive an approaching object, the LGMD neuron will be stimulated to output a spike signal to warn the locust of potential danger at distance. The high sensitivity of LGMD neurons in perceiving objects in motion3Xu J. Park S.H. Zhang X. A temporally irreversible visual attention model based on motion sensitive neurons.IEEE Trans. Ind. Inf. 2020; 16: 855-865Crossref Scopus (10) Google Scholar is suitable to promote vehicles to detect the approaching object. To take advantage of bio-inspired vision in collision detection, in this commentary, an innovative visual neural network that possesses similar collision selectivity of LGMD neurons is reported. The modeling of LGMD is shown in Figure 1A. The visual stimuli are perceived from the photoreceptor (P) layer to the excitation (E) and inhibition (I) layers, then the visual signals are decomposed into ON/OFF channels and fused separately in the summation (S) layer, and the grouping (G) layer is further aggregated for the output. The visual signals input to ON channels indicate the increased brightness in its navigation, and vice versa. When an obstacle approaches, visual stimulation will activate the LGMD neuron to detect potential collisions. The locust will then update its behavior to avoid collisions. This commentary discusses the impact of two human factors on driving: central fixation bias and binocular vision. So far, only a few studies have been conducted thoroughly on modeling human factors in driving and investigating its potential in ground robotics and aerial drones. Central fixation bias, which is a key factor in driving tasks as evidenced in Xu et al.,4Xu J. Guo K. Sun P.Z. Driving performance under violations of traffic rules: novice vs. experienced drivers.IEEE Trans. Intell. Veh. 2022; 7: 908-917Crossref Scopus (8) Google Scholar particularly for experienced drivers, is not thoroughly reflected in previous research. It is reported that human vision exerts a tendency of central fixation bias to actively search the gist of approaching scenes and bind visual features for higher cognitive understanding; that is, the visual information from the central visual field/region is paid more attention while that from outer regions is paid less attention.3Xu J. Park S.H. Zhang X. A temporally irreversible visual attention model based on motion sensitive neurons.IEEE Trans. Ind. Inf. 2020; 16: 855-865Crossref Scopus (10) Google Scholar Once potential hazards approach, the drivers' gaze or attention distribution exhibits high selectivity at the central visual area to enhance their alertness. To reflect this selectivity of gaze distribution in the obstacle detection process, our reported LGMD considers the effect of central fixation bias in receiving visual stimuli and transmitting the processed signals to the P layer. Specifically, the Gaussian filter shown in Figure 1B is incorporated into LGMD to keep the central region of the image as clear as in the original image while blurring the outer regions of the image. This in turn sharpens the sensitivity of LGMD to forward motion. Another human factor in driving, binocular vision, should also be fully considered. Compared with monocular vision, binocular vision could fuse depth measurement in navigation, which is conducive to accurate perception of obstacle distance. Different from most of the existing LGMD models1Fu Q. Hu C. Peng J. et al.A robust collision perception visual neural network with specific selectivity to darker objects.IEEE Trans. Cybern. 2020; 50: 5074-5088Crossref PubMed Scopus (18) Google Scholar,5Lei F. Peng Z. Liu M. et al.A robust visual system for looming cue detection against translating motion.IEEE Transact. Neural Networks Learn. Syst. 2022; Google Scholar that only verified the collision detection performance under monocular vision, our commentary provides the collision detection performance of LGMDs under binocular vision. Based on the above discussion on LGMD and the two human factors in driving, a collision avoidance strategy is reported, which is inspired by the visual-based obstacle avoidance mechanism of locusts and considers two human factors of central fixation bias and binocular vision. In this strategy, central fixation bias is modeled and placed in front of the P layer of LGMD. Two LGMDs, left and right LGMDs, are adapted to process the images of the left and right cameras in front of the vehicle, respectively. During the motion of the object, once the collision is detected by either the left or right LGMD, two procedures of direction selection and collision body distance prediction are triggered simultaneously. In the direction selection procedure, the values of normalized membrane potentials (NMPs) output by the left and right LGMD models are compared. The side with a larger NMP value is then selected. In the collision body distance prediction program, NMP and feedforward inhibition of the left and right LGMD models and the current robot motion speed are fed into the trained multi-layer perceptron model. Then, the model will output the predicted distance. Based on the direction selection and distance prediction, the best motor control is finally derived for optimizing the steering selection of collision avoidance under the emergent situation. LGMD is a promising bio-inspired model for collision avoidance. In addition to its superior performance, this kind of bio-inspired vision mimetics approach can reduce the dependence of algorithms on graphics processing units and central processing units, which becomes its unique dominance for next-generation automation. The next step of the bio-inspired vision metrics approach should focus on the processing of road scenes with complex backgrounds. On real roads, different weather and lighting conditions will lead to large variance in visual appearance of driving scenes, which is a difficult challenge that should be further studied. Besides, human factors, particularly in the attention process, are crucial for safe driving. Like human driving, after a quick scan through the entire visual field, LGMD tends to select a target area (ie, focus of attention) and then invests more attention resources to extract detailed information about the target and suppress irrelevant surrounding information, which is the primary foundation of situation awareness. This allows us to use limited attention resources to quickly screen out high-value targets from a large amount of information. As a survival mechanism evolved in human evolution, the visual attention mechanism considerably improves the efficiency of visual information processing. This key block would further enhance the performance of ground robots and aerial drones guided by human factors in driving. This work was supported in part by the EU's Seventh Framework Program (EU FP7) International Research Staff Exchange Scheme (IRSES) Project–Towards zero road Accidents—Nature Inspired Hazard Perception under grant 318907 and in part by the Zhejiang Provincial Natural Science Foundation under grant LY23F030001. The authors declare no competing interests.

A Fast and Accurate Vascular Tissue Simulation Model Based on Point Primitive Method

Virtual surgery simulation is indispensable for virtual vascular interventional training system, which provides the doctor with visual scene between catheter and vascular. Soft tissue deformation, as the most significant part, determines the success or failure of the virtual surgery simulation. However, most soft tissue deformation model cannot simultaneously meet the requirement of high deformation accuracy and real-time interaction. To solve the challenge mentioned above, this paper proposes a fast and accurate vascular tissue simulation model based on point primitive method. Firstly, the proposed model simulates a deformation of the internal structure of the vascular tissue by adopting a point primitive method. Besides, the stretching constraint and elastic potential energy constraint are introduced to control and correct node motion. Furthermore, a mapping function from the interior to the surface of the vascular tissue is constructed based on moving least squares algorithm to render the visual effect of deformation. Finally, a training system based on the proposed model is set up on the PHANTOM OMNI force-tactile feedback device to realize the deformation simulation of the virtual vascular tissue. Experimental results shows that the proposed model can enhance real-time performance of the training system under the premise of ensuring deformation accuracy, as well as simulate the elasticity of soft tissue.

Learning unified brain region representation for cross-dataset EEG-based emotion recognition

Decoding Hand Movement Types and Kinematic Information From Electroencephalogram

Brain-computer interfaces (BCIs) have achieved successful control of assistive devices, e.g. neuroprosthesis or robotic arm. Previous research based on hand movements Electroencephalogram (EEG) has shown limited success in precise and natural control. In this study, we explored the possibilities of decoding movement types and kinematic information for three reach-and-execute actions using movement-related cortical potentials (MRCPs). EEG signals were acquired from 12 healthy subjects during the execution of pinch, palmar and precision disk rotation actions that involved two levels of speeds and forces. In the case of discrimination between hand movement types under each of four different kinematics conditions, we obtained the average peak accuracies of 83.44% and 73.83% for the binary and 3-class classification, respectively. In the case of discrimination between different movement kinematics for each of three actions, the average peak accuracies of 82.9% and 58.2% could be achieved for the two and 4-class scenario. In both cases, peak decoding performance was significantly higher than the subject-specific chance level. We found that hand movement types all could be classified when these actions were executed at four different kinematic parameters. Meanwhile, for each of three hand movements, we decoded movement parameters successfully. Furthermore, the feasibility of decoding hand movements during hand retraction process was also demonstrated. These findings are of great importance for controlling neuroprosthesis or other rehabilitation devices in a fine and natural way, which would drastically increase the acceptance of motor impaired users.

Deep dictionary learning with reconstruction for texture recognition