Publications

Continuous shared control of a mobile robot with brain–computer interface and autonomous navigation for daily assistance

Although the electroencephalography (EEG) based brain–computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97% and an average total correct rate of over 80%. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.

Combining spatial clustering and tour planning for efficient full area exploration

Abstract Autonomous exploration in unknown environments has become a critical capability of mobile robots. Many methods often suffer from problems such as exploration goal selection based solely on information gain and inefficient tour optimization. Recent reinforcement learning-based methods do not consider full area coverage and the performance of transferring learned policy to new environments cannot be guaranteed. To address these issues, a dual-stage exploration method has been proposed, which combines spatial clustering of possible exploration goals and Traveling Salesman Problem (TSP) based tour planning on both local and global scales, aiming for efficient full-area exploration in highly convoluted environments. Our method involves two stages: exploration and relocation. During the exploration stage, we introduce to generate local navigation goal candidates straight from clusters of all possible local exploration goals. The local navigation goal is determined through tour planning, utilizing the TSP framework. Moreover, during the relocation stage, we suggest clustering all possible global exploration goals and applying TSP-based tour planning to efficiently direct the robot toward previously detected but yet-to-be-explored areas. The proposed method is validated in various challenging simulated and real-world environments. Experimental results demonstrate its effectiveness and efficiency. Videos and code are available at https://github.com/JiatongBao/exploration .

Optimal path planning in field based on traversability prediction for mobile robot

This paper presents a novel method on building relationship between the optimal path and the terrain traversability. some color and texture features are used for the input set to train a self learning function. The trained function is used for the traversability prediction. Considering the traveling smoothness of the field robot, the sub-regions with minimal original traversability is not the optimal path. The distance coefficient is suggested which is depending on the optimal subregion in the last searching row and the original traversability prediction is transformed to computed traversability prediction based on the distance coefficient. The pathes with different initial sub-regions is formed and the optimal path is picked up following the minimal sum of computed traversability prediction of all sub-regions in this path. And two experiments are shown and discussed to demonstrate the effectiveness and efficiency of the method mentioned in this paper.

Design of Intelligent Controller for Mobile Robot Based on Fuzzy Logic

In order to improve a mobile robot's autonomy in unknown environments, a novel intelligent controller is designed. The proposed controller is based on fuzzy logic with the aim of assisting a multi-sensor equipped mobile robot to safely navigate in an indoor environment. First, the designs of two behaviors for a robot's autonomous navigation are described, including path tracking and obstacle avoidance, which emulate human driving behaviors and reduce the complexity of the robot's navigation problems in unknown environments. Secondly, the two behaviors are combined by using a finite state machine (FSM), which ensures that the robot can safely track a predefined path in an unknown indoor environment. The inputs to this controller are the readings from the sensors. The corresponding output is the desired direction of the robot. Finally, both the simulation and experimental results verify the effectiveness of the proposed method.

Development of a Multidirectional Controlled Small-Scale Spherical MR Actuator for Haptic Applications

In robotics and haptics, actuators that move at multiple degrees of freedom (DOFs) without the intermediate transmission mechanisms and have high force/torque output with compact size are widely expected to improve the stability and transparency of interactions. For this reason, utilizing the characteristics that the rheological properties of magnetorheological (MR) fluid can be continuously and reversibly changed by an external magnetic field within a few milliseconds, a multidirectional controlled three-DOF spherical MR actuator is proposed in this paper. Through the special design of the stator part, the actuator can implement force feedback control in multiple directions. Then, based on the calculated torque model and analysis of the magnetic circuit, finite-element analysis is used to optimize the geometry and internal magnetic field distribution of the actuator. In order to achieve precise control and positioning of multi-DOF motion, a small inertial measurement unit is integrated in the upper part of the joystick. We built a prototype of this actuator and tested its performance under various control conditions. The results show that the actuator can provide force feedback with reasonable magnitude and direction to users according to the change of interaction conditions, thus overcoming the disadvantage of the existing spherical MR actuators that limit the movement of the user in all directions after being activated.

Enhanced haptic rendering with a hyper-threading architecture

High force update rate is a key factor for achieving high performance haptic rendering, which imposes a stringent real time requirement upon the execution environment of the haptic system. This requirement confines the haptic system to simplified environment for reducing the computation cost of haptic rendering algorithms. In this paper, we present a novel "hyper-threading" architecture consisting of several threads for haptic rendering. The high force update rate is achieved with relatively large computation time interval for each haptic loop. The proposed method was testified and proved to be effective with experiments on virtual wall prototype haptic system via Delta Haptic Device.

Deformable Convolutional Networks for Multimodal Human Activity Recognition Using Wearable Sensors

Recent years have witnessed significant success of convolutional neural networks (CNNs) in human activity recognition (HAR) using wearable sensors. Nevertheless, prior works have an obvious drawback. An activity sample may contain heterogeneous sensor modalities from different body parts. Moreover, the significance of each modality will change over time. Because a normal convolution filter usually samples activity data at a fixed regular grid, it is hard to capture salient features of activities along different sensor modalities or time intervals. What is the best filter form for activity recognition still remains a challenging task. In this article, to resolve this issue, we present a new deformable convolutional network for recognizing human activities from intricate sensory data. Specifically, the learned offsets and the feature amplitudes are added into standard convolution, which can be modulated to allow more free-form deformation over the sampling grid for sensory data. Comparing previous results, we achieve state-of-the-art recognition accuracies, e.g., 82.91%, 80.02%, 97.35%, and 99.21%, respectively, on several benchmark HAR datasets, including OPPORTUNITY, UNIMIB-SHAR, USC-HAD, and WISDM, hence indicating the advantage of the proposed method. The visual analysis is provided, which shows that the deformation could be conditioned on different input activity samples. The receptive field and the sampling locations can be adjusted in an adaptive manner, which leads to better interpretability for deep model behaviors. Installing PyTorch on a Raspberry Pi 3 B plus system, we evaluate actual run time of the deformable model. The results show that the deformable filter is able to still maintain almost the same inference time, which is very beneficial for activity recognition tasks. Our work can promote further research by leveraging intermodulating information to connect the deformable convolution and attention modules.

Visual Positioning of a Satellite for Efficient On-Orbit Docking

WaveHAR: Learning Wavelet Representation for Wearable Human Activity Recognition

sEMG Feature Optimization Strategy for Finger Grip Force Estimation

Intuitive Environmental Perception Assistance for Blind Amputees Using Spatial Audio Rendering

Vision and touch are essential sensory systems for human to interact with the environment. For the blind amputees, how to quickly and intuitively convey the environmental information to them is one of the key issues for recovering their daily living ability. Inspired by the auditory localization ability of human, we constructed a virtual scene almost identical to reality, and concurrently added a virtual sound source to the interactive object. Leveraging the method of spatial audio rendering (SAR), the three-dimensional motion of the virtual sound source can be vividly simulated in real-time. Finally, a myoelectric prosthetic control system was developed to assist blind amputees in their daily activities, The Fitts' law test on target localization was conducted on both SAR and voice prompt (VP) based path guidance methods, the results indicate that SAR significantly improves the information transfer rate. The results of prosthetic control test show that SAR reduces the completion time by half than the VP, while restoring the natural grasping path. With the advantage of intuitive and rich perception, the SAR demonstrated the potential applications for blind amputees to reconstruct the control and sensory loops.

3D-point-cloud registration and real-world dynamic modelling-based virtual environment building method for teleoperation

SUMMARY It is a challenging task for a human operator to manipulate a robot from a remote distance, especially in an unknown environment. Excellent teleoperation provides the human operator with a sense of telepresence, mainly including real-world vision, haptic perception, etc. This paper presents a novel virtual environment building method using the red–green–blue (RGB) colour information, the surface normal feature-based 3D-point-cloud registration method and the weighted sliding-average least-square-method-based real-world dynamic modelling for teleoperation. The experiments prove the method to be an accurate and effective means of teleoperation.

Wearable solar fluidic system

Plants’ transpiration, driven by solar energy to flow water and ions, provides an inspiring blueprint for self-powered systems that will revolutionize our interaction with environment and sustain human life. However, mimicking plant transpiration and osmosis in artificial materials remains a major challenge. Here, we present a wearable solar fluidic system that harnesses human sweat to enable self-sufficient freshwater production, energy supply, and information exchange. By designing a photothermal fabric that can generate a temperature gradient between skin and environment, sweat evaporation and ion flow can drive to provide sustained electrical power without external energy input. In outdoor wearing tests, these fluidic fabrics can produce fresh water of 24.2 liters per kilogram and deliver 8.50 volts of power, ultimately powering a Mars rover wirelessly. Our approach provides an emerging pathway toward self-sufficient, portable ecosystems capable of sustaining human life and communication, with the potential to advance wearable technology, fluid engineering, and human exploration.

A bronchoscopic navigation method based on neural radiation fields

Distributed Force Sensing on Large-Area E-Skin Inspired by Human’s Visual and Tactile Synergy

A Flexible Hall Effect-based Tactile Sensor with a Halbach Array

An Interpretable Nonlinear Decoupling and Calibration Approach to Wheel Force Transducers

The multi-dimensional force/torque decoupling and calibration is extremely crucial to increase the accuracy of the Wheel Force Transducer/Sensor (WFT). A novel interpretable nonlinear decoupling and calibration approach to WFT is presented. A physical interpretable prime-error framework is developed such that the linear prime part accounts for most force-voltage responses while the nonlinear error part accounts for the gross error deviation. The conventional least-square decoupling is improved with the delicate nonlinear error modeling using a polynomial base module and a hyperbolic activation function. The developed framework is proved to be mathematically solvable and physically feasible by a two-step calibration scheme. A two-axis WFT is tested and compared with the proposed interpretable nonlinear decoupling model (IND), the least-square-based method (LSM), and the error-based neural network model (eNN). Results demonstrate that the proposed IND provides an accurate, practical, and effective scheme for modeling and calibrating WFTs and maintains a good balance among accuracy, generalization ability, and computational efficiency for real applications.

LineSpyX: A Power Line Inspection Robot Based on Digital Radiography

Most of the current power line inspection robots use cameras and LiDARs to inspect the power line surfaces and the surrounding environment. But it is still difficult to detect the internal defects of the power lines. In this letter, the design and implementation of LineSpyX, a novel power line inspection robot based on digital radiography (DR), is introduced to solve the problem of non-destructive testing (NDT) of the overhead Aluminum Conductor Composite Core (ACCC) wires. The proposed robot has a stable wrapped mechanical structure with a moving system, a live work system, and a NDT system. The wheeled moving system enables the robot to move on the wires and cross obstacles such as vibration dampers. The NDT system consists of a portable X-ray generator and a DR detection panel. When the robot performs the inspection task, the X-ray goes up through the ACCC wire to the panel, where the X-ray images of the internal carbon fiber cores are recorded. A deep learning based defect diagnosis method combined with manual diagnosis is proposed to detect potential defects. The main functionalities of the developed robot are verified by lab experiments and field tests.