Publications

This paper presents the design and implementation of a new modular self-reconfigurable robot, called Transmote. The proposed robot has several novel features that make it suitable for search and rescue operations in complex environments. A single module can move independently by coordinating three joints. Multiple modules can connect with each other to form versatile assembled robotic structures with more powerful locomotion capabilities. The assembled structures can also transform between each other or fully disassemble into individual modules again. The modules, which communicate with each other through ZigBee compliant protocols, can build emergency communication and monitoring networks when spread to the areas without communication infrastructures. Preliminary experimental results demonstrate locomotion, mechanical docking, and transformation of several example configurations.

The tendon–sheath system (TSS) has emerged as a promising solution for force transmission in robot applications, owing to the lightweight, low inertia, and good flexibility. However, the performance of tension control is affected by the configuration-dependent nonlinearities, and achieving precise control of the distal force of TSS in time-varying configurations remains a challenge. To address this issue, this article proposes a comprehensive method combining feedforward and feedback control to enhance the control accuracy of TSS by estimating the bending angle in real time, even when the system configuration is unknown. A bending angle estimation approach based on a friction model is proposed, and an improved friction model is developed to simulate the nonlinear friction more accurately. Furthermore, a double-TSS transmission structure is designed to facilitate the integration of control and configuration feedback in mechanical systems. The experimental results confirm the effectiveness of the proposed method in enhancing force-tracking performance in unknown configurations.

Aerial manipulator (AM) systems are significantly more effective than both conventional manipulators and flying robots, especially in disaster rescue settings, because they can perform human-like interaction tasks in flight. However, controlling the contact force of an AM is difficult due to the coupling between its flying platform and manipulator. This paper proposes a contact force control framework to address this problem. First, the UAV/AM position response under an external force is studied, and it is theoretically shown that a closed-loop UAV/AM behaves as a spring-mass-damper system. Second, a contact force controller is designed using an inverse-dynamics method. Third, an attitude feed-forward approach is employed to improve the force tracking performance. Then, the real-time contact position is introduced into the control system to achieve interaction with an actively moving environment. Finally, an innovative AM is developed and subjected to flight experiments, validating the proposed framework. An AM’s characteristics in interaction operations are summarized, and general conclusions are drawn. This study is novel in that 1) the contact force control is implemented without relying on force sensors, 2) the whole framework is applicable for controlling a constant/variable contact force with high closed-loop performance, and 3) it can also perform reliable interaction with a dynamic and unknown environment. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This study is motivated by the contact force control problem of an aerial manipulator (AM) during interaction operations. Previous approaches have explored the feasibility of controlling contact forces to some extent, but they lack universality and assume a static environment. Ensuring sufficient safety and providing reliable solutions in real-world applications remain challenging. This paper aims to investigate the position response of a closed-loop aircraft system under external forces, without disrupting its existing steady flight. The proposed method transforms contact force control into position control, eliminating the need for a force sensor. A series of aerial experiments were conducted to validate the effectiveness and applicability of the method in various scenarios. Our ongoing work will focus on migrating such an AM system from a laboratory scenario to the real world.

Brain-machine interface (BMI) can be used to control the robotic arm to assist paralysis people for performing activities of daily living. However, it is still a complex task for the BMI users to control the process of objects grasping and lifting with the robotic arm. It is hard to achieve high efficiency and accuracy even after extensive trainings. One important reason is lacking of sufficient feedback information for the user to perform the closed-loop control. In this study, we proposed a method of augmented reality (AR) guiding assistance to provide the enhanced visual feedback to the user for a closed-loop control with a hybrid Gaze-BMI, which combines the electroencephalography (EEG) signals based BMI and the eye tracking for an intuitive and effective control of the robotic arm. Experiments for the objects manipulation tasks while avoiding the obstacle in the workspace are designed to evaluate the performance of our method for controlling the robotic arm. According to the experimental results obtained from 8 subjects, the advantages of the proposed closed-loop system (with AR feedback) over the open-loop system (with visual inspection only) have been verified. The number of trigger commands used for controlling the robotic arm to grasp and lift the objects with AR feedback has reduced significantly and the height gaps of the gripper in the lifting process have decreased more than 50% compared to those trials with normal visual inspection only. The results reveal that the hybrid Gaze-BMI user can benefit from the information provided by the AR interface, improving the efficiency and reducing the cognitive load during the grasping and lifting processes.

With the wide application of touch screen technology in daily life, more and more studies have tried to use touch screen as a carrier to convey the surface shape features of virtual objects to people. Current studies have proposed methods such as direct height mapping and haptic illusion based on lateral force to display the geometric shape of virtual surfaces on the touch screen. However, whether there are differences in the performance of these methods in displaying the shape features, and whether the lateral force-based haptic illusion can be used to display the shape features of complex surfaces, has not been fully explored. Aiming at this situation, in this article, we carry out user experiments on two regular shapes and eight natural images under displacement field and two types of lateral force fields with the help of three haptic devices previously designed. We also evaluate the effect of image smoothing on users' perception of natural image shape features. Experimental results show that the current shape display devices and methods that can simulate the lateral force field proposed by (Minsky et al., 1990) cannot achieve the display effect of the displacement field on complex shapes, and the shape display effect is the worst when only resistive lateral force feedback is employed. Nevertheless, since methods based on lateral force field, such as electrovibration, allow people to directly perceive the shape features of the virtual surface on the touch screen with their bare fingers, continuous research on this shape expression method still has a very attractive prospect.

Virtual reality technology brings a new experience to human-computer interaction, while wearable force feedback devices can enhance the immersion of users in interaction. This paper proposes a wearable fingertip force feedback device that uses a tendon drive mechanism, with the aim of simulating the stiffness characteristics of objects within virtual scenes. The device adjusts the rotation angle of the torsion spring through a DC motor, and then uses a wire to convert the torque into a feedback force at the user’s index fingertips, with an output force of up to 4 N and a force change rate of up to 10 N/s. This paper introduces the mechanical structure and design process of the force feedback device, and conducts a mechanical analysis of the device to select the appropriate components. Physical and psychological experiments are conducted to comprehensively evaluate the device’s performance in conveying object stiffness information. The results show that the device can simulate different stiffness characteristics of objects, and users can distinguish objects with different stiffness characteristics well when wearing the force feedback device and interacting with the three-dimensional virtual environments.

We propose a novel approach for enabling comic-style conversation in mixed reality to assist face-to-face conversation on-site or remotely. Our approach brings word balloons of comic-style conversation to the real world. The word balloons can adapt to mixed reality scenes, such as the 3-D head motion of the speaker, the comic styles, and the speech. During the conversation, our approach updates the word balloons continuously in the object space and discretely in the image space, guided by a field learned from comics. Quantitative experiments and perceptual studies were conducted to evaluate and compare our approach with alternatives. The results from the user study and ablation study demonstrated that our approach turns out to be practical for assisting face-to-face conversation.