Publications

Quantitative assessment of upper limb motor function aids therapists in providing appropriate rehabilitation strategies, which plays an essential role in post-stroke rehabilitation. Traditional assessments, relying on clinical scales or kinematic metrics, often involve subjective scores or are influenced by compensatory strategies. Recently, the use of muscle synergies, representing simplified neuromuscular control, has emerged as a promising approach for post-stroke assessment. In general, muscle synergies are decomposed into two components: synergy vectors and synergy activation. Synergy vectors represent the relative weighting of each muscle within each synergy, that is muscle coordination; synergy activation represents the recruitment of the muscle synergy over time, that is muscle activation strength. Both components are vital for adequately assessing patients' motor function. Therefore, we integrate the spatial domain and temporal domain features extracted from synergy vectors and synergy activation, constructing a multi-domain assessment system using a Random Forest classifier, which may provide great qualitative classification accuracy. Furthermore, a novel functional score is generated from the probabilities belonging to the pathological group. Finally, A study involving ten healthy subjects and ten post-stroke patients validates the proposed method. The experimental results show that the classification accuracy was enhanced to 98.56% by fusing the characteristics derived from different domains, which was higher than that based on spatial domain (94.90%) and temporal domain (91.08%), respectively. Furthermore, the assessment score generated by multi-domain fusion framework exhibited a significant correlation with the clinical score. These promising results show the potential of applying the proposed method to clinical assessments for post-stroke patients.

Early-exiting has recently provided an ideal solution for accelerating activity inference by attaching internal classifiers to deep neural networks. It allows easy activity samples to be predicted at shallower layers, without executing deeper layers, hence leading to notable adaptiveness in terms of accuracy-speed trade-off under varying resource demands. However, prior most works typically optimize all the classifiers equally on all types of activity data. As a result, deeper classifiers will only see hard samples during test phase, which renders the model suboptimal due to the training-test data distribution mismatch. Such issue has been rarely explored in the context of activity recognition. In this paper, to close the gap, we propose to organize all these classifiers as a dynamic-depth network and jointly optimize them in a similar gradient-boosting manner. Specifically, a gradient-rescaling is employed to bound the gradients of parameters at different depths, that makes such training procedure more stable. Particularly, we perform a prediction reweighting to emphasize current deep classifier while weakening the ensemble of its previous classifiers, so as to relieve the shortage of training data at deeper classifiers. Comprehensive experiments on multiple HAR benchmarks including UCI-HAR, PAMAP2, UniMiB-SHAR, and USC-HAD verify that it is state-of-the-art in accuracy and speed. A real implementation is measured on an ARM-based mobile device.

This paper presents the design and implementation of a networked monitoring system for home automation. The proposed system architecture organizes the deployed sensor nodes to form a multihop mesh network in the home environment. Multiple communication interfaces are designed to enable local and remote user interactions through Internet or Bluetooth. This system offers a complete, flexible and reusable solution for various home monitoring and control applications.

Abstract Robotic systems are usually controlled to repetitively perform specific actions for manufacturing tasks. The traditional control methods are domain-dependent and model-dependent with cost of much human efforts. They cannot meet the new requirements of generality and flexibility in many areas such as intelligent manufacturing and customized production. This paper develops a general model-free approach to enable robots to perform multi-step object sorting tasks through deep reinforcement learning. Taking projected heightmap images from different time steps as input without extra high-level image analysis and understanding, critic models are designed to produce a pixel-wise Q value map for each type of action. It is a new trial to apply pixel-wise Q value-based critic networks to solve multi-step sorting tasks that involve many types of actions and complex action constraints. The experimental validations on simulated and realistic object sorting tasks demonstrate the effectiveness of the proposed approach. Qualitative results (videos), code for simulated and realistic experiments, and pre-trained models are available at https://github.com/JiatongBao/DRLSorting

Human-to-robot (H2R) handovers are critical in human–robot interaction but are challenged by complex environments that impact robot perception. Traditional RGB-based perception methods exhibit severe performance degradation under harsh lighting (e.g., glare and darkness). Furthermore, H2R handovers occur in unstructured environments populated with fine-grained visual details, such as multi-angle hand configurations and novel object geometries, where conventional semantic segmentation and grasp generation approaches struggle to generalize. To overcome lighting disturbances, we present an H2R handover system with a dual-path perception pipeline. The system fuses perception data from a stereo RGB-D camera (eye-in-hand) and a time-of-flight (ToF) camera (fixed scene) under normal lighting, and switches to the ToF camera for reliable perception under glare and darkness. In parallel, to address the complex spatial and geometric features, we augment the Point Transformer v3 (PTv3) architecture by integrating a T-Net module and a self-attention mechanism to fuse the relative positional angle features between human and robot, enabling efficient real-time 3D semantic segmentation of both the object and the human hand. For grasp generation, we extend GraspNet with a grasp selection module optimized for H2R scenarios. We validate our approach through extensive experiments: (1) a semantic segmentation dataset with 7500 annotated point clouds covering 15 objects and 5 relative angles and tested on 750 point clouds from 15 unseen objects, where our method achieves 84.4% mIoU, outperforming Swin3D-L by 3.26 percentage points with 3.2× faster inference; (2) 250 real-world handover trials comparing our method with the baseline across 5 objects, 5 hand postures, and 5 angles, showing an improvement of 18.4 percentage points in success rate; (3) 450 trials under controlled adverse lighting (darkness and glare), where our dual-path perception method achieves 82.7% overall success, surpassing single-camera baselines by up to 39.4 percentage points; and (4) a comparative experiment against a state-of-the-art multimodal H2R handover method under identical adverse lighting, where our system achieves 75.0% success (15/20) versus the baseline’s 15.0% (3/20), further confirming the lighting robustness of our design. These results demonstrate the system’s robustness and generalization in challenging H2R handover scenarios.