Understanding and Improving Channel Attention for Human Activity Recognition by Temporal-Aware and Modality-Aware Embedding

Unlike image data, it is often hard to understand intricate sensor data for human activity, which generally contains heterogeneous sensor modalities from different body positions. The importance of every modality might also vary over time. Recent studies have witnessed the success of channel attention in boosting model performance. To maintain considerably low computational overhead, it utilizes global pooling operation to squeeze channel information, but neglects the importance of temporal-aware and modality-aware information that is very vital for activity recognition. In this paper, we propose a novel attention mechanism called TAMA (i.e., temporal-aware and modality-aware) to factorize global pooling operation into a pair of parallel activity feature embedding processes, which is able to simultaneously highlight the varying importance of temporal-aware and modality-aware information. Extensive ablation experiments verify that our TAMA attention can achieve competitive results on several standard HAR benchmarks without incurring extra computational burden. Moreover, a series of visualizing analysis is provided to show the improved interpretability by telling which temporal steps or which modalities is more determinant, which is in good line with human common intuition.