Visually Impaired People Learning Virtual Textures Through Multimodal Feedback Combining Vibrotactile and Voice

In recent years, various haptic rendering methods have been proposed to help people obtain interactive experiences with virtual textures through vibration feedback. However, due to impaired vision, the blind or visually impaired (BVI) is still unable to effectively perceive and learn virtual textures through these methods. To help BVIs have the opportunity to improve their object cognition by learning virtual textures, we built a virtual texture learning system based on multimodal feedback. We first propose an Informer based haptic texture rendering model that can fuse texture images with real-time action information to generate vibration acceleration (VA) signals. We further propose a texture classification method using the generated VA signals, and broadcast the classified texture description information to BVI through a speaker. We described the construction process of rendering model and classification method in detail, and compared the perceptual effects of subjects on textures under four rendering models through user experiments, as well as the accuracy of texture matching under two learning modes. The experimental results show that the proposed rendering model can accurately and efficiently generate VA signals, providing subjects with realistic vibration feedback. The constructed learning system enables BVI to know the type, material and other attribute information of virtual texture in the process of obtaining vibrotactile sensation. By establishing the correspondence between haptic stimuli and texture attributes, the system enables BVIs to enhance their ability to recognize objects through learning a large number of virtual textures.