More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch

For humans, the process of grasping an object relies heavily on rich tactile\nfeedback. Most recent robotic grasping work, however, has been based only on\nvisual input, and thus cannot easily benefit from feedback after initiating\ncontact. In this paper, we investigate how a robot can learn to use tactile\ninformation to iteratively and efficiently adjust its grasp. To this end, we\npropose an end-to-end action-conditional model that learns regrasping policies\nfrom raw visuo-tactile data. This model -- a deep, multimodal convolutional\nnetwork -- predicts the outcome of a candidate grasp adjustment, and then\nexecutes a grasp by iteratively selecting the most promising actions. Our\napproach requires neither calibration of the tactile sensors, nor any\nanalytical modeling of contact forces, thus reducing the engineering effort\nrequired to obtain efficient grasping policies. We train our model with data\nfrom about 6,450 grasping trials on a two-finger gripper equipped with GelSight\nhigh-resolution tactile sensors on each finger. Across extensive experiments,\nour approach outperforms a variety of baselines at (i) estimating grasp\nadjustment outcomes, (ii) selecting efficient grasp adjustments for quick\ngrasping, and (iii) reducing the amount of force applied at the fingers, while\nmaintaining competitive performance. Finally, we study the choices made by our\nmodel and show that it has successfully acquired useful and interpretable\ngrasping behaviors.\n