Combining Self-Supervised Learning and Imitation for Vision-Based Rope\n Manipulation

Manipulation of deformable objects, such as ropes and cloth, is an important\nbut challenging problem in robotics. We present a learning-based system where a\nrobot takes as input a sequence of images of a human manipulating a rope from\nan initial to goal configuration, and outputs a sequence of actions that can\nreproduce the human demonstration, using only monocular images as input. To\nperform this task, the robot learns a pixel-level inverse dynamics model of\nrope manipulation directly from images in a self-supervised manner, using about\n60K interactions with the rope collected autonomously by the robot. The human\ndemonstration provides a high-level plan of what to do and the low-level\ninverse model is used to execute the plan. We show that by combining the high\nand low-level plans, the robot can successfully manipulate a rope into a\nvariety of target shapes using only a sequence of human-provided images for\ndirection.\n