Rotating without Seeing: Towards In-hand Dexterity through Touch

Zhao-Heng Yin
Binghao Huang
University of California San Diego
Yuzhe Qin
University of California San Diego
Qifeng Chen
Xiaolong Wang
University of California San Diego
Paper Website

Paper ID 36

Session 5. Simulation and Sim2Real

Poster Session Wednesday, July 12

Poster 4

Abstract: Tactile information plays a critical role in human dexterity. It reveals useful contact information that may not be inferred directly from vision. In fact, humans can even perform in-hand dexterous manipulation without using vision. Can we enable the same ability for the multi-finger robot hand? In this paper, we propose to perform in-hand object rotation using only touching without seeing the object. Instead of relying on precise tactile sensing in a small region, we introduce a new system design using dense binary force sensors (touch or no touch) overlaying one side of the whole robot hand (palm, finger links, fingertips). Such a design is low-cost, giving a larger coverage of the object, and minimizing the Sim2Real gap at the same time. We train an in-hand rotation policy using Reinforcement Learning on diverse objects in simulation. Relying on touch-only sensing, we can directly deploy the policy in a real robot hand and rotate novel objects that are not presented in training. Extensive ablations are performed on how tactile information help in-hand manipulation.