AnyRotate: Gravity Invariant In-Hand Object Rotation with Sim-to-Real Touch

In-hand manipulation is an integral component of human dexterity. Our hands rely on tactile feedback for stable and reactive motions to ensure objects do not slip away unintentionally during manipulation. For a robot hand, this level of dexterity requires extracting and utilizing rich contact information for precise motor control. In this paper we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. Our policy is trained in simulation with rich tactile feedback and zero-shot transferred to the real-world. We introduce an approach for bridging the sim-to-real gap for a dense tactile representation. Our experiments highlight the benefit of detailed contact information when handling objects with varying properties. In the real world, we demonstrate a successful transfer of the dense tactile policy, generalizing to a diverse range of objects for various rotation axes and hand directions, outperforming other forms of low-dimensional touch. Interestingly, despite not having explicit slip detection, rich multi-fingered tactile sensing can implicitly detect object movement within grasp and provide a reactive behavior that improves the robustness of the policy, highlighting the importance of information-rich tactile sensing for in-hand manipulation.