Publications

2024

1. 

   HuDOR: Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards

   Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirange, and Lerrel Pinto

   (Under Review) 2024

   Abs PDF Website

   Training robots directly from human videos is an emerging area in robotics and computer vision. While there has been notable progress with two-fingered grippers, learning autonomous tasks without teleoperation remains a difficult problem for multi-fingered robot hands. A key reason for this difficulty is that a policy trained on human hands may not directly transfer to a robot hand with a different morphology. In this work, we present HUDOR, a technique that enables online fine-tuning of the policy by constructing a reward function from the human video. Importantly, this reward function is built using object-oriented rewards derived from off-the-shelf point trackers, which allows for meaningful learning signals even when the robot hand is in the visual observation, while the human hand is used to construct the reward. Given a single video of human solving a task, such as gently opening a music box, HUDOR allows our four-fingered Allegro hand to learn this task with just an hour of online interaction. Our experiments across four tasks, show that HUDOR outperforms alternatives with an average of 4× improvement.

2023

1. 

   See to Touch: Learning Tactile Dexterity through Visual Incentives

   Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala, and Lerrel Pinto

   ICRA 2024 2023

   Abs HTML PDF Website

   Equipping multi-fingered robots with tactile sensing is crucial for achieving the precise, contact-rich, and dexterous manipulation that humans excel at. However, relying solely on tactile sensing fails to provide adequate cues for reasoning about objects’ spatial configurations, limiting the ability to correct errors and adapt to changing situations. In this paper, we present Tactile Adaptation from Visual Incentives (TAVI), a new framework that enhances tactile-based dexterity by optimizing dexterous policies using vision-based rewards. First, we use a contrastive-based objective to learn visual representations. Next, we construct a reward function using these visual representations through optimal-transport based matching on one human demonstration. Finally, we use online reinforcement learning on our robot to optimize tactile-based policies that maximize the visual reward. On six challenging tasks, such as peg pick-and-place, unstacking bowls, and flipping slender objects, TAVI achieves a success rate of 73% using our four-fingered Allegro robot hand. The increase in performance is 108% higher than policies using tactile and vision-based rewards and 135% higher than policies without tactile observational input.

2. 

   Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play

   Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto

   CoRL 2023 2023

   Abs HTML PDF Website

   Teaching dexterity to multi-fingered robots has been a longstanding challenge in robotics. Most prominent work in this area focuses on learning controllers or policies that either operate on visual observations or state estimates derived from vision. However, such methods perform poorly on fine-grained manipulation tasks that require reasoning about contact forces or about objects occluded by the hand itself. In this work, we present T-Dex, a new approach for tactile-based dexterity, that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self-supervised tactile encoders. This is necessary to bring high-dimensional tactile readings to a lower-dimensional embedding. In the second phase, given a handful of demonstrations for a dexterous task, we learn non-parametric policies that combine the tactile observations with visual ones. Across five challenging dexterous tasks, we show that our tactile-based dexterity models outperform purely vision and torque-based models by an average of 1.7X. Finally, we provide a detailed analysis on factors critical to T-Dex including the importance of play data, architectures, and representation learning.

2022

1. 

   Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

   Sridhar Pandian Arunachalam, Irmak Guzey, Soumith Chintala, and Lerrel Pinto

   ICRA 2023 2022

   Abs HTML PDF Website

   A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present HOLO-DEX, a framework for dexterous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of generalpurpose dexterous tasks. Given these demonstrations, we use powerful feature learning combined with non-parametric imitation to train dexterous skills. Our experiments on six common dexterous tasks, including in-hand rotation, spinning, and bottle opening, indicate that HOLO-DEX can both collect high-quality demonstration data and train skills in a matter of hours. Finally, we find that our trained skills can exhibit generalization on objects not seen in training.

2020

1. 

   Human Motion Prediction With Graph Neural Networks

   Irmak Guzey, Ahmet E. Tekden, Evren Samur, and Emre Ugur

   2020

   Abs HTML PDF

   In this work, we propose to use graph neural networks, propagation networks in particular, to investigate the problem of modelling full-body motion. The body parts and the relations between them are encoded as the nodes of a graph and edges between these nodes. How the nodes are related to each other is learned, and how the effects of multiple nodes on each node should be accumulated is computed in graph structure.