Publications

ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

Wenlong Huang, Chen Wang*, Yunzhu Li*, Ruohan Zhang, Li Fei-Fei (*Equal Contribution)

Conference on Robot Learning. November 2024

Paper Project Video Code

TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction

Yunfan Jiang, Chen Wang, Ruohan Zhang, Jiajun Wu, Li Fei-Fei

Conference on Robot Learning. November 2024

Paper Project Video Code

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C. Karen Liu

Robotics: Science and Systems. June 2024

Paper Project Video Code

BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation

Yunhao Ge*, Yihe Tang*, Jiashu Xu*, Cem Gokmen*, Chengshu Li, Wensi Ai, Benjamin Jose Martinez, Arman Aydin, Mona Anvari, Ayush K Chakravarthy, Hong-Xing Yu, Josiah Wong, Sanjana Srivastava, Sharon Lee, Shengxin Zha, Laurent Itti, Yunzhu Li, Roberto Martín-Martín, Miao Liu, Pengchuan Zhang, Ruohan Zhang, Li Fei-Fei, Jiajun Wu

Conference on Computer Vision and Pattern Recognition. June 2024

Paper Project Video Code

NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities

Ruohan Zhang*, Sharon Lee*, Minjune Hwang*, Ayano Hiranaka*, Chen Wang, Wensi Ai, Jin Jie Ryan Tan, Shreya Gupta, Yilun Hao, Gabrael Levine, Ruohan Gao, Anthony Norcia, Li Fei-Fei, Jiajun Wu

Conference on Robot Learning. November 2023

Paper Project Video Code

Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation

Yuanpei Chen*, Chen Wang*, Li Fei-Fei, and C. Karen Liu*

Conference on Robot Learning. November 2023

Paper Project Video Code

MimicPlay: Long-horizon Imitation Learning by Watching Human Play

Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu*, and Anima Anandkumar*

Conference on Robot Learning. November 2023

Paper Project Video Code

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei

Conference on Robot Learning. November 2023

Paper Project Video Code

Primitive Skill-based Robot Learning from Human Evaluative Feedback

Ayano Hiranaka*, Minjune Hwang*, Sharon Lee, Chen Wang, Li Fei-Fei, Jiajun Wu, Ruohan Zhang

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). October 2023

Paper Project Video Code

VIMA: General Robot Manipulation with Multimodal Prompts

Yunfan Jiang, Agrim Gupta, Zichen "Charles" Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, Linxi "Jim" Fan

International Conference on Machine Learning (ICML). July 2023

Paper Project Video Code

BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation

Chengshu Li*, Ruohan Zhang*, Josiah Wong*, Cem Gokmen*, Sanjana Srivastava*, Roberto Martín-Martín*, Chen Wang*, Gabrael Levine*, Wensi Ai*, Benjamin Jose Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, Li Fei-Fei

Conference on Robot Learning. December 2022

Paper Project Video Code

A Dual Representation Framework for Robot Learning with Human Guidance

Ruohan Zhang*, Dhruva Bansal*, Yilun Hao*, Ayano Hiranaka, Jialu Gao, Chen Wang, Roberto Martín-Martín, Li Fei-Fei, Jiajun Wu

Conference on Robot Learning. December 2022

Paper Project Video Code

iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks

Chengshu Li*, Fei Xia*, Roberto Martín-Martín*, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, Silvio Savarese

Conference on Robot Learning. November 2021

Paper Project

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Roberto Martín-Martín

Conference on Robot Learning. November 2021

Paper Video

Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, Chelsea Finn*

Conference on Robot Learning. November 2021

Paper

Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks

Bohan Wu, Suraj Nair, Li Fei-Fei*, Chelsea Finn*

Conference on Robot Learning. November 2021

Paper

Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration

Chen Wang, Claudia Pérez-D'Arpino, Danfei Xu, Li Fei-Fei, C. Karen Liu, Silvio Savarese

Conference on Robot Learning. November 2021

Paper Project Video

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín

Conference on Robot Learning. November 2021

Paper Project Video

BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

Sanjana Srivastava*, Chengshu Li*, Michael Lingelbach*, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, Li Fei-Fei

Conference on Robot Learning. November 2021

Paper Project

iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes

Bokui Shen* , Fei Xia* , Chengshu Li* , Roberto Martín-Martín* , Linxi Fan, Guanzhi Wang, Claudia Pérez-D'Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, Micael Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, Silvio Savarese

2021 Robotics: Science and Systems (RSS). September 2021

Paper Project

Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control

Chen Wang*, Rui Wang*, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Danfei Xu

2021 Robotics: Science and Systems (RSS). September 2021

Paper Project Video

Discovering Generalizable Skills via Automated Generation of Diverse Tasks

Kuan Fang, Yuke Zhu, Silvio Savarese, Li Fei-Fei

2021 Robotics: Science and Systems (RSS). June 2021

Paper Project

SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies

Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, Anima Anandkumar

IEEE/RSJ Int. Conference. on Intelligent Robots and Systems (IROS). June 2021

Paper Project Video

Adaptive Procedural Task Generation for Hard-Exploration Problems

Kuan Fang, Yuke Zhu, Silvio Savarese, Li Fei-Fei

IEEE International Conference on Robotics and Automation (ICRA). June 2021

Paper Project Video

Deep Affordance Foresight: Planning Through What Can Be Done in the Future

Danfei Xu, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Silvio Savarese, Li Fei-Fei

IEEE International Conference on Robotics and Automation (ICRA). June 2021

Paper Project Video

ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

Fei Xia*, Chengshu Li*, Roberto Martín-Martín, Or Litany, Alexander Toshev, Silvio Savarese (* Equal Contribution)

IEEE International Conference on Robotics and Automation (ICRA). June 2021

Paper Project Video

Multi-Layer Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search

Andrey Kurenkov, Roberto Martín-Martín, Jeff Ichnowski, Ken Goldberg, Silvio Savarese

Under Review. December 2020

Paper Project Video

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

Albert Tung*, Josiah Wong*, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese

Under Review. December 2020

Paper Project Video

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Ajay Mandlekar, Danfei Xu*, Roberto Martin-Martin*, Yuke Zhu, Li Fei-Fei, Silvio Savarese

Under Review. December 2020

Paper Project Video

Multimodal Sensor Fusion with Differentiable Filters

Michelle A. Lee*, Brent Yi*, Roberto Martín-Martín, Silvio Savarese, Jeannette Bohg

IEEE/RSJ Int. Conference. on Intelligent Robots and Systems (IROS). October 2020

Paper Project

Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter

Andrey Kurenkov*, Joseph Taglic*, Rohun Kulkarni, Marcus Dominguez-Kuhne, Animesh Garg, Roberto Martín-Martín, Silvio Savarese

IEEE/RSJ Int. Conference. on Intelligent Robots and Systems (IROS). October 2020

Paper Project Video

GTI: Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations

Ajay Mandlekar*, Danfei Xu*, Roberto Martin-Martin, Silvio Savarese, Li Fei-Fei

Robotics: Science and Systems (RSS). July 2020

Paper Project Video

Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments

Fei Xia, William B. Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev, Li Fei-Fei, Roberto Martín-Martín, Silvio Savarese

IEEE International Conference on Robotics and Automation (ICRA). May 2020

Paper Project Video

6-PACK: Category-Level 6D Pose Tracker with Anchor-Based Keypoints

Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, Yuke Zhu

IEEE International Conference on Robotics and Automation (ICRA). May 2020

Paper Project Video Code

KETO: Learning Keypoint Representations for Tool Manipulation

Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, Silvio Savarese

IEEE International Conference on Robotics and Automation (ICRA). May 2020

Paper Project

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, Dieter Fox

IEEE International Conference on Robotics and Automation (ICRA). May 2020

Paper Project Video

Motion Reasoning for Goal-Based Imitation Learning

De-An Huang, Yu-Wei Chao*, Chris Paxton*, Xinke Deng, Li Fei-Fei, Juan Carlos Niebles, Animesh Garg, Dieter Fox

IEEE International Conference on Robotics and Automation (ICRA). May 2020

Paper

Regression Planning Networks

Danfei Xu, Roberto Martín-Martín, De-An Huang, Yuke Zhu, Silvio Savarese, Li Fei-Fei

NeurIPS. December 2019

Paper Project

Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity

Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) - Best Cognitive Robotics Paper Finalist. November 2019

Paper Project Blog

Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learning

De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, Juan Carlos Niebles

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). November 2019

Paper

Variable Impedance in End-Effector Space: An Action Space for Reinforcement Learning of Contact-Rich Tasks

Roberto Martín-Martín, Michelle A. Lee, Rachel Gardner, Jeannette Bohg, Silvio Savarese, Animesh Garg

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). November 2019

Paper Project Code

AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers

Andrey Kurenkov*, Ajay Mandlekar*, Roberto Martin-Martin, Silvio Savarese, Animesh Garg

Conference on Robot Learning (CoRL). October 2019

Paper Project Blog

Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation

Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei

Conference on Robot Learning (CoRL). October 2019

Situational Fusion of Visual Representation for Visual Navigation

William B. Shen, Danfei Xu, Yuke Zhu, Leo Guibas, Li Fei-Fei, Silvio Savarese

International Conference on Computer Vision (ICCV). October 2019

Paper Project

Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision

Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, Silvio Savarese

The International Journal of Robotics Research (IJRR). August 2019

Paper Project Video

DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, Silvio Savarese

IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2019

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Michelle A. Lee*, Yuke Zhu*, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg

IEEE International Conference on Robotics and Automation (ICRA) - Best Paper Award. May 2019

Paper Project Video

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

Michael Danielczuk, Andrey Kurenkov, Ashwin Balakrishna, Matthew Matl, David Wang, Roberto Martin-Martin, Animesh Garg, Silvio Savarese, Ken Goldberg

IEEE International Conference on Robotics and Automation (ICRA). May 2019

Paper Project Video

RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation

Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, Li Fei-Fei

Conference on Robot Learning (CoRL). October 2018

Paper Project Code

SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark

Linxi Fan*, Yuke Zhu*, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costa, Silvio Savarese, Li Fei-Fei

Conference on Robot Learning (CoRL). October 2018

Paper Project Video Code

Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration

De-an Huang*, Suraj Nair*, Danfei Xu*, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, Juan Carlos Niebles

arXiv preprint. August 2018

Paper Project

Learning Task-Oriented Grasping for Tool Manipulation with Simulated Self-Supervision

Kuan Fang, Yuke Zhu, Animesh Garg, Viraj Mehta, Andrey Kurenkov, Li Fei-Fei, Silvio Savarese

Robotics: Science and Systems (RSS). June 2018

Paper Project Video

Finding “It”: Weakly-Supervised Reference-Aware Visual Grounding in Instructional Video

De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, Juan Carlos Niebles

IEEE Conference on Computer Vision & Pattern Recognition (CVPR). June 2018

Paper Video

Neural Task Programming: Learning to Generalize Across Hierarchical Tasks

Danfei Xu*, Suraj Nair*, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese

IEEE International Conference on Robotics and Automation (ICRA). May 2018

Paper Project Video Code

DeformNet: Free-Form Deformation Network for 3d Shape Reconstruction From a Single Image

Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, JunYoung Gwak, Chris Choy, Silvio Savarese

IEEE Winter Conference on Applications of Computer Vision (WACV). March 2018

Paper Project

AdaPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems

James Harrison*, Animesh Garg*, Boris Ivanovic, Yuke Zhu, Silvio Savarese, Li Fei-Fei, Marco Pavone

International Symposium on Robotics Research (ISRR). December 2017

Paper

Visual Semantic Planning using Deep Successor Representations

Yuke Zhu*, Daniel Gordon*, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, Ali Farhadi

International Conference on Computer Vision (ICCV). October 2017

Paper Video Code

Adversarially Robust Policy Learning through Active Construction of Physically-Plausible Perturbations

Ajay Mandlekar*, Yuke Zhu*, Animesh Garg*, Li Fei-Fei, Silvio Savarese

International Conference on Intelligent Robots and Systems (IROS). September 2017

Paper Project Video

Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning

Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi

International Conference on Robotics and Automation (ICRA). May 2017

Paper Project Video