Pulkit Agrawal

I am an Assistant Professor in the department of Electrical Engineering and Computer Science (EECS) at MIT. My lab is a part of the Computer Science and Artificial Intelligence Lab (CSAIL), is affiliated with the Laboratory for Information and Decision Systems (LIDS) and involved with NSF AI Institute for Artificial Intelligence and Fundamental Interactions ( IAIFI ).

I completed my Ph.D. at UC Berkeley; undergraduate studies from IIT Kanpur. Co-founded SafelyYou Inc. that builds fall prevention technology.

Email  /  CV  /  Biography  /  Google Scholar  /  LinkedIn

profile photo

The overarching research interest is to build machines that can automatically and continuously learn about their environment. The hope is that the end result of such learning will be similar to development of what humans call common sense. I refer to this line of work as "computational sensorimotor learning" and it encompasses computer vision, robotics, reinforcement learning, and other learning based approaches to control. Some of my past work has also touched upon principles of cognitive science, neuroscience to draw upon inspiration from these discplines. My key papers are highlighted.

Ph.D. Thesis (Computational Sensorimotor Learning)  /  Thesis Talk


Computational Sensorimotor Learning: SP'22, SP'20
Graduate Machine Learning: FA'20, FA'21
Intelligent Robot Manipulation: FA'19

Professional Education
These courses are intended for industry professionals and not MIT students.
Advanced Reinforcement Learning: Summer'22 (enrollments open)
Reinforcement Learning: Summer'22 (enrollments open)

Research Group

The lab is an unsual collection of folks working on something that is unconceivable/unthinkable, but not impossible in our lifetime: General Artificial Intelligence. Life is short, do what you must do :-) I like to call my group: Improbable AI Lab.

Post Docs
Abhishek Gupta

Graduate Students
Anurag Ajay is interested in transfer and reinforcement learning.
Lara Zlokapa (co-advised with Wojciech Matusik) is interested in design of robotic hands.
Tao Chen is interested in robotics, reinforcement learning.
Jacob Huh (co-advised with Phillip Isola) is interested in computer vision and understanding deep networks.
Andi Peng is interested in human-robot interaction.
Anthony Simeonov (co-advised with Alberto Rodriguez) is interested in robotic manipulation.
Zhang-wei Hong is interested in RL and learning commonsense.
Aviv Netanyahu is interested in computer vision, theory of mind.
Richard Li is interested in robotics.
Gabe Margolis is interested in locomotion and robotics.
Xiang Fu (co-advised with Tommi Jaakkola) is interested in RL and learning commonsense.

MEng. Students
Albert Yue is working on natural language guided navigation.
Haokuan Luo is working on navigation.
Matthew Stallone is working on scaling ML infrastructure.

Undergraduate Researchers (UROPs)
Jerry Mao is working on tool-chain for ML infrastructure.

Brian Cheung is PostDoc in Brain and Cognitive Science (BCS), interested in continual learning.
Ge Yang is an IAIFI Postdoc fellow interested in reinforcement learning.

We have openings for Ph.D. Students, PostDocs, and MIT UROPs/SuperUROPs. If you would like to apply for the Ph.D. program, please apply directly to MIT EECS admissions. For all other positions, send me an e-mail with your resume.

Recent Talks

Rethinking Robot Learning , Learning to Learn: Robotics Workshop, ICRA'21.
Self-Supervised Robot Learning, Robotics Seminar, Robot Learning Seminar, MILA.
Challenges in Real-World Reinforcement Learning, IAIFI Seminar, MIT.
The Task Specification Problem, Embodied Intelligence Seminar, MIT.

Papers Coming Soon

Stacking Objects using Contact Plane Registration, Li et al., ICRA 2022
An Integrated Design Pipeline for Tactile Sensing Robotic Manipulators, Zlokapa et al., ICRA 2022


Agile Locomotion via Model-free Learning

Gabriel Margolis*, Ge Yang*, Kartik Paigwar, Tao Chen, Pulkit Agrawal
Preprint, 2022

paper (coming soon) / project page

High-speed running and spinning on diverse terrains with a model-free controller.

sym The Low-Rank Simplicity Bias in Deep Networks
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, Phillip Isola
arXiv, 2021

paper / website / bibtex

Deeper Networks find simpler solutions! Also learn why ResNets overcome the challenges associated with very deep networks.

sym A System for General In-Hand Object Re-Orientation
Tao Chen, Jie Xu, Pulkit Agrawal
CoRL, 2021 (Best Paper Award)

paper / bibtex / project page
Press: MIT News

A framework for general in-hand object reorientation.

sym Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Anthony Simeonov*, Yilun Du*, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal**, Vincent Sitzmann**
(*equal contribution, order determined by coin flip. **equal advising)
ICRA, 2022

paper / website and code / bibtex

An SE(3) Equivariant method for specifiying and finding correspondences which enables data efficient object manipulation.

“ter” Topological Experience Replay
Zhang-Wei Hong, Tao Chen, Yen-Chen Lin, Joni Pajarinen, Pulkit Agrawal
ICLR, 2022

paper / bibtex

Sampling data from the replay buffer informed by topological structure of the state space improves performance.

“ter” Bilinear Value Networks for Multi-goal Reinforcement Learning
Zhang-Wei Hong*, Ge Yang*, Pulkit Agrawal (*equal contribution)
ICLR, 2022

paper / bibtex

Bilinear decomposition of the Q-value function improves generalization and data efficiency.

sym Equivariant Contrastive Learning
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, Marin Soljacic
ICLR , 2022

paper / bibtex

Study revealing complementarity of invariance and equivariance in contrastive learning.

sym Overcoming The Spectral Bias of Neural Value Approximation
Ge Yang*, Anurag Ajay*, Pulkit Agrawal (*equal contribution)
ICLR, 2022

paper coming soon / bibtex

Fourier features improve value estimation and consequently data efficiency.

sym Learning to Jump from Pixels
Gabriel Margolis, Tao Chen, Kartik Paigwar, Xiang Fu,
Donghyun Kim, Sangbae Kim, Pulkit Agrawal
CoRL, 2021

paper / bibtex / project page
Press: MIT News

A hierarchical control framework for dynamic vision-aware locomotion.

sym 3D Neural Scene Representations for Visuomotor Control
Yunzhu Li*, Shuang Li*, Vincent Sitzmann,Pulkit Agrawal, Antonio Torralba (*equal contribution)
CoRL, 2021 (Oral)

paper / website / bibtex

Extreme viewpoint generalization via 3D representations based on Neural Radiance Fields.

sym An End-to-End Differentiable Framework for Contact-Aware Robot Design
Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojciech Matusik,
Shinjiro Sueda, Pulkit Agrawal
RSS, 2021

paper / website / bibtex / video /
Press: MIT News

Computational method for design task-specific robotic hands.

sym Learning Task Informed Abstractions
Xiang Fu, Ge Yang, Pulkit Agrawal, Tommi Jaakkola
ICML, 2021

paper / website / bibtex

A MDP formulation that dissociates task relevant and irrelevant information.

sym Residual Model Learning for Microrobot Control
Joshua Gruenstein, Tao Chen, Neel Doshi, Pulkit Agrawal
ICRA, 2021

paper / bibtex

Data efficient learning method for controlling microrobots.

sym OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, Ofir Nachum
ICLR, 2021

paper / website / bibtex

Learning action primitives for data efficient online and offline RL.

sym A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Hogan, Joshua Tenenbaum, Pulkit Agrawal, Alberto Rodriguez
CoRL, 2020

paper / website / bibtex

A framework that achieves the best of TAMP and robot-learning for manipulating rigid objects.

sym Towards Practical Multi-object Manipulation using Relational Reinforcement Learning
Richard Li, Allan Jabri, Trevor Darrell, Pulkit Agrawal
ICRA, 2020

paper / website / code / bibtex

Combining graph neural networks with curriculum learning for solve long horizon multi-object manipulation tasks.

sym Superposition of Many Models into One
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, Bruno Olshausen,
NeurIPS, 2019

arxiv / video tutorial / code / bibtex

A method for storing multiple neural network models for different tasks into a single neural network.

sym Real-time Video Detection of Falls in Dementia Care Facility and Reduced Emergency Care
Glen L Xiong, Eleonore Bayen, Shirley Nickels, Raghav Subramaniam, Pulkit Agrawal, Julien Jacquemot, Alexandre M Bayen, Bruce Miller, George Netscher
American Journal of Managed Care , 2019

paper / SafelyYou / bibtex

Computer Vision based Fall Detection system reduces number of falls and emergency room visits in people with Dementia.

sym Zero Shot Visual Imitation
Deepak Pathak*, Parsa Mahmoudieh*, Michael Luo, Pulkit Agrawal*,
Evan Shelhamer, Alexei A. Efros, Trevor Darrell (* equal contribution)
ICLR, 2018   (Oral)

paper / website / code / slides / bibtex

Self-supervised learning of skills helps an agent imitate the task presented as a sequence of images. Forward consistency loss overcomes key challenges of inverse and forward models.

sym Investigating Human Priors for Playing Video Games
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Alexei A. Efros, Tom Griffiths
ICML, 2018

paper / website / youtube cover / media / bibtex

An empirical study of various kinds of prior information used by humans to solve video games. Such priors make them significantly more sample efficient as compared to Deep Reinforcement Learning algorithms.

sym Learning Instance Segmentation by Interaction
Deepak Pathak*, Yide Shentu*, Dian Chen*, Pulkit Agrawal*, Trevor Darrell, Sergey Levine, Jitendra Malik   (*equal contribution)
CVPR Workshop, 2018

paper / website bibtex

A self-supervised method for learning to segment objects by interacting with them.

sym Fully Automated Echocardiogram Interpretation in Clinical Practice: Feasibility and Diagnostic Accuracy
Jeffrey Zhang, Sravani Gajjala, Pulkit Agrawal, Geoffrey H Tison, Laura A Hallock, Lauren Beussink-Nelson, Mats H Lassen, Eugene Fan, Mandar A Aras, ChaRandle Jordan, Kirsten E Fleischmann, Michelle Melisko, Atif Qasim, Sanjiv J Shah, Ruzena Bajcsy, Rahul C Deo
Circulation, 2018

paper / arxiv / bibtex

Computer vision method for building fully automated and scalable analysis pipeline for echocardiogram interpretation.

sym Curiosity Driven Exploration by Self-Supervised Prediction
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell
ICML, 2017

arxiv / video / talk / code / project website / bibtex

Intrinsic curiosity of agents enables them to learn useful and generalizable skills without any rewards from the environment.

sym What Will Happen Next?: Forecasting Player Moves in Sports Videos
Panna Felsen, Pulkit Agrawal, Jitendra Malik
ICCV, 2017

paper / bibtex

Feature learning by making use of an agent's knowledge of its motion.

sym Combining Self-Supervised Learning and Imitation for Vision-based Rope Manipulation
Ashvin Nair*, Dian Chen*, Pulkit Agrawal*, Phillip Isola, Pieter Abbeel, Jitendra Malik, Sergey Levine
(*equal contribution)
ICRA, 2017

arxiv / website / video / bibtex

Self-supervised learning of low-level skills enables a robot to follow a high-level plan specified by a single video demonstration. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

sym Learning to Perform Physics Experiments via Deep Reinforcement Learning
Misha Denil, Pulkit Agrawal*, Tejas D Kulkarni, Tom Erez, Peter Battaglia, Nando de Freitas
ICLR, 2017

arxiv / media / video / bibtex

Deep reinforcement learning can equip an agent with the ability to perform experiments for inferring physical quanities of interest.

sym Reduction in Fall Rate in Dementia Managed Care through Video Incident Review: Pilot Study
Eleonore Bayen, Julien Jacquemot, George Netscher, Pulkit Agrawal, Lynn Tabb Noyce, Alexandre Bayen
Journal of Medical Internet Research, 2017

paper / bibtex

Analysis how continuous video monitoring and review of falls of individuals with dementia can support better quality of care.

sym Human Pose Estimation with Iterative Error Feedback
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik
CVPR, 2016   (Spotlight)

arxiv / code / bibtex

Iterative Error Feedback (IEF) is a self-correcting model that progressively changes an initial solution by feeding back error predictions. In contrast to feedforward CNNs that only capture structure in inputs, IEF captures structure in both the space of inputs and outputs.

sym Learning to Poke by Poking: Experiential Learning of Intuitive Physics
Pulkit Agrawal*, Ashvin Nair*, Pieter Abbeel, Jitendra Malik, Sergey Levine
(*equal contribution)
NIPS, 2016, (Oral)

arxiv / talk / project website / data / bibtex

Robot learns how to push objects to target locations by conducting a large number of pushing experiments. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

sym What makes Imagenet Good for Transfer Learning?
Jacob Huh , Pulkit Agrawal, Alexei A. Efros
NIPS LSCVS Workshop, 2016,   (Oral)

arxiv / project website / code / bibtex

An empirical investigation into various factors related to the statistics of Imagenet dataset that result in transferrable features.

sym Learning Visual Predictive Models of Physics for Playing Billiards
Katerina Fragkiadaki*, Pulkit Agrawal*, Sergey Levine, Jitendra Malik
(*equal contribution)
ICLR, 2016

arxiv / code / bibtex

This work explores how an agent can be equipped with an internal model of the dynamics of the external world, and how it can use this model to plan novel actions by running multiple internal simulations (“visual imagination”).

sym Generic 3d Representation via Pose Estimation and Matching
Amir R. Zamir, Tilman Wekel, Pulkit Agrawal, Colin Weil, Jitendra Malik, Silvio Savarese
ECCV, 2016

arxiv / website / dataset / code / bibtex

Large-scale study of feature learning using agent's knowledge of its motion. This paper extends our ICCV 2015 paper.

sym Learning to See by Moving
Pulkit Agrawal, Joao Carreira, Jitendra Malik
ICCV, 2015

arxiv / code / bibtex

Feature learning by making use of an agent's knowledge of its motion.

sym Analyzing the Performance of Multilayer Neural Networks for Object Recognition
Pulkit Agrawal, Ross Girshick, Jitendra Malik
ECCV, 2014

arxiv / bibtex

A detailed study of how to finetune neural networks and the nature of the learned representations.

sym Pixels to Voxels: Modeling Visual Representation in the Human Brain
Pulkit Agrawal, Dustin Stansbury, Jitendra Malik, Jack Gallant
(*equal contribution)
arXiv, 2014

arxiv / unpublished results / bibtex

Comparing the representations learnt by a Deep Neural Network optimized for object recognition against the human brain.

sym The Automatic Assessment of Knowledge Integration Processes in Project Teams
Gahgene Gweon, Pulkit Agrawal, Mikesh Udani, Bhiksha Raj, Carolyn Rose
Computer Supported Collaborative Learning , 2011   (Best Student Paper Award)

arxiv / bibtex

Method for identifying important parts of a group conversation directly from speech data.

System and Method for Detecting, Recording and Communicating Events in the Care and Treatment of Cognitively Impaired Persons
George Netscher, Julien Jacquemot, Pulkit Agrawal, Alexandre Bayen
US Patent: US20190287376A1, 2019
Invariant Object Representation of Images Using Spiking Neural Networks
Pulkit Agrawal, Somdeb Majumdar, Vikram Gupta
US Patent: US20150278628A1, 2015
Invariant Object Representation of Images Using Spiking Neural Networks
Pulkit Agrawal, Somdeb Majumdar
US Patent: US20150278641A1, 2015

Area Chair, ICML, 2021
Area Chair, ICLR, 2021
Area Chair, NeurIPS, 2020
Area Chair, CoRL, 2020, 2019

Lab Alumni
Eric Chen, 2021
Joshua Gruenstein, 2021 (now CEO Tutor Intelligence)
Alon Z. Kosowsky-Sachs, 2021 (now CTO Tutor Intelligence)
Avery Lamp (now at stealth startup) Sanja Simonkovj, 2021
Oran Luzon, 2021
Blake Tickell, 2020
Ishani Thakur,2020 (Undergraduate Researcher)

template / accessibility