Pulkit Agrawal

I am an Assistant Professor in the department of Electrical Engineering and Computer Science (EECS) at MIT. My lab is a part of the Computer Science and Artificial Intelligence Lab (CSAIL), is affiliated with the Laboratory for Information and Decision Systems (LIDS) and involved with NSF AI Institute for Artificial Intelligence and Fundamental Interactions ( IAIFI ).

I completed my Ph.D. at UC Berkeley; undergraduate studies from IIT Kanpur. Co-founded SafelyYou Inc. that builds fall prevention technology.

Email  /  CV  /  Biography  /  Google Scholar  /  LinkedIn

profile photo

The overarching research interest is to build machines that can automatically and continuously learn about their environment. The hope is that the end result of such learning will be similar to development of what humans call common sense. I refer to this line of work as "computational sensorimotor learning" and it encompasses computer vision, robotics, reinforcement learning, and other learning based approaches to control. Some of my past work has also touched upon principles of cognitive science, neuroscience to draw upon inspiration from these discplines. My key papers are highlighted.

Ph.D. Thesis (Computational Sensorimotor Learning)  /  Thesis Talk


Computational Sensorimotor Learning: SP'21, SP'20
Graduate Machine Learning: FA'20
Intelligent Robot Manipulation: FA'19

Professional Education
These courses are intended for industry professionals and not MIT students.
Advanced Reinforcement Learning: Summer'21 (Registrations Open)
Reinforcement Learning: Winter'20-21

Research Group

The lab is an unsual collection of folks working on something that is unconceivable/unthinkable, but not impossible in our lifetime: General Artificial Intelligence. Life is short, do what you must do :-) I like to call my group: Improbable AI Lab.

Graduate Students
Anurag Ajay is interested in transfer and reinforcement learning.
Lara Zlokapa (co-advised with Wojciech Matusik) is interested in design of robotic hands.
Tao Chen is interested in robotics, reinforcement learning.
Jacob Huh (co-advised with Phillip Isola) is interested in computer vision and understanding deep networks.
Andi Peng is interested in human-robot interaction.
Anthony Simeonov (co-advised with Alberto Rodriguez) is interested in robotic manipulation.
Xiang Fu (co-advised with Tommi Jaakkola) is interested in RL and learning commonsense.
Zhang-wei Hong is interested in RL and learning commonsense.
Aviv Netanyahu is interested in computer vision, theory of mind.
Richard Li is interested in robotics.
Gabe Margolis is interested in locomotion and robotics.

MEng. Students
Eric Chen is investigating curiosity-driven exploration.
Albert Yue is working on natural language guided navigation.
Haokuan Luo is working on navigation.
Matthew Stallone is working on scaling ML infrastructure.

Undergraduate Researchers (UROPs)
Jerry Mao is working on tool-chain for ML infrastructure.

Brian Cheung is PostDoc in Brain and Cognitive Science (BCS), interested in continual learning.
Ge Yang is an IAIFI Postdoc fellow interested in reinforcement learning.
Hyojin Bahng is a Ph.D. student interested in computer vision.
Felix Wang is interested in robotics and navigation.

We have openings for Ph.D. Students, PostDocs, and MIT UROPs/SuperUROPs. If you would like to apply for the Ph.D. program, please apply directly to MIT EECS admissions. For all other positions, send me an e-mail with your resume.

Recent Talks

Rethinking Robot Learning , Learning to Learn: Robotics Workshop, ICRA'21.
Self-Supervised Robot Learning, Robotics Seminar, Robot Learning Seminar, MILA.
Challenges in Real-World Reinforcement Learning, IAIFI Seminar, MIT.
The Task Specification Problem, Embodied Intelligence Seminar, MIT.

Papers Coming Soon

Learning Task Informed Abstractions, Fu et al., ICML' 21
An End-to-End Differentiable Framework for Contact-Aware Robot Design, Xie et al., RSS' 21
Going Beyond Images: 3D-Aware Representation Learning for Visuomotor Control, Li et al.
Visually Guided Agile Quadruped Locomotion, Margolis et al.
Analyzing the Generalization Gap in Visual Reinforcement Learning, Ajay et al.
Topological Experience Replay for Fast Q-Learning, Hong et al.
Inverse Reinforcement Learning from Suboptimal Demonstrations, Peng and Netanyahu et al.

sym Residual Model Learning for Microrobot Control
Joshua Gruenstein, Tao Chen, Neel Doshi, Pulkit Agrawal
ICRA, 2021

paper / bibtex

Data efficient learning method for controlling microrobots.

sym The Low-Rank Simplicity Bias in Deep Networks
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, Phillip Isola
arXiv, 2021

paper / website / bibtex

Deeper Networks find simpler solutions! Also learn why ResNets overcome the challenges associated with very deep networks.

sym OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, Ofir Nachum
ICLR, 2021

paper / website / bibtex

Learning action primitives for data efficient online and offline RL.

sym A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Hogan, Joshua Tenenbaum, Pulkit Agrawal, Alberto Rodriguez
CoRL, 2020

paper / website / bibtex

A framework that achieves the best of TAMP and robot-learning for manipulating rigid objects.

sym Towards Practical Multi-object Manipulation using Relational Reinforcement Learning
Richard Li, Allan Jabri, Trevor Darrell, Pulkit Agrawal
ICRA, 2020

paper / website / code / bibtex

Combining graph neural networks with curriculum learning for solve long horizon multi-object manipulation tasks.

sym Learning to Recover from Failures using Memory
Tao Chen, Pulkit Agrawal
ICML BIG workshop, 2020

website / paper / bibtex

Remembering failures aids faster learning by preventing the agent to oscillate between mistakes.

sym Superposition of Many Models into One
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, Bruno Olshausen,
NeurIPS, 2019

arxiv / video tutorial / code / bibtex

A method for storing multiple neural network models for different tasks into a single neural network.

sym Real-time Video Detection of Falls in Dementia Care Facility and Reduced Emergency Care
Glen L Xiong, Eleonore Bayen, Shirley Nickels, Raghav Subramaniam, Pulkit Agrawal, Julien Jacquemot, Alexandre M Bayen, Bruce Miller, George Netscher
American Journal of Managed Care , 2019

paper / SafelyYou / bibtex

Computer Vision based Fall Detection system reduces number of falls and emergency room visits in people with Dementia.

sym Zero Shot Visual Imitation
Deepak Pathak*, Parsa Mahmoudieh*, Michael Luo, Pulkit Agrawal*,
Evan Shelhamer, Alexei A. Efros, Trevor Darrell (* equal contribution)
ICLR, 2018   (Oral)

paper / website / code / slides / bibtex

Self-supervised learning of skills helps an agent imitate the task presented as a sequence of images. Forward consistency loss overcomes key challenges of inverse and forward models.

sym Investigating Human Priors for Playing Video Games
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Alexei A. Efros, Tom Griffiths
ICML, 2018

paper / website / youtube cover / media / bibtex

An empirical study of various kinds of prior information used by humans to solve video games. Such priors make them significantly more sample efficient as compared to Deep Reinforcement Learning algorithms.

sym Learning Instance Segmentation by Interaction
Deepak Pathak*, Yide Shentu*, Dian Chen*, Pulkit Agrawal*, Trevor Darrell, Sergey Levine, Jitendra Malik   (*equal contribution)
CVPR Workshop, 2018

paper / website bibtex

A self-supervised method for learning to segment objects by interacting with them.

sym Fully Automated Echocardiogram Interpretation in Clinical Practice: Feasibility and Diagnostic Accuracy
Jeffrey Zhang, Sravani Gajjala, Pulkit Agrawal, Geoffrey H Tison, Laura A Hallock, Lauren Beussink-Nelson, Mats H Lassen, Eugene Fan, Mandar A Aras, ChaRandle Jordan, Kirsten E Fleischmann, Michelle Melisko, Atif Qasim, Sanjiv J Shah, Ruzena Bajcsy, Rahul C Deo
Circulation, 2018

paper / arxiv / bibtex

Computer vision method for building fully automated and scalable analysis pipeline for echocardiogram interpretation.

sym Curiosity Driven Exploration by Self-Supervised Prediction
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell
ICML, 2017

arxiv / video / talk / code / project website / bibtex

Intrinsic curiosity of agents enables them to learn useful and generalizable skills without any rewards from the environment.

sym What Will Happen Next?: Forecasting Player Moves in Sports Videos
Panna Felsen, Pulkit Agrawal, Jitendra Malik
ICCV, 2017

paper / bibtex

Feature learning by making use of an agent's knowledge of its motion.

sym Combining Self-Supervised Learning and Imitation for Vision-based Rope Manipulation
Ashvin Nair*, Dian Chen*, Pulkit Agrawal*, Phillip Isola, Pieter Abbeel, Jitendra Malik, Sergey Levine
(*equal contribution)
ICRA, 2017

arxiv / website / video / bibtex

Self-supervised learning of low-level skills enables a robot to follow a high-level plan specified by a single video demonstration. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

sym Learning to Perform Physics Experiments via Deep Reinforcement Learning
Misha Denil, Pulkit Agrawal*, Tejas D Kulkarni, Tom Erez, Peter Battaglia, Nando de Freitas
ICLR, 2017

arxiv / media / video / bibtex

Deep reinforcement learning can equip an agent with the ability to perform experiments for inferring physical quanities of interest.

sym Reduction in Fall Rate in Dementia Managed Care through Video Incident Review: Pilot Study
Eleonore Bayen, Julien Jacquemot, George Netscher, Pulkit Agrawal, Lynn Tabb Noyce, Alexandre Bayen
Journal of Medical Internet Research, 2017

paper / bibtex

Analysis how continuous video monitoring and review of falls of individuals with dementia can support better quality of care.

sym Human Pose Estimation with Iterative Error Feedback
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik
CVPR, 2016   (Spotlight)

arxiv / code / bibtex

Iterative Error Feedback (IEF) is a self-correcting model that progressively changes an initial solution by feeding back error predictions. In contrast to feedforward CNNs that only capture structure in inputs, IEF captures structure in both the space of inputs and outputs.

sym Learning to Poke by Poking: Experiential Learning of Intuitive Physics
Pulkit Agrawal*, Ashvin Nair*, Pieter Abbeel, Jitendra Malik, Sergey Levine
(*equal contribution)
NIPS, 2016, (Oral)

arxiv / talk / project website / data / bibtex

Robot learns how to push objects to target locations by conducting a large number of pushing experiments. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

sym What makes Imagenet Good for Transfer Learning?
Jacob Huh , Pulkit Agrawal, Alexei A. Efros
NIPS LSCVS Workshop, 2016,   (Oral)

arxiv / project website / code / bibtex

An empirical investigation into various factors related to the statistics of Imagenet dataset that result in transferrable features.

sym Learning Visual Predictive Models of Physics for Playing Billiards
Katerina Fragkiadaki*, Pulkit Agrawal*, Sergey Levine, Jitendra Malik
(*equal contribution)
ICLR, 2016

arxiv / code / bibtex

This work explores how an agent can be equipped with an internal model of the dynamics of the external world, and how it can use this model to plan novel actions by running multiple internal simulations (“visual imagination”).

sym Generic 3d Representation via Pose Estimation and Matching
Amir R. Zamir, Tilman Wekel, Pulkit Agrawal, Colin Weil, Jitendra Malik, Silvio Savarese
ECCV, 2016

arxiv / website / dataset / code / bibtex

Large-scale study of feature learning using agent's knowledge of its motion. This paper extends our ICCV 2015 paper.

sym Learning to See by Moving
Pulkit Agrawal, Joao Carreira, Jitendra Malik
ICCV, 2015

arxiv / code / bibtex

Feature learning by making use of an agent's knowledge of its motion.

sym Analyzing the Performance of Multilayer Neural Networks for Object Recognition
Pulkit Agrawal, Ross Girshick, Jitendra Malik
ECCV, 2014

arxiv / bibtex

A detailed study of how to finetune neural networks and the nature of the learned representations.

sym Pixels to Voxels: Modeling Visual Representation in the Human Brain
Pulkit Agrawal, Dustin Stansbury, Jitendra Malik, Jack Gallant
(*equal contribution)
arXiv, 2014

arxiv / unpublished results / bibtex

Comparing the representations learnt by a Deep Neural Network optimized for object recognition against the human brain.

sym The Automatic Assessment of Knowledge Integration Processes in Project Teams
Gahgene Gweon, Pulkit Agrawal, Mikesh Udani, Bhiksha Raj, Carolyn Rose
Computer Supported Collaborative Learning , 2011   (Best Student Paper Award)

arxiv / bibtex

Method for identifying important parts of a group conversation directly from speech data.

System and Method for Detecting, Recording and Communicating Events in the Care and Treatment of Cognitively Impaired Persons
George Netscher, Julien Jacquemot, Pulkit Agrawal, Alexandre Bayen
US Patent: US20190287376A1, 2019
Invariant Object Representation of Images Using Spiking Neural Networks
Pulkit Agrawal, Somdeb Majumdar, Vikram Gupta
US Patent: US20150278628A1, 2015
Invariant Object Representation of Images Using Spiking Neural Networks
Pulkit Agrawal, Somdeb Majumdar
US Patent: US20150278641A1, 2015

Area Chair, ICML, 2021
Area Chair, ICLR, 2021
Area Chair, NeurIPS, 2020
Area Chair, CoRL, 2020, 2019

Lab Alumni
Joshua Gruenstein, 2021 (now CEO Tutor Intelligence)
Alon Z. Kosowsky-Sachs, 2021 (now CTO Tutor Intelligence)
Avery Lamp (now at stealth startup) Sanja Simonkovj, 2021
Oran Luzon, 2021
Blake Tickell, 2020
Ishani Thakur,2020 (Undergraduate Researcher)

template / accessibility