Pulkit Agrawal

I am an Assistant Professor in the department of Electrical Engineering and Computer Science (EECS) at MIT. My lab is part of Computer Science and Artificial Intelligence Lab (CSAIL) and affiliated with Laboratory for Information and Decision Systems (LIDS).

I completed my Ph.D. at UC Berkeley; undergraduate studies from IIT Kanpur; co-founded SafelyYou Inc. and the AI Foundry.

Email  /  CV  /  Biography  /  Google Scholar  /  LinkedIn

profile photo
Research

The overarching research interest is to build machines that can automatically and continously learn about their environment. The hope is that the end result of such learning will be similar to what we might call as common sense. I call this line of work as computational sensorimotor learning and it encompasses computer vision, robotics, reinforcement learning, and other learning based approaches to control. Some of my past work has also touched upon cognitive science, neuroscience and draw inspiration from these discplines. My representative papers are highlighted.

Ph.D. Thesis (Computational Sensorimotor Learning)  /  Thesis Talk

Research Group

The lab is an unsual collection of folks working on something which is improbable, but not impossible in our lifetime: General Artificial Intelligence. Life is short, do what you must do :-) I like to call my group: Improbable AI Lab.

PostDocs
Brian Cheung

Ph.D. Students
Tao Chen, Anthony Simeonov (co-advised with Alberto Rodriguez), Jacob Huh (co-advised with Phillip Isola), Felix Wang

Research Assistants
Richard Li, Blake Tickell

Undergraduate Researchers
Alon Z. Kosowsky-Sachs, Eric Chen, Julia Gonik, Catherine Zeng, Joshua Gruenstein

Openings
We have openings for PostDocs, Ph.D. Students and MIT UROPS / SuperUROPs. If you are want to apply for a Ph.D., please donot send me an e-mail, apply directly to MIT EECS admissions. For other positions, send me an e-mail with your resume. Due to large volume of e-mails, I may not be able to respond all e-mails.

Publications
sym Superposition of Many Models into One
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, Bruno Olshausen,
NeurIPS , 2019

arxiv / video tutorial / code / bibtex

A method for storing multiple neural network models for different tasks into a single neural network.

sym Real-time Video Detection of Falls in Dementia Care Facility and Reduced Emergency Care
Glen L Xiong, Eleonore Bayen, Shirley Nickels, Raghav Subramaniam, Pulkit Agrawal, Julien Jacquemot, Alexandre M Bayen, Bruce Miller, George Netscher
American Journal of Managed Care , 2019

paper / SafelyYou / bibtex

Computer Vision based Fall Detection system reduces number of falls and emergency room visits in people with Dementia.

sym Zero Shot Visual Imitation
Deepak Pathak*, Parsa Mahmoudieh*, Michael Luo, Pulkit Agrawal*,
Evan Shelhamer, Alexei A. Efros, Trevor Darrell (* equal contribution)
ICLR, 2018   (Oral)

paper / website / code / slides / bibtex

Self-supervised learning of skills helps an agent imitate the task presented as a sequence of images. Forward consistency loss overcomes key challenges of inverse and forward models.

sym Investigating Human Priors for Playing Video Games
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Alexei A. Efros, Tom Griffiths
ICML, 2018

paper / website / youtube cover / media / bibtex

An empirical study of various kinds of prior information used by humans to solve video games. Such priors make them significantly more sample efficient as compared to Deep Reinforcement Learning algorithms.

sym Learning Instance Segmentation by Interaction
Deepak Pathak*, Yide Shentu*, Dian Chen*, Pulkit Agrawal*, Trevor Darrell, Sergey Levine, Jitendra Malik   (*equal contribution)
CVPR Workshop, 2018

paper / website bibtex

A self-supervised method for learning to segment objects by interacting with them.

sym Fully Automated Echocardiogram Interpretation in Clinical Practice: Feasibility and Diagnostic Accuracy
Jeffrey Zhang, Sravani Gajjala, Pulkit Agrawal, Geoffrey H Tison, Laura A Hallock, Lauren Beussink-Nelson, Mats H Lassen, Eugene Fan, Mandar A Aras, ChaRandle Jordan, Kirsten E Fleischmann, Michelle Melisko, Atif Qasim, Sanjiv J Shah, Ruzena Bajcsy, Rahul C Deo
Circulation, 2018

paper / arxiv / bibtex

Computer vision method for building fully automated and scalable analysis pipeline for echocardiogram interpretation.

sym Curiosity Driven Exploration by Self-Supervised Prediction
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell
ICML, 2017

arxiv / video / talk / code / project website / bibtex

Intrinsic curiosity of agents enables them to learn useful and generalizable skills without any rewards from the environment.

sym What Will Happen Next?: Forecasting Player Moves in Sports Videos
Panna Felsen, Pulkit Agrawal, Jitendra Malik
ICCV, 2017

paper / bibtex

Feature learning by making use of an agent's knowledge of its motion.

sym Combining Self-Supervised Learning and Imitation for Vision-based Rope Manipulation
Ashvin Nair*, Dian Chen*, Pulkit Agrawal*, Phillip Isola, Pieter Abbeel, Jitendra Malik, Sergey Levine
(*equal contribution)
ICRA, 2017

arxiv / website / video / bibtex

Self-supervised learning of low-level skills enables a robot to follow a high-level plan specified by a single video demonstration. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

sym Learning to Perform Physics Experiments via Deep Reinforcement Learning
Misha Denil, Pulkit Agrawal*, Tejas D Kulkarni, Tom Erez, Peter Battaglia, Nando de Freitas
ICLR, 2017

arxiv / media / video / bibtex

Deep reinforcement learning can equip an agent with the ability to perform experiments for inferring physical quanities of interest.

sym Reduction in Fall Rate in Dementia Managed Care through Video Incident Review: Pilot Study
Eleonore Bayen, Julien Jacquemot, George Netscher, Pulkit Agrawal, Lynn Tabb Noyce, Alexandre Bayen
Journal of Medical Internet Research, 2017

paper / bibtex

Analysis how continuous video monitoring and review of falls of individuals with dementia can support better quality of care.

sym Human Pose Estimation with Iterative Error Feedback
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik
CVPR, 2016   (Spotlight)

arxiv / code / bibtex

Iterative Error Feedback (IEF) is a self-correcting model that progressively changes an initial solution by feeding back error predictions. In contrast to feedforward CNNs that only capture structure in inputs, IEF captures structure in both the space of inputs and outputs.

sym Learning to Poke by Poking: Experiential Learning of Intuitive Physics
Pulkit Agrawal*, Ashvin Nair*, Pieter Abbeel, Jitendra Malik, Sergey Levine
(*equal contribution)
NIPS, 2016, (Oral)

arxiv / talk / project website / data / bibtex

Robot learns how to push objects to target locations by conducting a large number of pushing experiments. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

sym What makes Imagenet Good for Transfer Learning?
Jacob Huh , Pulkit Agrawal, Alexei A. Efros
NIPS LSCVS Workshop, 2016,   (Oral)

arxiv / project website / code / bibtex

An empirical investigation into various factors related to the statistics of Imagenet dataset that result in transferrable features.

sym Learning Visual Predictive Models of Physics for Playing Billiards
Katerina Fragkiadaki*, Pulkit Agrawal*, Sergey Levine, Jitendra Malik
(*equal contribution)
ICLR, 2016

arxiv / code / bibtex

This work explores how an agent can be equipped with an internal model of the dynamics of the external world, and how it can use this model to plan novel actions by running multiple internal simulations (“visual imagination”).

sym Generic 3d Representation via Pose Estimation and Matching
Amir R. Zamir, Tilman Wekel, Pulkit Agrawal, Colin Weil, Jitendra Malik, Silvio Savarese
ECCV, 2016

arxiv / website / dataset / code / bibtex

Large-scale study of feature learning using agent's knowledge of its motion. This paper extends our ICCV 2015 paper.

sym Learning to See by Moving
Pulkit Agrawal, Joao Carreira, Jitendra Malik
ICCV, 2015

arxiv / code / bibtex

Feature learning by making use of an agent's knowledge of its motion.

sym Analyzing the Performance of Multilayer Neural Networks for Object Recognition
Pulkit Agrawal, Ross Girshick, Jitendra Malik
ECCV, 2014

arxiv / bibtex

A detailed study of how to finetune neural networks and the nature of the learned representations.

sym Pixels to Voxels: Modeling Visual Representation in the Human Brain
Pulkit Agrawal, Dustin Stansbury, Jitendra Malik, Jack Gallant
(*equal contribution)
arXiv, 2014

arxiv / unpublished results / bibtex

Comparing the representations learnt by a Deep Neural Network optimized for object recognition against the human brain.

sym The Automatic Assessment of Knowledge Integration Processes in Project Teams
Gahgene Gweon, Pulkit Agrawal, Mikesh Udani, Bhiksha Raj, Carolyn Rose
Computer Supported Collaborative Learning , 2011   (Best Student Paper Award)

arxiv / bibtex

Method for identifying important parts of a group conversation directly from speech data.

Patents
System and Method for Detecting, Recording and Communicating Events in the Care and Treatment of Cognitively Impaired Persons
George Netscher, Julien Jacquemot, Pulkit Agrawal, Alexandre Bayen
US Patent: US20190287376A1, 2019
Invariant Object Representation of Images Using Spiking Neural Networks
Pulkit Agrawal, Somdeb Majumdar, Vikram Gupta
US Patent: US20150278628A1, 2015
Invariant Object Representation of Images Using Spiking Neural Networks
Pulkit Agrawal, Somdeb Majumdar
US Patent: US20150278641A1, 2015
Service
Area Chair, CoRL 2020
Reviewer for CVPR, ICCV, ECCV, NeurIPS, ICML, ICLR, ICRA, IJRR, IJCV, IEEE RA-L, TPAMI etc.

template