Pulkit Agrawal

I am an Associate Professor in the department of Electrical Engineering and Computer Science (EECS) at MIT. My lab is a part of the Computer Science and Artificial Intelligence Lab (CSAIL), is affiliated with the Laboratory for Information and Decision Systems (LIDS) and involved with NSF AI Institute for Artificial Intelligence and Fundamental Interactions ( IAIFI ).

I completed my Ph.D. at UC Berkeley; undergraduate studies from IIT Kanpur. Co-founded SafelyYou Inc. that builds fall prevention technology. Advisor to Tutor Intelligence, Lab0 Inc., and Common Sense Machines.

Follow @pulkitology  /  LinkedIn  /  Email  /  CV  /  Biography  /  Google Scholar

The overarching research interest is to build machines that have similar manipulation and locomotion abilities as humans. These machines will automatically and continuously learn about their environment and exhibit both common sense and physical intuition. I refer to this line of work as "computational sensorimotor learning". It encompasses problems in peception, control, hardware design, robotics, reinforcement learning, and other learning approaches to control. My past work has also drawn inspiration from cognitive science, and neuroscience.

Ph.D. Thesis (Computational Sensorimotor Learning)  /  Thesis Talk  /  Bibtex

TEDxMIT Talk: Why machines can play chess but can't open doors? (i.e., why is robotics hard?)

Courses
Computational Sensorimotor Learning
Graduate Machine Learning: FA'20, FA'21
Intelligent Robot Manipulation: FA'19

Professional Education
These courses are intended for industry professionals and not MIT students.
Advanced Reinforcement Learning: Summer'24
Reinforcement Learning: Summer'24
AI in Robotics: Summer'24

Pulkit recieves 2024 IEEE Early Academic Career Award in Robotics and Automation .
Meenal Parakh wins the 2024 Charles and Jennifer Johnson MEng Thesis Award.
Idan Shenfeld and Zhang-Wei Hong win the 2024 Qualcomm Innvoation Fellowship.
Srinath Mahankali wins the 2024 Jeremy Gerstle UROP Award for undergraduate research.
Srinath Mahankali wins the 2024 Barry Goldwater Scholarship.
Gabe Margolis wins the 2022 Ernst A. Guillemin Thesis Award in Artificial Intelligence and Decision Making.
Best Paper Award at Conference on Robot Learning (CoRL) 2021 to our work on in-hand object re-orientation.

The lab is an unsual collection of folks working on something that is unconceivable/unthinkable, but not impossible in our lifetime: General Artificial Intelligence. Life is short, do what you must do :-) I like to call my group: Improbable AI Lab.

Post Docs
Haoshu Fang
Branden Romero

Graduate Students
Antonia Bronars
Gabe Margolis
Zhang-wei Hong
Nolan Fey
Younghyo Park
Jyothish Pari
Idan Shenfeld
Aviv Netanyahu
Bipasha Sen
Richard Li
Seungwook Han

Masters of Engineering (MEng. Students) and Undergraduate Researchers (UROPs)
Srinath Mahankali, Jagdeep Bhatia, Arthur Hu, Gregory Pylypovych, Kevin Garcia, Yash Prabhu, Locke Cai.

Visiting Researchers
Sandor Felber, Lars Ankile

Openings
We have openings for Ph.D. Students, PostDocs, and MIT UROPs/SuperUROPs. If you would like to apply for the Ph.D. program, please apply directly to MIT EECS admissions. For all other positions, send me an e-mail with your resume.

Pathway to Robotic Intelligence , MIT Schwarzman College of Computing Talk, 2024.
Making Robots as Intelligent as ChatGPT, Forbes, 2023.
Robot Learning for the Real World, Forum for Artificial Intelligence, UT Austin, March 2023.
Fun with Robots and Machine Learning , Robotics Colloqium, University of Washington, Nov 2022.
Navigating Through Contacts , RSS 2022 Workshop in The Science of Bumping into Things.
Coming of Age of Robot learning , Technion Robotics Seminar (April 14 2022) / MIT Robotics Seminar (March 2022).
Rethinking Robot Learning , Learning to Learn: Robotics Workshop, ICRA'21.
Self-Supervised Robot Learning, Robotics Seminar, Robot Learning Seminar, MILA.
Challenges in Real-World Reinforcement Learning, IAIFI Seminar, MIT.
The Task Specification Problem, Embodied Intelligence Seminar, MIT.

A robotic system that peels vegetables with a dexterous robot hand.

A method for parallel training of large models on computers with limited memory.

Refining behavior-cloned diffusion model policies using RL.

A framework to train robots on scans of real-world scenes.

State-of-the-art exploration by optimizing the agent to achieve randomly sampled latent goals.

An LLM-based task planner that can learn new skills opens doors for continual learning.

Learning to control the force applied by a legged robot's arm for compliant and forceful manipulation.

Principled energy minimization increases robot's agility.

Learning complex assembly skills from few human demonstrations.

Learning reward functions from videos of human demonstrations.

Robotic finger designed to perform every day tasks.

A real-time controller that dynamically reorients complex and novel objects by any amount using a single depth camera.

Composing existing foundation models operating on different modalities to solve long-horizon tasks.

Method for guiding goal-directed exploration with asynchronous human feedback.

Optimizing the sampling distribution enables offline RL to learn a good policy in skewed datasets primarily composed of sub-optimal trajectories.

Relational rearrangement with multi-modal placing and generalization over scene layouts via diffusion and local scene conditioning.

Learn to perceive physical properties of terrains in front of the robot (i.e., a digital twin).

Learning to navigate by moving the camera across random images.

Leveraging crowdsourced non-expert human feedback to guide exploration in robot policy learning.

An algorithm for automatically balancing learning from teacher's guidance and task reward.

A set of suggestions that simplifies training of vector quantization layers.

Scaling Q-learning algorithms to 10K+ workers.

A step towards using counterfactuals for improving policy adaptation.

In-distribution error for certain features predicts their out-of-distribution sensitivity.

Dynamic legged object manipulation on diverse terrains with onboard compute and sensing.

Localize, identify, and fetch a target object in the dark with tactile sensors.

Return conditioned generative models offer a powerful alternative to temporal-difference learning for offline decision making and reasoning with constraints.

Transductive reparameterization converts out-of-support generalization problem into out-of-combination generalization which is possible under low-rank style conditions.

Return reweighted sampling of trajectories enables offline RL algorithms to work with skewed datasets.

Deeper Networks find simpler solutions! Also learn why ResNets overcome the challenges associated with very deep networks.

Method that automatically balances exploration bonus or curiosity against task rewards leading to consistent performance improvement.

Learning relational tasks with a few demonstrations in a way that generalizes to new configurations of objects.

One learned policy embodies many dynamic behaviors useful for different tasks.

Being adaptive instead of being robust results in faster adaption to out-of-distribution tasks.

Tactile Simulator for complex shapes training on which transfers to real-world.

High-speed running and spinning on diverse terrains with a RL based controller.

State-of-the-art Performance on Habitat Navigation Challenge without any machine learning for navigation.

An SE(3) Equivariant method for specifiying and finding correspondences which enables data efficient object manipulation.

A method for users to easily design a variety of robotic manipulators with integrated tactile sensors.

Predicting contact points with a CVAE and plane segmentation improves object generalization and handles multimodality.

Graph-based one-shot reward learning via active learning for object rearrangement tasks.

Online adaptation of offline RL policies using evaluation data improves performance.

Sampling data from the replay buffer informed by topological structure of the state space improves performance.

Bilinear decomposition of the Q-value function improves generalization and data efficiency.

Study revealing complementarity of invariance and equivariance in contrastive learning.

Fourier features improve value estimation and consequently data efficiency.

A framework for general in-hand object reorientation.

A hierarchical control framework for dynamic vision-aware locomotion.

Extreme viewpoint generalization via 3D representations based on Neural Radiance Fields.

Computational method for design task-specific robotic hands.

A MDP formulation that dissociates task relevant and irrelevant information.

Data efficient learning method for controlling microrobots.

Learning action primitives for data efficient online and offline RL.

A framework that achieves the best of TAMP and robot-learning for manipulating rigid objects.

Combining graph neural networks with curriculum learning for solve long horizon multi-object manipulation tasks.

A method for storing multiple neural network models for different tasks into a single neural network.

Computer Vision based Fall Detection system reduces number of falls and emergency room visits in people with Dementia.

Self-supervised learning of skills helps an agent imitate the task presented as a sequence of images. Forward consistency loss overcomes key challenges of inverse and forward models.

An empirical study of various kinds of prior information used by humans to solve video games. Such priors make them significantly more sample efficient as compared to Deep Reinforcement Learning algorithms.

A self-supervised method for learning to segment objects by interacting with them.

Computer vision method for building fully automated and scalable analysis pipeline for echocardiogram interpretation.

Intrinsic curiosity of agents enables them to learn useful and generalizable skills without any rewards from the environment.

Feature learning by making use of an agent's knowledge of its motion.

Self-supervised learning of low-level skills enables a robot to follow a high-level plan specified by a single video demonstration. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

Deep reinforcement learning can equip an agent with the ability to perform experiments for inferring physical quanities of interest.

Analysis how continuous video monitoring and review of falls of individuals with dementia can support better quality of care.

Iterative Error Feedback (IEF) is a self-correcting model that progressively changes an initial solution by feeding back error predictions. In contrast to feedforward CNNs that only capture structure in inputs, IEF captures structure in both the space of inputs and outputs.

Robot learns how to push objects to target locations by conducting a large number of pushing experiments. The code for the paper Zero Shot Visual Imitation subsumes this project's code release.

An empirical investigation into various factors related to the statistics of Imagenet dataset that result in transferrable features.

This work explores how an agent can be equipped with an internal model of the dynamics of the external world, and how it can use this model to plan novel actions by running multiple internal simulations (“visual imagination”).

Large-scale study of feature learning using agent's knowledge of its motion. This paper extends our ICCV 2015 paper.

Feature learning by making use of an agent's knowledge of its motion.

A detailed study of how to finetune neural networks and the nature of the learned representations.

Comparing the representations learnt by a Deep Neural Network optimized for object recognition against the human brain.

Method for identifying important parts of a group conversation directly from speech data.

template / accessibility