Hierarchical Task and Motion Planning in Belief Space

by Leslie Pack Kaelbling and Tomas Lozano-Perez, MIT CSAIL

Journal Publications

Leslie Pack Kaelbling and Tomás Lozano-Pérez, Integrated Task and Motion Planning in Belief Space, International Journal of Robotics Research, 2013 PDF

Detailed Technical Reports (as of July, 2012)

MIT-CSAIL-TR-2012-018 focuses on hierarchical planning in the fully observable case.
MIT-CSAIL-TR-2012-019 focuses on belief space planning in the case with uncertainty.

Conference Publications

Leslie Pack Kaelbling and Tomás Lozano-Pérez, Unifying Perception, Estimation and Action for Mobile Manipulation via Belief Space Planning, IEEE Conference on Robotics and Automation (ICRA), May 2012. PDF
Leslie Pack Kaelbling and Tomás Lozano-Pérez, Pre-image backchaining in belief space for mobile manipulation, International Symposium on Robotics Research (ISRR), August 2011. PDF
Leslie Pack Kaelbling and Tomás Lozano-Pérez, Hierarchical Planning in the Now (Finalist, Best Manipulation Paper Award), IEEE Conference on Robotics and Automation (ICRA), May 2011. PDF

PR2 Videos (in MOV format)

These videos have been edited to cut out the "thinking" time. The planning time is about the same duration as the motion time. The underlying motion planner is a simple RRT with some smoothing, but not much, and so it generates motions that are often "surprising".

The actual PR2 manipulating two objects on a table (4X). See description in MIT-CSAIL-TR-2012-019.
The view from Rviz while PR2 manipulates two objects on a table. Pay special attention to the denser point clouds from the head sensor; the view points are directed by the plan. The red highlights indicate object detections; the blue rectangle indicates the table detection. See description in MIT-CSAIL-TR-2012-018.
The actual PR2 manipulating two objects in a cupboard (8X). See description in MIT-CSAIL-TR-2012-019.

Simulation Videos (in MOV format)

In the belief-space movies, the "shadows" around the objects and the robot reflect the current uncertainty. In the planning sequences, the cyan robot reflects paths and configurations generated during planning.

A long simulated sequence of the PR2 in an "observable" kitchen/apartment. See description in MIT-CSAIL-TR-2012-018.

State sequence

Simulated PR2 manipulating two objects on a table; This requires placing at the destination with high accuracy; the initial place fails and has to be re-planned and executed. See description in MIT-CSAIL-TR-2012-019.
- Belief states.
- Planning sequence
Simulated PR2 moving an object to look at a second object. See description in MIT-CSAIL-TR-2012-019.
- Belief states
- Planning sequence
- Belief state + Oct-Tree (plan differs since travel space must be observed)
Simulated PR2 manipulating across two tables. See description in MIT-CSAIL-TR-2012-019.
- Belief states
- Planning sequence

Some Planning and Execution Trees

Cycle 1 in the cupboard experiment with actual PR2. See description in MIT-CSAIL-TR-2012-019.
Cycle 4 in the cupboard experiment with actual PR2. See description in MIT-CSAIL-TR-2012-019.
Cycle 5 in the cupboard experiment with actual PR2. See description in MIT-CSAIL-TR-2012-019.
Swapping two objects - A then B See description in MIT-CSAIL-TR-2012-018.
Swapping two objects - B then A See description in MIT-CSAIL-TR-2012-018.
Serve food and tidy See description in MIT-CSAIL-TR-2012-018.

Cupboard experiment - Cycle 1

Cupboard experiment - Cycle 5

Acknowledgments

We thank all the members of the LIS research group for their help in this project, and in particular: George Konidaris, Lawson Wong, Jennifer Barry and Jon Binney for comments on drafts; Jared Glover, Aaron Fryman, Will Grathwol, Sanja Popovic and Dawit Zewdie for developing the 3D perception subsystem; Lawson Wong for the UKF implementation; Ashwin Deshpande and Dylan Hadfield-Menell for help with hierarchical planning; Jennifer Barry for help in testing the planner and with PR2 software; and Caelan Reed Garrett for help with Cython.

This work was supported in part by the NSF under Grant No. 1117325. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We also gratefully acknowledge support from ONR MURI grant N00014-09-1-1051, from AFOSR grant AOARD-104135 and from the Singapore Ministry of Education under a grant to the Singapore-MIT International Design Center. We thank Willow Garage for the use of the PR2 robot as part of the PR2 Beta Program.