"A Voice-Commandable Robotic Forklift Working Alongside Humans in Minimally-Prepared Outdoor Environments"

Matthew Walter
Postdoctoral Associate
Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology

The following documents the presentation of our Agile Robotics systems paper at the 2010 ICRA Conference in Anchorage, Alaska. The presentation described the overall system that we have developed to automate materiel handling in human-occupied, semi-structured environments. The work and paper are a collaboration with many others at MIT and BAE Systems.

Teller, S., Walter, M.R., Antone, M., Correa, A., Davis, R., Fletcher, L., Frazzoli, E., Glass, J., How, J.P., Huang, A.S., Jeon, J.H., Karaman, S., Luders, B., Roy, N., and Sainath, T., A Voice-Commandable Robotic Forklift Working Alongside Humans in Minimally-Prepared Outdoor Environments. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Anchorage, Alaska, May 2010.
[bibtex] [pdf]


One long-standing challenge in robotics is the realization of mobile autonomous robots able to operate safely in existing human workplaces in a way that their presence is accepted by the human occupants. We describe the development of a multi-ton robotic forklift intended to operate alongside human personnel, handling palletized materials within existing, busy, semi-structured outdoor storage facilities.

The system has three principal novel characteristics. The first is a multimodal tablet that enables human supervisors to use speech and pen-based gestures to assign tasks to the forklift, including manipulation, transport, and placement of palletized cargo. Second, the robot operates in minimally-prepared, semi-structured environments, in which the forklift handles variable palletized cargo using only local sensing (and no reliance on GPS), and transports it while interacting with other moving vehicles. Third, the robot operates in close proximity to people, including its human supervisor, other pedestrians who may cross or block its path, and forklift operators who may climb inside the robot and operate it manually. This is made possible by novel interaction mechanisms that facilitate safe, effective operation around people.

We describe the architecture and implementation of the system, indicating how real-world operational requirements motivated the development of the key subsystems, and provide qualitative and quantitative descriptions of the robot operating in real settings.


Presentation slides (see below for links to referenced movies)

pdf (11MB)

Creative Commons License



[mp4 (h264, 67MB)]

This video demonstrates the multimodal interaction mechanisms whereby a human supervisor conveys task-level commands to the robot via a hand-held tablet. These task-level commands include: directing the robot to pickup a desired pallet by circling it in an image from one of the robot's cameras; summoning the robot to a particular destination by speaking to the tablet; and directing the robot to place a pallet by circling the desired location on the ground or a truck.


[mp4 (h264, 24MB)]

A visualization of the pallet detection process, showing the different steps of the hierarchical classifier. Having been commanded to pickup an a priori unknown pallet from truck, the bot searches for the pallet by scanning with a tine-mounted LIDAR. Once the pallet has been found, the bot drives towards it and then reacquires the pallet prior to insertion.


[mp4 (h264, 9MB)]

This video demonstrates the result of our vision-based object reacquisition strategy. Using multiple views of an object acquired opportunistically and automatically from different vantage points, we are able to "reacquire" objects of interest. This video shows one application in which the user is able to circle multiple pallets for pickup at the outset, upon which the pallets are recognized and the gestures reconstituted (purple paths).The numbers accompanying each gesture indicate the number of views comprising the object's model.


[mp4 (h264, 61MB)]

A video that shows the forklift operating unmanned, picking up, transporting, and placing pallets as directed by the user via the tablet interface.

Last Modified: February 21, 2013