/ home / info / publications / teaching / projects /
/ archive / current projects / art+hacks /

Current Projects

My current research is focused on Perceptual User-Interfaces (PUI). I.e. devices that use sensor technologies that could read and partly understand issues like: proximity, sounds, motion, and electronic "eyes" for identification and awareness.

Perceptual User-Interfaces

Humans communicate using speech, gesture, and body motion, yet today's computers do not use this valuable information. Instead, computers force users to sit at a typewriter keyboard, stare at a TV-like display, and learn an endless set of arcane commands -- often leading to frustration, inefficiencies, and disuse.

The idea behind PUI is that a computer system "hears" users' voice commands and "sees" their gestures and body positions. Interactions are natural, more like human-to-human interactions. PUI use here machine perception to allow users to interact with computers. By reading gestures, motions and speech we should be able to in a much more natural way interact with computers.

But sensor systems deliver only raw position and pose information. For interface use, these are not the desired quantities—we need to understand the abstractions appropriate for a natural interface and consider how the various perceptual input degrees of freedom should relate to available commands and options.

Index

These are a some examples of projects that use vision-based perceptual user-interfaces. Each of these projects work on a particular possibility: the use of PUI to create reactive environments, incorporating PUI to everyday appliances and the possibility of integrating PUI in portable devices.

Interactive Wall - Gesture + Play
Intelligent Work Space
Post It - Do It
Perceptive Presence
Face IT

Interactive Wall - Gesture + Play

The Gesture + Play project examines the interaction between physical and perceptual interfaces in games and virtual environments. The central part of the project is to demonstrate different abstract interface regimes for body- and gesture-based control of games and virtual environments.

To study this we have developed a toolkit for vision-based perceptual interface that can track users’ movement and gesture in real-time. The toolkit is based on stereo vision that allows accurate estimation of 3-D position and orientation cues and is also robust to crowds, lighting changes, motion, and clothing variation. Our toolkit returns the 3D position and posture of a multiple users as they move in an arbitrary workspace. Gestures, actions, and motions in this space can cause motion and action in a virtual workspace, e.g. in a game or avatar-world. For this installation we are focusing primarily on game environments, for the playful and evocative nature of their visual content and the variety of possible perceptual interaction styles. We are using a game engine based on the HALF-LIFE system, which is similar in spirit to the well-known DOOM program, but which allows user designed virtual environments.

/ UIST 2002 - Tollmar, Dimmirijian and Darrell /

/ AI Lab Abstracts /

Intelligent Work Spaces

The Intelligent Work Space project is an ongoing research initiative at MIT AI Lab in the field of ubiquitous computing. Ubiquitous computing, or calm technology, is a paradigm shift where technology becomes virtually invisible in our lives. Instead of having a desk-top or lap-top machine, the technology we use will be embedded in our environment. We are still in an early phase of this trend and the questions of how we make ubiquitous computing accessible anywhere, adaptable to changes and ultimate how these new systems could enrich our experience is a fundamental part in our research within the Oxygen project at MIT AI Lab.

In this project will we focus our attention towards how we could create a flexible work environment in an office space that enables a flexible use of computing resources in everyday work situations. In a project meeting one might need a shared table for work material, or multiple displays for participants’ notes. In a brainstorming meeting one might need floor space to enable several people to sketch simultaneously on a large whiteboard. One might also want to welcome visitors or work undisturbed, so having a way to indicate one’s availability may prove useful.

To test these ideas we are rebuilding an office within our lab as a prototype platform for novel interaction technology and smart interior design solutions. We have informed our design of the space by observing how people use their offices and by studying literature on workplace habits. In addition have several of AI Labs faculty members equipped their offices with some technology that enable them to do visual presentations, have discussions with students etc. This gives us an opportunity to later test our ideas in real situations.

/ WACS'02 /

/ Activity Maps for Location-Aware Computing /

/ An Intelligent Workspace /

The Perceptive Presence Lamp

The Perceptive Presence Lamp is a widgetbased Perceptual User Interface (PUI) application that conveys presence information between two physically remote locations via a color-changing lamp. The lamp
uses computer vision-based approaches in order to determine when a person is present, busy (with another person), away, or desiring to interact with their peer on the other side of the lamp. Work in progress include a user study, defintion of a PUI widget set and demonstrating the qualitative and quantitative effectiveness of this approach.

/ Perceptive Presence in Intelligent Environments /

Face IT

A key component of proposed pervasive computing environments is the ability to use natural and intuitive actions to interact with computer systems. In this project at MIT AI Lab are we studying the use of the face as one interaction style when we communicate with computers.

The main reason for face based user interfaces being attractive is that the human face can express such a wide and broad interests of the user - it is a potential porthole into the current mindset through the direction, gaze, pose and expression that are all fundamental modes of human communication. People signal intent, interest, and direction with their faces; new, perceptually enabled interfaces can allow them to do so with computer systems as well.

We are currently constructing a series of prototypes that use a simple, real-time tracking system that enable us to explore different aspects of using facial clues to understand a users intentions and attention.

/ UbiComp 2002 - Darrell, Tollmar, Bentley, Checka, Morency, Rahimi and Oh /

Post it - Do it

One of the interaction mechanisms that we are studying is a concept that we labeled Post It – Do It. Imagine that you scribble small messages on post-it notes but instead of being read and remembered by human a computer system read the notes via a camera. When you later would like to trigger a note, for example send an email, turn on some music or locate your cell phone among many many more, simple point to the note again with your fingertip and the computer will try to perform that task. This enables an interaction that is without any of the standard interaction devices, like keyboard, mouse, microphones etc. that has a proven disadvantage in being highly inflexible and often leashed by some cables.