Chris Stauffer's Research Summary
<table class="headertable" width="100%">
<tr>
<td class="bannercorner" align="center" width=150><a href="index.php" target="_top">Chris Stauffer<br>Home</a>
</td>
<td class="bannermain" align="center"><a href="stauffer_publications.php" target="_top">Publications</a>
</td>
<td class="bannermain" align="center"><a href="stauffer_research.php" target="_top">Research</a>
</td>
<td class="bannermain" align="center"><a href="stauffer_resources.php" target="_top">Resources</a>
</td>
<td class="bannermain" align="center"><a href="stauffer_personal.php" target="_top">Personal</a>
</td>
<td class="bannermain" align="center"><a href="stauffer_misc.php" target="_top">Misc</a>
</td>
<td class="bannercorner" align="right" width=150><b>User:</b> Anonmyous<br><a href="stauffer_login.php?login=true" target="_top">Log In</a></td>
</tr>
</table>
<script type="text/javascript">
function Img(ref,src) {
this.ref=ref;
this.src=src;
this.img=new Image();
this.img.src=src;
}
function ImgList() {
this.imgs=new Array;
this.numImgs=0;
}
ImgList.prototype.addImg=function(tempimg) {
this.imgs[this.imgs.length]=tempimg;
this.numImgs=this.numImgs+1;
}
ImgList.prototype.getImg=function(imgref) {
//alert('length '+this.imgs.length);
var i=0;
while (i<this.imgs.length) {
//alert('foo '+this.imgs[i].ref+' ... '+imgref);
if (this.imgs[i].ref==imgref) {
return this.imgs[i];
}
i=i+1;
}
}
function addImgsToImgList(imgarray) {
ilist=new ImgList();
for (var i=0; i<imgarray.length; i++) {
ilist.addImg(new Img(imgarray[i][0],imgarray[i][1]));
}
return ilist; // must be stored globally
}
function foo () {
//alert('got here 1');
ilist=new ImgList();
//alert('got here 2');
//alert('got here 3');
}
function msover(objref,imgref) {
//alert('setting '+objref+' to '+imgref);
//alert(document.getElementById(objref));
//alert(ilist.getImg(imgref).src);
document.getElementById(objref).src = ilist.getImg(imgref).src;
}
/* This is the Mouseout function. When the mouse moves off a button, it
* will restore the button to the normal state, bring the picture back to
* the default state
*/
function msout(num) {
}
</script>
<script type="text/javascript">
// define array
imgarray=[
["quadoverview1","_papers/thesis.jpg"],
["resoverview1","images/ResearchOverview_small.gif"],
["trackingover","images/Ct.gif"],
["calibover","images/techsquareoverviewgraph.jpg"],
["networkover","images/overlappingexample1.jpg"],
["classover","images/classover.gif"],
["shapeover","images/hiershapeslide.gif"],
["activityover","images/heirslide1.gif"],
["appearanceover","images/peddecomp_shirtclust.gif"],
["temporalover","images/temporalover.gif"],
["listenover","images/listen.jpg"],
["avover","images/avover.jpg"],
["avdetect","images/avover.jpg"],
["avcluster","images/avaudcluster.jpg"],
["avactivity","images/avclustering.jpg"],
["lowlevelover","images/lowlevelover.jpg"],
["lastimg","images/listen.jpg"]
];
// setup array
ilist=addImgsToImgList(imgarray);
</script>
<table width="100%">
<tr>
<td>
<div align=center style="font:2em">Outline</div>
<div align=center style="font:1em;color:#CC0000;" alt="This page is currently under construction. This notice will be removed when this is no longer the case.">(page under construction)</div>
</td>
</tr>
<tr>
<td align="center">
<img style="border:3 inset" src="images/ResearchOverview_small.gif" id="resimg" name="resimg" height="100">
</td>
</tr>
<tr>
<td>
<div class="res1">
<a onmouseover="msover('resimg','resoverview1')"
target="stmain"
href="stauffer_research.php?stover=x">Overview</a>
<div class="res3">
<a onmouseover="msover('resimg','resoverview1')"
target="stmain"
href="stauffer_research.php?stover=x#highlights">Highlights</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','resoverview1')"
target="stmain"
href="stauffer_research.php?stover=x#motivation">Motivation</a>
</div>
</div>
<div class="res1">
<div onmouseover="msover('resimg','quadoverview1')"
target="stmain"
>Vision</div>
<div class="res3">
<a onmouseover="msover('resimg','trackingover')"
target="stmain"
href="sttracking.php#tracking">Tracking</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','calibover')"
target="stmain"
href="stcalibration.php">Calibration</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','networkover')"
target="stmain"
href="stcalibration#networks">Sensor Nets</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','classover')"
target="stmain"
href="stclassification.php">Classification</a>
<div class="res4">
<a onmouseover="msover('resimg','shapeover')"
target="stmain"
href="stclassification.php#shape">Shape-based</a>
</div>
<div class="res4">
<a onmouseover="msover('resimg','activityover')"
target="stmain"
href="stclassification.php#activity">Activity</a>
</div>
<div class="res4">
<a onmouseover="msover('resimg','appearanceover')"
target="stmain"
href="stclassification.php#appearance">Appearance</a>
</div>
</div>
<div class="res3">
<a onmouseover="msover('resimg','temporalover')"
target="stmain"
href="stclassification.php#context">Temporal Context</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','resoverview1')"
target="stmain"
href="stclassification.php#anomaly">Anomaly Detection</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','listenover')"
target="stmain"
href="stclassification.php#communication">Communication</a>
<div class="res4">
<a onmouseover="msover('resimg','listenover')"
target="stmain"
href="stclassification.php#communication">Query-based</a>
</div>
<div class="res4">
<a onmouseover="msover('resimg','listenover')"
target="stmain"
href="stclassification.php#communication">Learning Language</a>
</div>
</div>
</div>
<div class="res1">
<div onmouseover="msover('resimg','avover')">Audio-Visual</div>
<div class="res3">
<a onmouseover="msover('resimg','avdetect')"
target="stmain"
href="_audio/index.html#detection">Detection</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','avcluster')"
target="stmain"
href="_audio/index.html#classification">Classification</a>
</div>
<div class="res3">
<a onmouseover="msover('resimg','avactivity')"
target="stmain"
href="_audio/index.html#analysis">Activity Analysis</a>
</div>
</div>
<div class="res1">
<div onmouseover="msover('resimg','lowlevelover')">Learning</div>
<div class="res3">
<a onmouseover="msover('resimg','lowlevelover')"
target="stmain"
href="_lowlevel/index.html">Low-level Vision</a>
</div>
</div>
<div class="res1">
<div onmouseover="msover('resimg','resoverview1')">Other</div>
<div class="res3">
<a onmouseover="msover('resimg','activityover')"
target="_blank"
href="http://www.ai.mit.edu/projects/vsam">Forest of Sensors</a>
</div>
</div>
<!-- -research efforts
-hci room
-VSAM
-CzTS
-STTR w/ alphatech.
-Miscellaneous
-xxxkdjfs
-tracking-based automatic object recognition
-automatic scene activity modeling
-usupervised audio scene analysis
-similarity templates for detection and recognition
-learning a non-parametric articulated pedestrian representation
-classes and misc projects
-6.892 statistical computer vision
-4.208-6.891 Advanced graphics
-water simulation >
</td>
</tr>
</table>
<table width="100%" colgroup="*,150,*" border=3>
<tr>
<td rowspan=2 align="center">
<a href="images/ResearchOverview.gif">
<img width=180 src="images/ResearchOverview_small.gif">
</a>
</td>
<td align="center">
<div style="font:18pt; font-weight:bold">Chris Stauffer</div>
<div style="font:16pt;">Research Overview</div>
</td>
<td rowspan=2 align="center">
<a href="images/ResearchOverview.gif">
<img width=180 src="images/ResearchOverview_small.gif">
</a>
</td>
</tr>
<tr>
<td align="left">
<div style="font-weight:bold; text-decoration:underline;">Page Contents</div>
<a href="#overview">Overview</a><br>
<a href="#highlights">Highlights</a><br>
<a href="#motivation">Motivation</a><br>
</td>
</tr>
</table>
<a name="overview">
<h1>Overview</h1>
The primary goal of my research is to develop systems that are capable of being introduced to a new environment and capable of <b>automatically</b>:
<ul>
<li>Detecting the objects in the environment.</li>
<li>Tracking the objects over time within a camera</li>
<li>Tracking the objects across multiple cameras</li>
<li>Determining when an object reappears after being absent for a period of time</li>
<li>Calibrating the cameras</li>
<li>Extracting useful descriptions of the objects</li>
<li>Clustering and classifying the types of objects</li>
<li>Clustering and classifying the appearances of objects</li>
<li>Clustering and classifying the activities of objects</li>
<li>Clustering and classifying the interactions of objects</li>
<li>Learning the temporal context of the activity in an environment</li>
<li>Detecting anomalous objects, behavoir, or scene dynamics based on the learned models</li>
<li><b>RECENT</b> Capable of detecting, clustering, and understanding environmental sounds</li>
</ul>
<hr>
<a name="highlights">
<img style="border:3 inset;float:right;" height=150 name="highimg" src="images/alignABCi.gif">
<h1>Highlights</h1>
<p>If you are not on the frames version of this page, please click <a
href="stauffer_research.php" target="_top">here</a>. I have made
progress on every item in the <a href="#overview">list<a> above.
There are different projects highlighted throughout my <a
target="_top" href="http://www.csail.mit.edu/~stauffer/">home
page</a>. For those with less time, there is a set of highlight links
listed below that some may find useful.</p> <ul>
<li>
<a href="sttracking.php#tracking" target="stmain">Robust
Persistent Tracking</a>- This is the lynch-pin of my research. The
ability to reliably track over extended periods of time is required
for most of the research discussed below.
</li>
<li>
<a href="http://www.ai.mit.edu/projects/vsam/"
target="_blank">A Forest of Sensors</a>- This was the first major
funding effort I participated in. The link goes to the project page
and shows some of our early work in automated intelligent video
surveillance.
</li>
</ul>
<hr>
<a name="motivation">
<h1>Motivation</h1>
<p> Imagine if computers possessed a small child's ability to observe
and comprehend their dynamic environment. Very young children can
track moving objects, differentiate them, identify them, understand
their activity, and understand their interactions with static objects
and with other moving objects in the World. There are amazing
benefits in coupling these basic human abilities with the abilities of
computers: to communicate instantly with high bandwidth; to store huge
amounts of observations with perfect recall; to process in parallel;
and to draw inferences over extremely large stores of data. These
benefits have driven the increased use of automated computer vision
systems, such as intelligent visual surveillance systems, traffic
analysis systems, quantitative experimental mouse and drosophila
observation systems, and general tracking and classification as a
basis for high-level computer vision systems that are predicated on
such capabilities.</p>
<p>My research is focused on developing computational analogs for
basic human perception and exploiting the strengths of computers to
take full advantage of these capabilities. My research over the past
seven years has centered on the development of systems that are
capable of: automatically tracking multiple objects in real-time
across multiple overlapping and non-overlapping cameras in
unstructured indoor and outdoor environments; automatically modeling
the types of objects in a particular environment; automatically
modeling the activities that these objects perform; learning patterns
of the activities over extended periods of time; and detecting unusual
objects or behavior. I have made significant progress and variants of
my systems are in wide-spread use, but there is still considerable
work to be accomplished in this area of research.</p>
<p>
<a href="images/trackingoverlay1.jpg">
<img class="res1" style="float:right;"
src="images/trackingoverlay1.jpg" width="45%">
</a>
This research is built upon our capability to track moving objects in
complex, unstructured environments. When I first arrived at MIT
almost a decade ago, I was priveleged to work on one of the first
computer systems that enabled real-time video processing of any
consequence. I developed the first tracking system that was used for
real-time tracking of multiple objects in complex outdoor environments
continuously for periods of weeks or months at a time
[staufferCVPR1998a]. This project has been in virtually continuous
operation, longer than any autonomous visual tracking system to my
knowledge. Because this work forms a foundation for many other
computer vision techniques and presents many difficult problems, there
is a growing community pursuing these techniques. This is evident
from the more than 150 citations this work has received since its
original publication and the increased focus on these issues in
computer vision workshops, conferences, and journals. </p>
<p>
<a href="images/overlappingexample1.jpg">
<img style="float:left;"
src="images/overlappingexample1.jpg" width="45%">
</a>
Given the ability to track effectively within individual cameras, I
investigated coordinating tracking across multiple cameras and other
sensors. I developed a novel system for automatically calibrating
multiple overlapping cameras that may (or may not) have regions of
overlap using only the tracked objects. This system can reliably
calibrate a large number of overlapping cameras in complex
environments. I have also developed techniques for tracking objects
across non-overlapping camera networks. Currently, I am developing
the capability to integrate other sensor types into these sensor
networks including audio sensors and GPS sensors. As these
capabilities mature, we will be able to integrate information from
multiple sources to characterize individuals. For instance, we can
determine whether the person who left his bag at Northwest Airline
Gate C38 paid for his ticket with cash and has left the airport or
whether that person has just gone to the bathroom one minute prior to
the detection of the unattended luggage. The figure to the left shows
an automatically calibrated network of five cameras.
</p>
<a href="images/heirslide1.gif">
<img src="images/heirslide1.gif" width="53%">
</a>
<a href="images/hiershapeslide.gif">
<img src="images/hiershapeslide.gif" width="45%">
</a>
<p>
Given the capability to track literally millions of independent
objects over an extended scene over the period of a month, there is an
obvious need to develop a compact description of these objects and
their behaviors. I have developed multiple probabilistic, data-driven
classification and clustering techniques that limit the amount of
operator supervision and intervention required. Using a pair-wise
co-occurrence-based spectral clustering technique, I automatically
modeled the different types of objects in an outdoor scene as seen on
the right. The system automatically found a hierarchy of clusters for
vehicles, pedestrians, groups of pedestrians, and scene clutter based
solely on the statistics of the shape in the scene. I have extended
this work to allow human supervision to specify the types of objects
and iteratively correct misclassifications while still exploiting the
natural statistics of the observed data. Using a similar technique
shown on the left, I automatically clustered the activity in the same
environment into a similar hierarchical representation. This
representation divided the tracked objects based on directionality,
size, speed, and other factors based on which factors were the most
salient in the particular scene.</p>
<a href="images/peddecomp_w_color.gif">
<img src="images/peddecomp_w_color.gif" width="49%">
</a>
<a href="images/peddecomp_shirtclust.gif">
<img src="images/peddecomp_shirtclust.gif" width="49%">
</a>
<p>
I also employed a similarity-based decomposition method to
automatically infer a compact representation for a single class of
objects- pedestrians. By analyzing the global similarity statistics
across a set of un-segmented pedestrian images, a hierarchical
description of this class of object was automatically generated (upper
left). The resulting hierarchical description shows that for
pedestrians in Boston, the foreground region, background region,
shirt, pants, shoes, and head were among the most regular regions in
pedestrian images. Using this automatically-generated,
domain-specific regional decomposition, the colors of the clothes of
the pedestrians can automatically be analyzed, clustered, and stored
for future processing. Clustering and anomaly detection is shown
(upper right) for the region that corresponds to the shirt region. It
shows eight characteristic shirt colors with twelve examples of each,
as well as twelve pedestrians wearing "unusual" shirt colors. I am
currently working to extend this research to vehicles and other object
types.</p>
<p>Being the first researcher, to my knowledge, to automatically track
millions of objects over periods of months and years, I have also
investigated long-term models of object activity. I have
automatically determined cyclic patterns of activities resulting from
traffic signals, diurnal activity, and weekly activity patterns. I
have also investigated modeling interactions between pairs of objects
in an environment. Given the probabilistic, descriptive models of the
objects, activities, interactions, and patterns of activity over time,
these systems have the capability of describing anomalous activity,
including unusual object types with unusual appearances in unusual
places doing unusual things at unusual times. Detecting unusual
activities is particularly useful in persistent deployable systems
that can adapt to change in their environment and to detect salient
events in an environment.</p>
</body>
</html>