\subsection{Spatio-Temporal Feature Tracking}
\label{sec:feature-tracking} 
%
The first step in articulation learning from visual demonstration
involves visually observing and tracking features on the object while
it is being manipulated. We focus on unstructured environments without
fiducial markers.  Our algorithm combines interest-point detectors and
feature descriptors with traditional optical flow methods to construct
long-range feature trajectories. We employ Good Features To Track
(GFTT)~\cite{shi1994good} to initialize up to 1500 salient features
with a quality level of 0.04 or greater, across multiple image
scales. Once the features are detected, we populate a mask image that
captures regions where interest points are detected at each
pyramid scale. We use techniques from previous work on dense optical
flow~\cite{farneback2003two} to predict each feature at the next
timestep. Our implementation also employs median filtering as
suggested by~\citet{wang2011action} to reduce false positives.

% In order to
% improve on the robustness of the optical flow estimation, the local flow of
% features are computed at each scale, and accordingly predicted.

% Additionally, in high sensor noise regimes
% where the features are not robustly detected, providing feature predictions as
% an alternative can be especially advantageous.


% Due to the unstructured nature of indoor scenes,
% the feature detector is evaluated at 5 different scales, each down-sampled by a
% factor of $\sqrt[]{2}$. With the use of scale-space methods, we fix the feature
% block size parameter, a GFTT feature detection parameter, at 7 pixels. 
% The combination of these
% implementation details provide sufficient guidance to the initialized features
% for their prediction in future frames, and consequently, construction of
% long-range feature trajectories. 

We bootstrap the detection and tracking steps with a feature
description step that extracts and learns the description of the
feature trajectory. At each image scale, we compute the SURF
descriptor~\cite{baya2008speeded} over features that were predicted
from the previous step, denoted as $\hat{f^t}$, and compare them with
the description of the detected features at time $t$, denoted as
$f^t$. Subsequently, detected features $f^t$ that are sufficiently
close to predicted features $\hat{f^t}$ and that successfully meet a
desired match score are added to the feature trajectory, while the
rest are pruned. To combat drift, we use the detection mask as a guide
to reinforce feature predictions with feature
detections. Additionally, we incorporate flow failure detection
techniques~\cite{kalal2010forward} to reduce drift in feature
trajectories.


% The
% expectation that feature trajectories should fall on the same point on the
% object provides a strong assumption that the underlying description of the
% feature itself should remain unchanged. 

% This ensures features continue to lie on salient
% regions in the scene and reduces erroneous optical flow vectors determined. This
% step also provides robustness to sensor noise that tends to generate spurious
% features in the image.

Like other feature-based methods~\cite{katz2013interactive} our method
requires visual texture.  In typical video sequences, some features
are continuously tracked, while other features are lost due to
occlusion or lack of image saliency. To provide rich trajectory
information, we continuously add features to the scene as needed. We
maintain a constant number of feature trajectories tracked, by adding
newly detected features in regions that are not yet occupied. From
RGB-D depth information, image-space feature trajectories can be
easily extended to 3-D. As a result, each feature key-point is
represented by its normalized image coordinates $(u,v)$, position
$\vec{p} \in \mathbb{R}^3$ and surface normal $\vec{n}$, represented
as $(\vec{p},\vec{n}) \in \mathbb{R}^3 \times SO(2)$. We denote
$\textbf{\text{F}} = \{F_1, \dots, F_n\}$ as the resulting set of
feature trajectories constructed, where $F_i = \{(\vec{p_1},
\vec{n_1}), \dots, (\vec{p_t}, \vec{n_t})\}$. To combat noise inherent
in our consumer-grade RGB-D sensor, we post-process the point cloud
with a fast bilateral filter~\cite{paris2006fast} with parameters
$\sigma_s=20$~px, $\sigma_r=4$~cm.

% The resulting feature
% trajectory is denoted as $f^t = (\vec{p},\vec{n})^t \in \mathbb{R}^3 \times
% SO(2), \forall t \in T$ representing the trajectory taken by an object feature.

% Additionally, with the dense organized point cloud representation that
% RGB-D sensors provide, object surfaces and surface normals are easily extracted
% with the use of integral depth images~\citep{holz2012real}. 

% In order to understand the interaction between the
% user and the articulated object being manipulated, it is crucial to identify and
% trace the path that the object and its parts take while being manipulated. We
% emphasize the need for higher-fidelity observations of trajectories that enable
% stronger inference over the underlying kinematic model of the object. In this
% work, we explore the use of image-based interest-point features and extend them
% to 3-D to build trajectories of the object parts while they are under
% motion. This allows the robot to learn the traced path that the object takes
% while it is being manipulated, and later provide sufficient information about
% its relevance to the overall manipulation task.

% \subsection{Trajectories via Interest-Point Tracking and Learning} Since the
% manipulated object may undergo various viewpoint changes and occlusions, the
% tracking pipeline needs to be capable of adapting to these
% variations. We employ feature-based techniques
% to track and detect interest points on the articulated object. 

% \subsubsection{Feature Detector} Feature trajectories require an initialization
% step, where salient scene features are extracted based on certain criterion
% threshold. We employ Good Features To Track (GFTT) \cite{shi1994good} to
% initialize up to 1500 salient features whose quality level is greater that
% 0.04. Due to the unstructured nature of indoor scenes, the feature detector is
% evaluated at 5 different scales, each down-sampled by a factor of
% $\sqrt[]{2}$. This provides additional scale-space robustness to the feature
% detector. This allows us to fix the feature block size parameter, a feature
% detection parameter, to 7 pixels. Once the features are detected, we populate a
% mask image that keeps track of regions where interest points are detected at
% each of the pyramid scales. The detection step is visualized in
% Figure~\ref{fig:vis-feature-tracking-pipeline}-b

% \subsubsection{Feature Prediction} Once the features are initialized, the
% features need to be propagated in a manner that conforms to the motion of the
% object locally at that point. We draw techniques from previously established
% work on dense optical flow by~\citet{farneback2003two} to provide predictions
% for the initialized features at the next timestep. Our specific implementation
% also adds a median filtering step as suggested in~\cite{wang2011action} that
% reduces false positives in the typical dense optical flow implementation. In
% order to improve on the robustness of the optical flow estimation, we employ
% scale-space methods to predict the local flow of features at each scale. The
% combination of these implementation details provide sufficient guidance to the
% initialized features for their prediction in future frames. Additionally,
% in high sensor noise regimes where the features are not robustly detected,
% providing feature predictions as an alternative can be especially
% advantageous. The prediction step is visualized in
% Figure~\ref{fig:vis-feature-tracking-pipeline}-b.

% \subsubsection{Bootstrapped Learning and Detection} Optical flow inadvertently
% causes drift over larger time periods, and require correction steps to prevent
% this drift. We bootstrap the detection and tracking steps with a feature
% description step that extracts and learns the description of the feature
% trajectory. The expectation that feature trajectories should fall on the same
% point on the object provides a strong assumption that the underlying description
% of the feature itself should remain unchanged. We use this assumption to
% bootstrap feature description at every subsequent frame to avoid longer-range
% variations in drift that we allude to earlier. While we do not investigate the
% optimal parameter space of this feature learning interval, we do expect this
% interval to be longer than a few frames wide as it is designed to complement
% optical flow methods that tend to capture the variation over smaller time
% periods. At each image scale, we compute the SURF
% descriptor~\cite{baya2008speeded} over feature patches that were predicted from
% the previous step, and compare them with the description of the detected feature
% that is within a threshold distance from the predicted feature. Subsequently,
% features that successfully meet a desired match score are added to the feature
% trajectory, while the rest are pruned. The bootstrapping step is visualized in
% Figure~\ref{fig:vis-feature-tracking-pipeline}-(d,e)

% \subsubsection{Trajectory Correction} Apart from using feature descriptors to
% avoid drift in the trajectories, we also constantly correct the feature
% predictions with an additional feature detection step. Vanilla implementations
% of the feature detection-learning-update concept are prone to drift in the
% earlier stages of the feature trajectory. However, with an additional assumption
% that predicted feature trajectories, as suggested by the optical flow vector,
% tend to fall on or near regions of subsequent feature detections provide
% additional support for the correspondence and trajectory construction. We employ
% the previously computed detection mask as a guide to reinforce feature
% predictions with feature detections, and subsequently construct valid feature
% trajectories with little to no drift. This ensures features continue to lie on
% salient regions in the scene and reduces erroneous optical flow vectors
% determined. This step also provides robustness to sensor noise that tend to
% generate spurious features in the image. The trajectory correction step is
% visualized in
% Figure~\ref{fig:vis-feature-tracking-pipeline}-c. Figure~\ref{fig:feature-tracking-drift}
% shows the superior performance of our algorithm in tackling drift issues.

% \subsubsection{Trajectory Management} For typical video sequences, some features
% are continues tracked, while other features are lost due to occlusion or lack
% of salience in the image regions. To provide rich trajectory information, we
% continuously add features to the scene, when necessary, using the occupancy mask
% described earlier. We maintain a constant number of feature trajectories tracked,
% by adding newly detected features in regions that are not yet occupied as
% suggested by the occupancy mask. This step is shown in
% Figure~\ref{fig:vis-feature-tracking-pipeline}-f. As features are continually
% added for tracking, we incorporate track management capabilities that also
% provide relevant statistics about the feature trajectories and their validity. Some of
% the capabilities include statistics on continuous feature descriptions for a
% particular trajectory, unique trajectory identification, sub-pixel refinement,
% and depth uncertainty estimation from the RGB-D image.

% %
% %
% \begin{figure}[!t]
%     \centering
%     \includegraphics[width=\columnwidth]{graphics/pipeline-figure/feature-tracker.pdf}
%     \caption{Spatio-Temporal Feature Tracking steps combine traditional dense
%       optical flow methods with feature detection and matching techniques to
%       construct long range trajectories with negligible drift. }
%     \label{fig:vis-feature-tracking-pipeline}
% \end{figure}
% %
% %


