Unsupervised Joint Object Discovery and Segmentation in Internet Images

CVPR 2013 Unsupervised Joint Object Discovery and Segmentation in Internet Images

Image datasets collected from Internet search vary considerably in their appearance and typically contain many noise images that do not contain the object of interest (small subset of the car image dataset is shown in (a); the full dataset is available in the accompanying material). Our algorithm automatically discovers and segments out the common object (b). Note how no objects are discovered for noise images in (b). Most previous co-segmentation methods, in contrast, are designed for more homogeneous datasets in which every image contains the object of interest, and, therefore, their performance degrades in the presence of noise (c).

Abstract

We present a new unsupervised algorithm to discover and segment out common objects from large and diverse image collections. In contrast to previous co-segmentation methods, our algorithm performs well even in the presence of significant amounts of noise images (images not containing a common object), as typical for datasets collected from Internet search. The key insight to our algorithm is that common object patterns should be salient within each image, while being sparse with respect to smooth transformations across other images. We propose to use dense correspondences between images to capture the sparsity and visual variability of the common object over the entire database, which enables us to ignore noise objects that may be salient within their own images but do not commonly occur in others. We performed extensive numerical evaluation on established co-segmentation datasets, as well as several new datasets generated using Internet search. Our approach is able to effectively segment out the common object for diverse object categories, while naturally dentifying images where the common object is not present.

@article{Rubinstein13Unsupervised, author = {Michael Rubinstein and Armand Joulin and Johannes Kopf and Ce Liu}, title = {Unsupervised Joint Object Discovery and Segmentation in Internet Images}, journal = {IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)}, year = {2013}, month = {June} }

Paper: PDF

Supplementary Material: link

CVPR 2013 Poster: PDF (45mb)

Data and Code

Datasets and results (420mb) - all the datasets that appeared in the paper, including our Internet datasets with human foreground-background segmentations. It also contains our segmentation resutls and the results of several existing co-segmentation methods we compared with in the paper.

Matlab code - currently includes code for evaluating the segmentation accuracy and visualizing the results. Reproduces all the numerical evaluations and some of the
figures in the paper. See the README file for details.

Acknowledgements

We thank Antonio Torralba for his help in collecting human foreground-background segmentations for our Internet datasets. This work was done while Michael Rubinstein and Armand Joulin were interns at Microsoft Research Redmond. Michael Rubinstein is supported by the Microsoft Research PhD Fellowship. Armand Joulin is supported by the European Research Council (SIERRA and VIDEOWORLD projects).

Last updated: Jun 2013