Transfer Learning by Borrowing Examples for Multiclass Object Detection

Presented at 2011 NIPS

Joseph J. Lim, Ruslan Salakhutdinov, Antonio Torralba

Massachusetts Institute of Technology


Despite the recent trend of increasingly large datasets for object detection, there still exist many classes with few training examples. To overcome this lack of training data for certain classes, we propose a novel way of augmenting the training data for each class by borrowing and transforming examples from other classes. Our model learns which training instances from other classes to borrow and how to transform the borrowed examples so that they become more similar to instances from the target class. Our experimental results demonstrate that our new object detector, with borrowed and transformed examples, improves upon the current state-of-the-art detector on the challenging SUN09 object detection dataset.


Dataset: Transfer Learning Challenge for Object Detection

There has been an increasing attention to learning with borrowing/sharing for the fewer examples class. Here, we introduce a new challenge on transfer learning for the detection. This dataset is based on the SUN 09, and it contains 4082 training and 9518 testing images. The categories include person, chair, car, truck, door, statue, and bag.


This shows number of training examples per classes. The five classes with the most number of examples are plant (1356), person (600), cabinet (551), chair (501), and door (394). The five classes with the least number of examples are cake (17), trash can (13), keyboard (12), cars (10), and cup (8).

Learning which examples to borrow

Our goal is to develop a novel framework that enables borrowing examples from related classes for a generic object detector. Our model learns which examples are most useful to borrow for a model training together with original examples.

Weight indicating how much examplar i to be borrowed for class c
Loss function (from a standard binary classification)
Group Lasso regularization; these two terms together will regularize weights so that examples from the same class to be similar. In other words, examples from the same class will be gently forced to be borrowed (or not borrowed) together.
More details can be found in the paper


We perform two kinds of experiments:
(1) Borrowing examples from other classes within the same dataset, and
(2) Borrowing examples from the same class that come from a different dataset.
Both experiments require identifying which examples are beneficial to borrow for the target class.

Borrowing examples from OTHER CLASSES within the SAME DATASET

(a) Shelves for Bookcase
(b) Chair for Swivelchair
Examples ranked by their learned weights
(a) shelves examples to be borrowed for bookcase, and (b) chair examples to be borrowed for swivel chair.
Green border indicates examples are borrowed, and red border indicates examples are not borrowed.

MethodsWithout BorrowingBorrowing ALL examples from the same classesOur Best Method
AP score

Borrowing examples from the SAME CLASS that come from a DIFFERENT DATASET

Combining datasets is a non-trivial task as different datasets contain different biases. Here, we show that by borrowing a subset of examples with our method outperforms naively merging positive examples from two datasets.

(a) From SUN09
(b) From PASCAL
(c) PASCAL cars ranked by their learned weights to be SUN09
(a) shows car examples from SUN09, and (b) shows car examples from PASCAL dataset. Finally, (c) shows PASCAL car examples sorted by learned borrowing weights for SUN09.
Green border indicates examples are borrowed, and red border indicates examples are not borrowed.

Training setSUN09 onlyPASCAL onlySUN09 + all PASCALSUN09 + borrowing PASCAL
(a)Testing on the SUN09 dataset

Training setPASCAL onlySUN09 onlyPASCAL + all SUN09PASCAL + borrowing SUN09
(b)Testing on the PASCAL dataset
"SUN09 only" and "PASCAL only" are trained using the SUN09 dataset and the PASCAL dataset without borrowing any examples, respectively. "SUN09+PASCAL" is trained using positive examples from both SUN09 and PASCAL, and negative examples from the target dataset. "PASCAL+borrowing SUN09" and "SUN09+borrowing PASCAL" borrow selected examples from another dataset for each target dataset using our method. It is shown that our borrowing method works quite better than merging positive examples naively.


This work is funded by ONR MURI N000141010933, CAREER Award No. 07471 20, NSERC, and NSF Graduate Research Fellowship.

Last update: Oct 15. 2011