Amazon Mechanical Turk for LabelMe

Want to outsource your labeling task to the internet? Amazon Mechanical Turk allows access to many internet users who are ready to perform tasks for a fixed price.

The idea is simple: you provide a task and a selling price. Internet workers perform the task and are subsequently paid. In Mechnical Turk terminology, tasks are called "HITs", people requesting work are called "Requesters", and people who do the work are called "Workers".

This page describes how to set up LabelMe annotation tasks onto Mechanical Turk. The process is simple, as we have provided scripts for creating and sending LabelMe annotation tasks onto Mechanical Turk. All you have to do is follow the instructions below and pay workers on Mechanical Turk to label images. We collect the annotations, which are immediately available for download. In this way, everybody wins: Mechanical Turk workers get paid, you get your images annotated, and the computer vision community gets access to more hand-labeled data.

Instructions for setting up LabelMe on Mechanical Turk

Setting up LabelMe on Mechanical Turk is easy. The following are instructions for setting up LabelMe on Mechanical Turk.

1. Upload images onto LabelMe

2. Set up an Amazon Mechanical Turk account

You will need to set up an account as a Requester on Mechanical Turk. Instructions for setting up an account are here. Once you have created an account, sign in and try to access your account, along with the sandbox (used for debugging).

3. Download and install Amazon Mechanical Turk Command Line Tools

You will need to install the Amazon Mechanical Turk Command Line Tools. The tools provide the backbone for communicating with the Mechanical Turk servers. To start, you first need to request your access key and secret key. This is different than your username and password. To do this, create an Amazon Web Services account. Once the account is created, go to "Your Account->Access Identifiers", which is located at the top of the page. Here, you will find your access key and secret key.

Next, download the Amazon Mechanical Turk Command Line Tools (note that so far, we have only tested the Linux/Mac version). Unzip the file and follow the instructions inside the directory to install the Command Line Tools.

As a reminder, open and modify ./aws-mturk-clt-1.3.0/bin/mturk.properties to include your access key and secret key. Also, make sure you set the following environment variables (e.g. using the command "export VAR=/path/to/file" in bash):

a) $MTURK_CMD_HOME - this should point to the location of your Amazon Mechanical Turk Command Line Tools (root directory).

b) $JAVA_HOME - this should point to the location of your Java installation.

4. Download scripts for sending LabelMe jobs to Mechanical Turk

Scripts for sending LabelMe jobs to Mechanical Turk are available here (currently, the scripts are available for Linux/Mac only). After download, unzip the file and change to the directory.

The zip file contains a set of scripts that are used to interact with Mechanical Turk, along with files used to set how the task is performed (e.g. how much do the workers earn, list of jobs, etc.). In addition, a Matlab function is included to assist in setting which images are to be annotated.

5. Submit jobs to Mechanical Turk

Let us now submit jobs to Mechanical Turk. In this section, we will use Mechanical Turk's "sandbox" server. The sandbox is free to use (you have fake credits on this server), so you can test here to make sure everything works. When you are ready for the real thing, simply remove all -sandbox flags and set sandbox=0 in the MATLAB script below. Make sure to add money to your Amazon account before running on the real server.

First, go to the directory containing the scripts for sending LabelMe jobs to Mechanical Turk (see Step 4 above; cd /path/to/LabelMeMechanicalTurk/). Next, start MATLAB and run the following at the prompt (use the value of FOLDER from Step 1 above):

folder=FOLDER;
sandbox=1; // Set this to sandbox=0 when running on the real server
generateLabelMeInputFile(folder,sandbox);

The script will produce a file called labelme.input with all of your images listed as HITs. The help output
(type help generateLabelMeInputFile at the MATLAB prompt) gives information on further customizations for the
task.

We are now ready to submit to Mechanical Turk. To submit to the sandbox, run the following at the Linux command
prompt (not the MATLAB prompt):

./run.sh -sandbox

You will see messages indicating that the jobs are being submitted. At the end, there will be a URL that points to a preview page for the HIT. You can go to that URL and try out the HIT. All of the collected annotations are stored on the LabelMe servers, so you can download them immediately or view thumbnails of the images (http://labelme.csail.mit.edu/browseLabelMe/FOLDER.html).

Mechanical Turk also produces outputs for the HITs, which you can retrieve from the server. To do this, run the following at the Linux command prompt:

./getResults.sh -sandbox

This will produce a file called labelme.results.xls. This is a comma separated file, which can be viewed with Excel. This file lists the status of each HIT, as well as outputs from the annotation task, such as number of annotations labeled, last object labeled, etc.

You can also view the status of HITs on the requester page of the Mechanical Turk server (or on the requester page of the sandbox server). Note that you can navigate to the status page from the main requester page via the "Manage HITs individually" link at the top right of the page.