Performance Statistics

This page details some statistics on the time it took users to complete their questionnaire, and perform comparisons.

• Session time
• Comparison time
• Comparison time by image
• Question time

Total session time

The start and end time for each session is calculated from the the moment the user starts to enter his/her details, until he/she reaches the last page.

Upper 5 percent of the timings were filtered in the following figures (users might have paused during the survey).

Reference mean = 20.6583, std = 8.6485	No-reference mean = 12.7793, std = 5.4734
Volunteers mean = 21.7956, std = 9.4849	MT Workers mean = 13.8229, std = 6.0898

Comparison time

Another interesting question is whether users spent more time on specific comparisons, as well as how efficient our interface is. For this, we also measured the time it took the user to perform each comparison. To be as accurate as possible, we measured the time on the client machine, thus eliminating any performance issues on the range between the server and the user. A timer was started once the page was fully rendered, and stopped once the user submitted the selection. The following figures show the mean (blue) and 25th percentile (lower red curve) to 75th percentile (upper red curve) range for the two survey versions over all users. Notice the fast convergence, which indicates a moderate learning curve for using our interface.

Reference	No-reference

Comparison time by image

Mean comparison time is taken over all comparisons of each image. The plot is generated for percentiles 5-95 of the data. No special trend it apparent. Images and identifiers can be found here

Question time

Distribution of the time (in seconds) it took the users to answer the extra question (when given).