Objective Analysis

We compare between objective measures and the collected user data by correlating their rankings. Kendall tau distance (Equation 3 in the paper) is used to measure the degree of correlation between the rankings induced by the two measures. This page shows results of this analysis.

Analysis images

We examine the bias of each metric by simulating a voting process based on the calculated distances. For each operator, we count the number of times its result is smaller than another result, and accumulate over all images and metrics. The following figure shows the distribution of metric “votes” among the operators for the different metrics we tested.

The following tables show the correlation of objective and subjective measures for the full ranking, and with respect to the k top-ranked results, for k=2,3,4. For each k we also show the pairwise metric correlations (measured by correlating the rankings produced by each pair of metrics for each image).

Full ranking (k=inf)

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
0.040
0.190
0.060
0.167
-0.004
-0.012
0.083
0.268
0.017
BDS-PM
0.054
0.162
0.083
0.167
0.063
-0.024
0.097
0.232
0.013
BDW
0.031
0.048
-0.048
0.060
0.004
0.119
0.046
0.181
0.869
SIFTflow
0.097
0.252
0.119
0.218
0.085
0.071
0.145
0.262
0.031
EMD
0.220
0.262
0.107
0.226
0.237
0.500
0.251
0.272
0.000
EH
0.043
-0.076
-0.060
-0.079
0.103
0.298
0.004
0.334
0.641
CL
-0.023
-0.181
-0.071
-0.183
-0.009
0.214
-0.068
0.301
0.384

Correlation with users

In each column the mean correlation coefficient is shown, calculated over all images in the dataset with the corresponding attribute. The last three columns show the mean score, standard deviation, and respective p-value over all image types.

 

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.77
0.48
0.68
0.14
-0.01
-0.33
BDS-PM
0.77
1.00
0.53
0.77
0.15
0.01
-0.37
BDW
0.48
0.53
1.00
0.45
0.29
0.15
0.04
SIFTflow
0.68
0.77
0.45
1.00
0.21
-0.03
-0.36
EMD
0.14
0.15
0.29
0.21
1.00
0.24
0.31
EH
-0.01
0.01
0.15
-0.03
0.24
1.00
0.31
CL
-0.33
-0.37
0.04
-0.36
0.31
0.31
1.00

Correlation between metrics

Each cell indicates the mean rank coefficient between metric i and metric j over all images in the dataset.

k=2

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
-0.240
0.286
-0.208
0.155
-0.375
-0.500
-0.134
0.827
0.000
BDS-PM
-0.120
0.279
-0.208
0.233
-0.078
-0.208
-0.008
0.800
0.000
BDW
0.184
0.073
0.042
0.047
0.053
0.750
0.147
0.735
0.000
SIFTflow
-0.126
0.386
-0.542
0.355
0.037
-0.151
0.078
0.831
0.000
EMD
0.220
0.369
0.000
0.155
0.078
0.433
0.224
0.726
0.000
EH
-0.150
-0.467
-0.458
-0.514
0.016
-0.167
-0.236
0.810
0.000
CL
-0.676
-0.794
-0.792
-0.861
-0.688
-0.234
-0.673
0.593
0.000

Correlation with users

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.83
0.11
0.64
0.20
-0.44
-0.86
BDS-PM
0.83
1.00
0.17
0.71
0.18
-0.40
-0.86
BDW
0.11
0.17
1.00
-0.09
0.51
-0.18
-0.26
SIFTflow
0.64
0.71
-0.09
1.00
0.25
-0.36
-0.97
EMD
0.20
0.18
0.51
0.25
1.00
-0.01
-0.25
EH
-0.44
-0.40
-0.18
-0.36
-0.01
1.00
0.01
CL
-0.86
-0.86
-0.26
-0.97
-0.25
0.01
1.00

Correlation between metrics

k=3

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
0.062
0.280
0.134
0.249
-0.025
-0.247
0.108
0.532
0.005
BDS-PM
0.165
0.299
0.273
0.350
0.221
-0.052
0.210
0.464
0.000
BDW
0.213
0.141
0.123
0.115
0.212
0.439
0.200
0.395
0.002
SIFTflow
0.241
0.428
0.312
0.442
0.303
0.002
0.298
0.483
0.000
EMD
0.301
0.416
0.216
0.295
0.226
0.534
0.326
0.496
0.000
EH
-0.036
-0.207
-0.331
-0.177
0.111
0.294
-0.071
0.593
0.013
CL
-0.307
-0.336
-0.433
-0.519
-0.366
0.088
-0.320
0.543
0.000

Correlation with users

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.80
0.38
0.69
0.27
-0.21
-0.64
BDS-PM
0.80
1.00
0.35
0.76
0.24
-0.25
-0.76
BDW
0.38
0.35
1.00
0.25
0.42
0.14
-0.03
SIFTflow
0.69
0.76
0.25
1.00
0.30
-0.24
-0.76
EMD
0.27
0.24
0.42
0.30
1.00
0.12
0.09
EH
-0.21
-0.25
0.14
-0.24
0.12
1.00
0.27
CL
-0.64
-0.76
-0.03
-0.76
0.09
0.27
1.00

Correlation between metrics

k=4

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
0.063
0.236
0.122
0.146
-0.010
-0.057
0.097
0.392
0.052
BDS-PM
0.156
0.278
0.215
0.262
0.148
0.018
0.187
0.338
0.003
BDW
0.077
0.084
0.005
0.082
0.030
0.164
0.092
0.287
0.102
SIFTflow
0.218
0.416
0.354
0.360
0.214
0.148
0.272
0.376
0.000
EMD
0.299
0.429
0.154
0.297
0.243
0.676
0.338
0.436
0.000
EH
0.031
-0.014
-0.049
-0.030
0.108
0.349
0.021
0.401
0.003
CL
-0.112
-0.230
-0.137
-0.277
-0.158
0.155
-0.155
0.394
0.000

Correlation with users

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.79
0.47
0.68
0.18
-0.11
-0.47
BDS-PM
0.79
1.00
0.45
0.79
0.18
-0.11
-0.59
BDW
0.47
0.45
1.00
0.37
0.30
0.13
-0.07
SIFTflow
0.68
0.79
0.37
1.00
0.24
-0.13
-0.57
EMD
0.18
0.18
0.30
0.24
1.00
0.22
0.23
EH
-0.11
-0.11
0.13
-0.13
0.22
1.00
0.30
CL
-0.47
-0.59
-0.07
-0.57
0.23
0.30
1.00

Correlation between metrics

All user study images

When considering all images, we normalize by the number of times each result was chosen by the number of times it was shown. This number might vary between images with different number of results as different designs are used.

Full ranking (k=inf)

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
0.069
0.204
0.080
0.164
-0.018
0.106
0.081
0.305
0.056
BDS-PM
0.048
0.157
0.021
0.135
0.030
0.041
0.069
0.265
0.038
BDW
0.045
0.089
-0.077
0.069
0.002
0.088
0.056
0.197
0.191
SIFTflow
0.111
0.249
0.071
0.214
0.080
0.167
0.124
0.317
0.078
EMD
0.269
0.341
0.098
0.222
0.278
0.462
0.278
0.325
0.000
EH
0.000
-0.108
-0.015
-0.112
0.012
0.095
-0.031
0.339
0.477
CL
-0.035
-0.170
-0.083
-0.219
-0.034
-0.003
-0.053
0.342
0.560

Correlation with users

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.78
0.52
0.70
0.16
-0.06
-0.33
BDS-PM
0.78
1.00
0.57
0.76
0.17
-0.06
-0.36
BDW
0.52
0.57
1.00
0.48
0.29
0.07
0.00
SIFTflow
0.70
0.76
0.48
1.00
0.20
-0.08
-0.36
EMD
0.16
0.17
0.29
0.20
1.00
0.18
0.30
EH
-0.06
-0.06
0.07
-0.08
0.18
1.00
0.36
CL
-0.33
-0.36
0.00
-0.36
0.30
0.36
1.00

Correlation between metrics

k=2

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
-0.053
0.322
0.107
0.202
-0.225
-0.071
-0.023
0.794
0.000
BDS-PM
-0.017
0.288
0.107
0.249
-0.031
0.104
0.005
0.777
0.000
BDW
0.180
0.162
-0.071
0.064
0.140
0.536
0.133
0.737
0.000
SIFTflow
-0.023
0.311
-0.232
0.253
0.024
0.206
0.034
0.838
0.000
EMD
0.250
0.384
-0.023
0.143
0.152
0.384
0.224
0.719
0.000
EH
-0.226
-0.533
-0.259
-0.528
-0.107
-0.357
-0.300
0.783
0.000
CL
-0.613
-0.738
-0.411
-0.854
-0.585
-0.369
-0.616
0.634
0.000

Correlation with users

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.85
0.24
0.65
0.25
-0.52
-0.80
BDS-PM
0.85
1.00
0.28
0.67
0.19
-0.48
-0.88
BDW
0.24
0.28
1.00
0.08
0.51
-0.35
-0.33
SIFTflow
0.65
0.67
0.08
1.00
0.24
-0.45
-0.90
EMD
0.25
0.19
0.51
0.24
1.00
-0.14
-0.22
EH
-0.52
-0.48
-0.35
-0.45
-0.14
1.00
0.10
CL
-0.80
-0.88
-0.33
-0.90
-0.22
0.10
1.00

Correlation between metrics

k=3

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
0.079
0.316
0.026
0.205
-0.050
-0.015
0.093
0.562
0.000
BDS-PM
0.093
0.288
0.117
0.290
0.086
0.094
0.139
0.520
0.000
BDW
0.180
0.169
-0.023
0.102
0.159
0.319
0.172
0.411
0.003
SIFTflow
0.197
0.395
0.171
0.383
0.195
0.188
0.220
0.527
0.000
EMD
0.305
0.489
0.183
0.301
0.236
0.497
0.328
0.444
0.000
EH
-0.098
-0.221
-0.223
-0.173
-0.030
0.043
-0.127
0.567
0.006
CL
-0.281
-0.368
-0.225
-0.542
-0.331
-0.162
-0.294
0.551
0.000

Correlation with users

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.81
0.46
0.71
0.28
-0.25
-0.63
BDS-PM
0.81
1.00
0.44
0.76
0.25
-0.29
-0.72
BDW
0.46
0.44
1.00
0.36
0.41
0.03
-0.09
SIFTflow
0.71
0.76
0.36
1.00
0.27
-0.27
-0.73
EMD
0.28
0.25
0.41
0.27
1.00
0.03
0.10
EH
-0.25
-0.29
0.03
-0.27
0.03
1.00
0.32
CL
-0.63
-0.72
-0.09
-0.73
0.10
0.32
1.00

Correlation between metrics

k=4

Metric
Attribute
Total
Lines/Edges
Faces/People
Texture
Foreground Objects
Geometric Structures
Symmetry
Mean
Std
p-value
BDS
0.077
0.255
0.084
0.168
-0.028
0.087
0.099
0.399
0.000
BDS-PM
0.114
0.271
0.100
0.242
0.074
0.064
0.143
0.364
0.000
BDW
0.066
0.119
-0.051
0.090
0.005
0.111
0.087
0.282
0.695
SIFTflow
0.206
0.384
0.246
0.342
0.168
0.255
0.224
0.416
0.000
EMD
0.330
0.471
0.172
0.305
0.294
0.600
0.349
0.416
0.000
EH
-0.015
-0.066
0.043
-0.066
0.002
0.139
-0.033
0.407
0.001
CL
-0.118
-0.228
-0.104
-0.319
-0.147
-0.050
-0.139
0.438
0.002

Correlation with users

Metric
BDS
BDS-PM
BDW
SIFTflow
EMD
EH
CL
BDS
1.00
0.80
0.51
0.71
0.20
-0.16
-0.49
BDS-PM
0.80
1.00
0.52
0.77
0.20
-0.17
-0.57
BDW
0.51
0.52
1.00
0.43
0.32
0.03
-0.11
SIFTflow
0.71
0.77
0.43
1.00
0.23
-0.18
-0.56
EMD
0.20
0.20
0.32
0.23
1.00
0.13
0.22
EH
-0.16
-0.17
0.03
-0.18
0.13
1.00
0.34
CL
-0.49
-0.57
-0.11
-0.56
0.22
0.34
1.00

Correlation between metrics