Image statistics for surface reflectance perception

Lavanya Sharan¹      Isamu Motoyoshi²      Shin'ya Nishida²      Yuanzhen Li¹      Edward H. Adelson¹

¹ Massachusetts Institute of Technology      ² NTT Communication Science Labs

Figure 1: (Left panel) A smooth black paper can be made to look the same as a smooth white paper by adjusting the illumination, an effect first demonstrated by Gelb in 1929. When the mean luminance of the two papers discs is the same, they cannot be distinguished from each other. (Right panel) We demonstrated an Anti-Gelb effect (Sharan et al., 2008; Motoyoshi et al., 2007) by replacing the smooth surfaces used in the original Gelb demonstration with complex, real-world surfaces. Mean luminance of these two stucco-like surfaces is exactly the same. Yet, the black stucco-like surface on the right looks darker than the white one on the left.

We studied the visual perception of surface reflectance, specifically, the perception of albedo (a measure of diffuse reflectance) and surface gloss (a measure of specular reflectance). Previous work in surface perception had focused on smooth, Lambertian surfaces like those used in the Gelb demonstration (Figure 1, left panel). The perceived albedo, or lightness, of such smooth surfaces is mainly determined by the mean surface luminance (Gilchrist et al., 1999). We demonstrated that for isolated non-smooth, non-Lambertian surfaces, mean luminance is not sufficient to predict lightness perception (Figure 1, right panel). In our papers, we showed that higher moments of luminance like standard deviation and skewness, as well as percentile statistics like 10^th or 90^th percentile, can instead predict the perceived albedo and gloss for such surfaces.

This webpage summarizes our main findings. Please refer to the Nature paper, JOSA paper, or the thesis for details. This work was presented at VSS'05 (oral), VSS'06 (oral), VSS'07 (poster), and ECVP'07 (oral), and it was covered by CNET News, phys.org, and Nature.com blog. For a nice overview, check out this Nature News and Views article by Mike Landy.

Figure 2: Luminance histograms of light and dark materials exhibit systematic differences. The images of light modeling clay (top) and dark stucco (bottom) have been scaled so the mean image luminances are equal. Note how the histogram of the darker material is wider (higher standard deviation) and more asymmetric (more skewed) than the histogram of the lighter material.

A key observation is that the luminance distributions of light (high albedo) and matte (more diffuse reflection) surfaces are systematically different from those of dark (low albedo) and glossy (more specular reflection) surfaces. After normalizing for mean luminance, histograms for darker and glossier materials tend to be wider and have longer positive tails. These differences in the shapes of histograms, first noted by Nishida & Shinya (1998), can be attributed to physical factors - interreflections and specular highlights. Darker surfaces have fewer interreflections, leading to higher contrast. Dark and glossy surfaces have more prominent specular highlights, leading to heavier positive tails. Similar trends are observed when examining the outputs of derivative or center-surround filters rather than raw luminance values. Moment statistics like standard deviation and skewness, as well as percentile statistics can be used to quantify these systematic differences. Therefore, these image statistics are diagnostic of physical reflectance properties.

It is worth noting that this relationship between image statistics and surface reflectance only holds under certain conditions. The statistics have computed over image regions spanned by a single opaque surface with spatially homogenous reflectance properties. Moreover, each surface has to be uniformly illuminated and imaged in the fronto-parallel plane. These restrictions are not unique to our work; similar experimental conditions have been used in previous studies of reflectance perception (Gilchrist et al., 1999). If anything, our work extends the study of reflectance perception to include non-Lambertian, non-smooth real-world surfaces.

Figure 3: (Left panel) Examples of surfaces in our data set. (Right panel) Observers were shown mean-luminance normalized photographs of surfaces, one at a time, on a monitor and asked to rate the albedo in reference to a physical scale. Observers' ratings are plotted against the physical albedo. If observers were perfect, the data would lie on the black diagonal. If observers were unable to do the task, as per the Gelb prediction, the data would lie along a horizontal line. Slope of best fit lines (green) are shown in each plot (p<0.05); error bars are 1 s.e.m.

Our "anti-Gelb" effect (Figure 1, right panel) demonstrates that human observers can estimate reflectance properties of isolated surfaces in the absence of mean luminance information. We conducted psychophysical experiments to measure exactly how good observers are at estimating reflectance in Gelb conditions. We displayed mean luminance-normalized photographs of surfaces, like the ones shown in Figure 3 (left panel), and asked observers to judge the surface albedo and gloss in reference to physical scales. As shown in Figure 3 (right panel), we found that observers performed reasonably well. While they were not perfect at estimating physical reflectance properties, their performance was far from predicted Gelb behavior.

Surprisingly, we found that the moment and percentile statistics that were diagnostic of physical reflectance properties were also predictive of perceived reflectance properties. As shown in Figure 4 (left panel), the output of a linear regression model based on these image statistics was highly correlated with observers' reflectance judgments. Is it possible then that these diagnostic image statistics are used by the human visual system to estimate reflectance properties? If so, how might the human brain compute such statistics?

For the skewness statistic, we offered an explanation in terms of ON and OFF center-surround filtering mechanisms that are well known in the early visual pathway. To test for the existence of skewness detection mechanisms in the brain, we developed a novel visual aftereffect, the skewness aftereffect (Figure 4, right panel). After adapting to a positively skewed stimulus (e.g., white dots on a black background) or a negatively skewed stimulus (e.g., black dots on a white background), the reflectance of surfaces viewed subsequently was altered in the expected direction. Taken together, our findings suggest that some simple image statistics can have a strong influence on judgments of surface reflectance.

Figure 4: (Left panel) Ratings of perceived albedo for a representative observer are plotted against the output of a linear regression model that employs moment and percentile statistics of luminance and filter output histograms. The r2 value (0.78) measures the agreement between statistics and perception. (Right panel) A demo of the skewness aftereffect courtesy Isamu Motoyoshi. Keep staring at the tiny white dot in the middle and pay attention to the last frame of the movie. Which surface looks darker? Glossier? Both surfaces are actually identical, so if you see them as different it is due to the adaptation! For more movies, go here.

To cite our work, please use:

L. Sharan, Y. Li, I. Motoyoshi, S. Nishida, and E. H. Adelson, "Image statistics for surface reflectance perception", Journal of the Optical Society of America A, 25(4), pp. 846-865, 2008

I. Motoyoshi, S. Nishida, L. Sharan, and E. H. Adelson, "Image statistics and the perception of surface qualities", Nature, vol. 447, pp. 206-209, 2007

This research was supported by grants from the National Science Foundation (0345805) and the NTT-MIT Research Collaboration. We thank anonymous reviewers for their insightful comments, and Roland W. Fleming and Ruth Rosenholtz for helpful discussions.