Background and objectives: There is much still and moving image content at a low resolution. Much current NTSC programming may be desired to play on future high definition television (HDTV) players. By the time HDTV sets are commonplace, consumers may have come to expect that level of resolution quality. (Just as consumers expect to see color images now, instead of the old black and white ones).

Technical discussion: We use a training based approach. We examine many pairs of high resolution, and low resolution versions of the same image data. We divide each image into patches, both high resolution and low-resolution patches. We describe the patches as vectors in a continuous space, and model the probability densities as gaussian mixtures. (We reduced the dimensionality of the scene and image data within each patch by principal components analysis). We had approximately 20,000 patch samples from our training data, and typically used 9 dimensional representations for both the low-resolution patches (7x7 pixels) and the high resolution patches (3x3 pixels).

Each patch of the low and high resolution images is a node in a Markov network. Given some new image, we seek to infer the corresponding high resolution image components. During inference, we evaluate the prior and conditional distributions of the high resolution data, given the low resolution observation. The high resolution components are a sampling of those high resolution components which correspond to the observed low resolution components at that node. We think of it as a "lineup of suspects". Each node has its own set of suspects. Each scene in a node's lineup has in common the fact that it renders to the low-resolution observation at that node. We evaluate the likelihoods by a set of belief propagation equations. The computation converges in just 3 iterations. The iterations themselves take about 5 seconds each. However, the set-up time prior to beginning the computation takes about 1 hour. We hope to reduce that time with future research.

MERL technical report:
Example-based super-resolution (a summary of our work in this area)
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor.

TR2000-05
Learning low-level vision
(longer journal version, Intl. Journal of Computer Vision, 40(1), pp. 25-47, 2000) William T. Freeman, Egon C. Pasztor , and Owen T. Carmichael

TR99-12
Learning low-level vision
(conference version, Intl. Conf. on Computer Vision, Corfu, Greece, 1999) William T. Freeman, Egon C. Pasztor

TR99-08
Markov networks for low-level vision
William T. Freeman, Egon C. Pasztor