3DTV at Home: Eulerian-Lagrangian Stereo-to-Multiview Conversion, SIGGRAPH 2017


3DTV at Home: Eulerian-Lagrangian Stereo-to-Multiview Conversion

Petr Kellnhofer1,2     Piotr Didyk2,3     Szu-Po Wang1     Pitchaya Sitthi-Amorn4     William Freeman1     Fredo Durand1     Wojciech Matusik1

1 MIT CSAIL     2 MPI Informatik     3 Saarland University, MMCI     4 Chulangkorn University

Stereoscopic views of multiview content generated from stereoscopic content using our method and two previous approaches. Monocular insets highlight limitations of the Lagrangian approach when dealing with fuzzy depth edges and ringing artifacts caused by exhaustive input disparities in the case of the Eulerian approach. Our method successfully avoids such problems and produces outputs visually closest to the original input. "Scene copyright: Blender Foundation (https://orange.blender.org/)

Abstract

Stereoscopic 3D (S3D) movies have become widely popular in the movie theaters, but the adoption of S3D at home is low even though most TV sets support S3D. It is widely believed that S3D with glasses is not the right approach for the home. A much more appealing approach is to use automultiscopic displays that provide a glasses-free 3D experience to multiple viewers. A technical challenge is the lack of native multiview content that is required to deliver a proper view of the scene for every viewpoint. Our approach takes advantage of the abundance of stereoscopic 3D movies. We propose a real-time system that can convert stereoscopic video to a high-quality, multiview video that can be directly fed to automultiscopic displays. Our algorithm uses a wavelet-based decomposition of stereoscopic images with per-wavelet disparity estimation. A key to our solution lies in combining Lagrangian and Eulerian approaches for both the disparity estimation and novel view synthesis, which leverages the complementary advantages of both techniques. The solution preserves all the features of Eulerian methods, e.g. subpixel accuracy, high performance, robustness to ambiguous depth cases, and easy integration of inter-view aliasing while maintaining the advantages of Lagrangian approaches, e.g. robustness to large disparities and possibility of performing non-trivial disparity manipulations through both view extrapolation and interpolation. The method achieves real-time performance on current GPUs. Its design also enables an easy hardware implementation that is demonstrated using a field-programmable gate array. We analyze the visual quality and robustness of our technique on a number of synthetic and real-world examples. We also perform a user experiment which demonstrates benefits of the technique when compared to existing solutions.

Video


See also Stereo bonus (side-by-side)

Downloads

Paper (Full Author's Copy) (16.0 MiB) Official online version: ACM Digital Library
Main submission video (110 MiB) and stereoscopic bonus (37.3 MiB) (side-by-side Left-Right encoding)

Citation

Petr Kellnhofer, Piotr Didyk, Szu-Po Wang, Pitchaya Sitthi-Amorn, William Freeman, Fredo Durand, Wojciech Matusik
3DTV at Home: Eulerian-Lagrangian Stereo-to-Multiview Conversion
ACM Transactions on Graphics 36(4) (Proc. SIGGRAPH 2017, Los Angeles, California, USA)

@article{Kellnhofer2017,
	  author = { 
		Petr Kellnhofer and
		Piotr Didyk and 
		Szu-Po Wang and 
		Pitchaya Sitthi-Amorn and 
		William Freeman and 
		Fredo Durand and
		Wojciech Matusik},
	  title = {{3DTV} at Home: Eulerian-Lagrangian Stereo-to-Multiview Conversion},
	  journal = {ACM Transactions on Graphics (Proc. SIGGRAPH)},
	  year = {2017},
	  volume = {36},
	  number = {4},
	  doi = {10.1145/3072959.3073617},
	  url = {http://dx.doi.org/10.1145/3072959.3073617}
	}

© 2017 The Authors. This is the author's version of the work. It is posted here for your personal use. Not for redistribution.
The definitive version was published in ACM Transactions on Graphics 36(4) (Proc. SIGGRAPH 2017). http://dx.doi.org/10.1145/3072959.3073617

MIT CSAIL MPI Informatik Saarland University, MMCI Chulalongkorn University