Deep Multispectral Painting Reproduction via Multi-layer, Custom-Ink Printing

We present a spectral reproduction technique using a 3D printer. Our workflow targets accurate reproduction of paintings and provides faithful color reproductions under varying light sources. Above, we show three printed replicas of oil paintings with diferent image statistics, generated by our method, next to the original. On the right, we show cropped regions (three water lilies) from big water lily replica with the original under varying lighting sources. Paintings © Azadeh Asadi.

Abstract

We propose a worklow for spectral reproduction of paintings, which captures a painting's spectral color, invariant to illumination, and reproduces it using multi-material 3D printing. We take advantage of the current 3D printers' capabilities of combining highly concentrated inks with a large number of layers, to expand the spectral gamut of a set of inks. We use a data-driven method to both predict the spectrum of a printed ink stack and optimize for the stack layout that best matches a target spectrum. This bidirectional mapping is modeled using a pair of neural networks, which are optimized through a problem-speciic multi-objective loss function. Our loss function helps find the best possible ink layout resulting in the balance between spectral reproduction and colorimetric accuracy under a multitude of illuminants. In addition, we introduce a novel spectral vector error difusion algorithm based on combining color contoning and halftoning, which simultaneously solves the layout discretization and color quantization problems, accurately and eiciently. Our worklow outperforms the state-of-the-art models for spectral prediction and layout optimization. We demonstrate reproduction of a number of real paintings and historically important pigments using our prototype implementation that uses 10 custom inks with varying spectra and a resin-based 3D printer.

Downloads

• Paper (pdf, 54 MB)
• Supplementary Document (pdf, 0.6 MB)
• Multispectral Scans (zip, 1.3 GB)
• Ink Dataset (zip, 3.6 MB)
• Code (coming soon)

BibTeX

@article{Shi2018deep,
  author    = {Liang Shi and Vahid Babaei and Changil Kim and Michael Foshey and Yuanming Hu and Pitchaya Sitthi-amorn and Szymon Rusinkiewicz and Wojciech Matusik},
  title     = {Deep Multispectral Painting Reproduction via Multi-layer, Custom-Ink Printing},
  journal   = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia)},
  volume    = {37},
  number    = {6},
  year      = {2018},
}

Acknowledgments

We highly appreciate the help by David Kim for formulating our inks. We would like to thank Azadeh Asadi for painting all presented paintings, Todd Zickler and Ioannis Gkioulekas for spectral camera hardware and software, and Hossein Amirshahi for pointing to suitable pigments. We also thank our pigment providers: BASF, Lansco Colors, Penn Color, Sun Chemical and Toyo Ink Group. Vahid Babaei and Changil Kim are supported by the Swiss National Science Foundation (SNSF) fellowships P300P2 171212 and P2EZP2 168785, respectively. This work is partially supported by the U.S. National Science Foundation (NSF) grants IIS-1421435, CHS-1617236, IIS-1815070, and IIS-1815585.