Differentiable Monte Carlo Ray Tracing through Edge Sampling

 Tzu-Mao Li   Miika Aittala   Frédo Durand   Jaakko Lehtinen 
MIT CSAIL MIT CSAIL MIT CSAIL Aalto University
NVIDIA

Code available!



Abstract

Gradient-based methods are becoming increasingly important for computer graphics, machine learning, and computer vision. The ability to compute gradients is crucial to optimization, inverse problems, and deep learning. In rendering, the gradient is required with respect to variables such as camera parameters, light sources, scene geometry, or material appearance. However, computing the gradient of rendering is challenging because the rendering integral includes visibility terms that are not differentiable. Previous work on differentiable rendering has focused on approximate solutions. They often do not handle secondary effects such as shadows or global illumination, or they do not provide the gradient with respect to variables other than pixel coordinates.

We introduce a general-purpose differentiable ray tracer, which, to our knowledge, is the first comprehensive solution that is able to compute derivatives of scalar functions over a rendered image with respect to arbitrary scene parameters such as camera pose, scene geometry, materials, and lighting parameters. The key to our method is a novel edge sampling algorithm that directly samples the Dirac delta functions introduced by the derivatives of the discontinuous integrand. We also develop efficient importance sampling methods based on spatial hierarchies. Our method can generate gradients in times running from seconds to minutes depending on scene complexity and desired precision.

We interface our differentiable ray tracer with the deep learning library PyTorch and show prototype applications in inverse rendering and the generation of adversarial examples for neural networks.


Publication

Tzu-Mao Li, Miika Aittala, Frédo Durand, Jaakko Lehtinen.
Differentiable Monte Carlo Ray Tracing through Edge Sampling.
ACM Transactions on Graphics 37(6) (Proceedings of ACM SIGGRAPH Asia 2018)
BibTeX


Errata

See here for errata of an equation in the paper.


Downloads

SIGGRAPH Asia 2018 paper
supplementary webpage
Code
SIGGRAPH Asia 2018 presentation slides (Keynote)
SIGGRAPH Asia 2018 presentation slides (Powerpoint)


Acknowledgement

We thank the anonymous reviewers for their detailed comments (especially reviewer #1). The work started as an internship project at NVIDIA, where Marco Salvi and Aaron Lefohn provided immensely helpful advice. Luke Anderson and Prafull Sharma helped proofread the drafts. Nathaniel Jones modelled the conference scene in Figure 1 and gave helpful comments on the applications of inverse rendering for architectural design. The teapot model in Figure 5 was modelled by Martin Newell and the bunny in the same figure was created by Brian Curless and Marc Levoy. Both of them were downloaded from Morgan McGuire's website. The living room scene in Figure 6(f) was modelled by Wig42 and ported to Mitsuba scene format by Benedikt Bitterli. The stop sign in Figure 10 was modelled by Elijah Rai, and the street in the same figure was modelled by Pabong. Both of them were downloaded from Free3D.com. The work is funded by Toyota Research Institute.