NeRFactor: Neural Factorization of Shape and Reflectance
Under an Unknown Illumination

arXiv

Xiuming Zhang 1   Pratul P. Srinivasan 2   Boyang Deng 2   Paul Debevec 2   William T. Freeman 1, 2   Jonathan T. Barron 2

1 MIT CSAIL 2 Google Research

Abstract

We address the problem of recovering the shape and spatially-varying reflectance of an object from posed multi-view images of the object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and the environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks.

Given a set of posed images of an object captured from multiple views under just one unknown illumination condition (left), Neural Radiance Factorization (NeRFactor) is able to factorize the scene into 3D neural fields of surface normals, light visibility, albedo, and material (center), which enables applications such as free-viewpoint relighting and material editing (right).

Paper

NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, Jonathan T. Barron
arXiv
arXiv /  BibTeX

v0 (06/01/2020)

Video

Downloads:  1080p (212 MB)  /   720p (154 MB)

v0 (06/01/2020)

Code

This GitHub repository includes code for:

Downloads

All subsequent folders and files are specified w.r.t. this Google Drive root.

If you want to use our rendered/processed data, see "Data." If you want to render your own data, also see "Metadata."

Data

We release the images rendered from the synthetic scenes above and the real images (vasedeck and pinecone from NeRF) processed to be compatible with our model's data format:

Rendered Images Real Images
./rendered-images/ ./real-images/

Metadata

We release the four .blend scenes (modified from what NeRF released): lego, hotdog, ficus, and drums, the training/testing light probes used in the paper, and the training/validation/testing cameras (exactly the same as what NeRF released):

Scenes Light Probes Cameras
./blender-scenes.zip ./light-probes.zip ./cameras.zip

All Video Results

We release the video results of NeRFactor, its material editing, the NeRFactor variants used in our ablation studies, and Oxholm & Nishino [2014].

NeRFactor

Here are our results on all four synthetic scenes and two real captures:

lego_3072 hotdog_2163 drums_3072
ficus_2188 vasedeck pinecone

Material Editing via NeRFactor

Here are our material editing results on all four synthetic scenes and two real captures:

lego_3072 hotdog_2163 drums_3072
ficus_2188 vasedeck pinecone

Ablation Studies

Here are the NeRFactor variants' results on all four synthetic scenes.

Directly Using NeRF's Shape

This model variant fixes the shape to NeRF's, optimizing only the reflectance and illumination:

lego_3072 hotdog_2163 drums_3072 ficus_2188

No Smoothness Regularization

This model variant uses no smoothness regularization:

lego_3072 hotdog_2163 drums_3072 ficus_2188

Using Microfacet BRDFs

This model variant uses microfacet BRDFs instead of the learned BRDFs:

lego_3072 hotdog_2163 drums_3072 ficus_2188

No Geometry Pretraining

This model variant trains the surface normal and light visibility MLPs from scratch, together with the reflectance and lighting:

lego_3072 hotdog_2163 drums_3072 ficus_2188

Oxholm & Nishino [2014]

Here is how NeRFactor compares with an enhanced version of Oxholm & Nishino [2014] (implemented by ourselves due to the lack of the source code) on all four synthetic scenes:

lego_3072 hotdog_2163 drums_3072 ficus_2188
†Both this enhanced version and the original model require ground-truth lighting, which we provided.