"Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs"
ACM Transactions on Graphics, Volume 39, Issue 6, Article 258, December 2020
Abstract
We present deferred neural lighting, a novel method for free-viewpoint relighting from unstructured photographs of a scene captured with handheld devices. Our method leverages a scene-dependent neural rendering network for relighting a rough geometric proxy with learnable neural textures. Key to making the rendering network lighting aware are radiance cues: global illumination renderings of a rough proxy geometry of the scene for a small set of basis materials and lit by the target lighting. As such, the light transport through the scene is never explicitely modeled, but resolved at rendering time by a neural rendering network. We demonstrate that the neural textures and neural renderer can be trained end-to-end from unstructured photographs captured with a double hand-held camera setup that concurrently captures the scene while being lit by only one of the cameras’ flash lights. In addition, we propose a novel augmentation refinement strategy that exploits the linearity of light transport to extend the relighting capabilities of the neural rendering network to support other lighting types (e.g., environment lighting) beyond the lighting used during acquisition (i.e., flash lighting). We demonstrate our deferred neural lighting solution on a variety of real-world and synthetic scenes exhibiting a wide range of material properties, light transport effects, and geometrical complexity.
Download
Supplementary Material
Code
Bibtex
ACM Transactions on Graphics, Volume 39, Issue 6, Article 258, December 2020
AbstractWe present deferred neural lighting, a novel method for free-viewpoint relighting from unstructured photographs of a scene captured with handheld devices. Our method leverages a scene-dependent neural rendering network for relighting a rough geometric proxy with learnable neural textures. Key to making the rendering network lighting aware are radiance cues: global illumination renderings of a rough proxy geometry of the scene for a small set of basis materials and lit by the target lighting. As such, the light transport through the scene is never explicitely modeled, but resolved at rendering time by a neural rendering network. We demonstrate that the neural textures and neural renderer can be trained end-to-end from unstructured photographs captured with a double hand-held camera setup that concurrently captures the scene while being lit by only one of the cameras’ flash lights. In addition, we propose a novel augmentation refinement strategy that exploits the linearity of light transport to extend the relighting capabilities of the neural rendering network to support other lighting types (e.g., environment lighting) beyond the lighting used during acquisition (i.e., flash lighting). We demonstrate our deferred neural lighting solution on a variety of real-world and synthetic scenes exhibiting a wide range of material properties, light transport effects, and geometrical complexity.
Download
- Gao_SIGGRAPHAsia2020.pdf (16.59Mb)
Supplementary Material
- Video: Gao_SIGGRAPHAsia2020_video.mov (139.52Mb)
- Document: Gao_SIGGRAPHAsia2020_supplemental.pdf (5.89Mb)
- Video: vase_captured_seq.mov (18.66Mb)
Code
- Code Repository (github)
Bibtex
@article{Gao:2020:DNL,
author = {Gao, Duan and Chen, Guojun and Dong, Yue and Peers, Pieter and Xu, Kun and Tong, Xin},
title = {Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs},
month = {December},
year = {2020},
journal = {ACM Transactions on Graphics},
volume = {39},
number = {6},
article = {258},
doi = {https://doi.org/10.1145/3414685.3417767},
}
author = {Gao, Duan and Chen, Guojun and Dong, Yue and Peers, Pieter and Xu, Kun and Tong, Xin},
title = {Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs},
month = {December},
year = {2020},
journal = {ACM Transactions on Graphics},
volume = {39},
number = {6},
article = {258},
doi = {https://doi.org/10.1145/3414685.3417767},
}