"Single Image Neural Material Relighting"
ACM SIGGRAPH 2023 Conference Proceedings, Article 80, August 2023
Abstract
This paper presents a novel neural material relighting method for revisualizing a photograph of a planar spatially-varying material under novel viewing and lighting conditions. Our approach is motivated by the observation that the plausibility of a spatially varying material is judged purely on the visual appearance, not on the underlying distribution of appearance parameters. Therefore, instead of using an intermediate parametric representation (e.g., SVBRDF) that requires a rendering stage to visualize the spatially-varying material for novel viewing and lighting conditions, neural material relighting directly generates the target visual appearance. We explore and evaluate two different use cases where the relit results are either used directly, or where the relit images are used to enhance the input in existing multi-image spatially varying reflectance estimation methods. We demonstrate the robustness and efficacy for both use cases on a wide variety of spatially varying materials.
Download
Supplementary Material
Code
Bibtex
ACM SIGGRAPH 2023 Conference Proceedings, Article 80, August 2023
AbstractThis paper presents a novel neural material relighting method for revisualizing a photograph of a planar spatially-varying material under novel viewing and lighting conditions. Our approach is motivated by the observation that the plausibility of a spatially varying material is judged purely on the visual appearance, not on the underlying distribution of appearance parameters. Therefore, instead of using an intermediate parametric representation (e.g., SVBRDF) that requires a rendering stage to visualize the spatially-varying material for novel viewing and lighting conditions, neural material relighting directly generates the target visual appearance. We explore and evaluate two different use cases where the relit results are either used directly, or where the relit images are used to enhance the input in existing multi-image spatially varying reflectance estimation methods. We demonstrate the robustness and efficacy for both use cases on a wide variety of spatially varying materials.
Download
- Bieron_SIGGRAPH2023.pdf (18.33Mb)
Supplementary Material
- Video: Bieron_SIGGRAPH2023_video.mp4 (143.45Mb)
Code
- Code Archive: Bieron_SIGGRAPH2023_code.zip (396.14Mb)
Bibtex
@conference{Bieron:2023:SIN,
author = {Bieron, James and Tong, Xin and Peers, Pieter},
title = {Single Image Neural Material Relighting},
month = {August},
year = {2023},
articleno = {80},
booktitle = {ACM SIGGRAPH 2023 Conference Proceedings},
doi = {https://doi.org/10.1145/3588432.3591515},
}
author = {Bieron, James and Tong, Xin and Peers, Pieter},
title = {Single Image Neural Material Relighting},
month = {August},
year = {2023},
articleno = {80},
booktitle = {ACM SIGGRAPH 2023 Conference Proceedings},
doi = {https://doi.org/10.1145/3588432.3591515},
}