"Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks"
Xiao Li, Yue Dong, Pieter Peers, and Xin Tong

ACM Transactions on Graphics, Volume 36, Issue 4, Article 45, July 2017
Abstract
We present a convolutional neural network (CNN) based solution for modeling physically plausible spatially varying surface reflectance functions (SVBRDF) from a single photograph of a planar material sample under unknown natural illumination. Gathering a sufficiently large set of labeled training pairs consisting of photographs of SVBRDF samples and corresponding reflectance parameters, is a difficult and arduous process. To reduce the amount of required labeled training data, we propose to leverage the appearance information embedded in unlabeled images of spatially varying materials to self-augment the training process. Starting from an initial approximative network obtained from a small set of labeled training pairs, we estimate provisional model parameters for each unlabeled training exemplar. Given this provisional reflectance estimate, we then synthesize a novel temporary labeled training pair by rendering the exact corresponding image under a new lighting condition. After refining the network using these additional training samples, we re-estimate the provisional model parameters for the unlabeled data and repeat the self-augmentation process until convergence. We demonstrate the efficacy of the proposed network structure on spatially varying wood, metal, and plastics, as well as thoroughly validate the effectiveness of the self-augmentation training process.


Download
Supplementary Material
Code
Additional Notes
  • Errata: The description of the network structure was incorrect in the published paper (v1). A revised PDF (v2) now includes a correct description that matches the implementation in the code repository. The original version is still available for reference.

Bibtex
@article{Li:2017:MSA,
author = {Li, Xiao and Dong, Yue and Peers, Pieter and Tong, Xin},
title = {Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks},
month = {July},
year = {2017},
journal = {ACM Transactions on Graphics},
volume = {36},
number = {4},
article = {45},
doi = {http://dx.doi.org/10.1145/3072959.3073641},
}