De-rendering the World's Revolutionary Artefacts
In CVPR 2021


Shangzhe Wu1,4*
Ameesh Makadia4
Jiajun Wu2
Noah Snavely4
Richard Tucker4
Angjoo Kanazawa3,4


University of Oxford
Stanford University
University of California, Berkeley
Google Research




Teaser figure.

Given only a real single-view image collection of "revolutionary" (i.e., solid of revolution) artefacts with known silhouettes as training data (left), our framework learns to de-render a single image into shape, albedo and complex lighting and material components, allowing for novel-view synthesis and relighting (right).

* This work was primarily done while Shangzhe Wu was interning at Google Research.




Abstract

Recent works have shown exciting results in unsupervised image de-rendering -- learning to decompose 3D shape, appearance, and lighting from single-image collections without explicit supervision. However, many of these assume simplistic material and lighting models. We propose a method, termed RADAR (Revolutionary Artefact De-rendering And Re-rendering), that can recover environment illumination and surface materials from real single-image collections, relying neither on explicit 3D supervision, nor on multi-view or multi-light images. Specifically, we focus on rotationally symmetric artefacts that exhibit challenging surface properties including specular reflections, such as vases. We introduce a novel self-supervised albedo discriminator, which allows the model to recover plausible albedo without requiring any ground-truth during training. In conjunction with a shape reconstruction module exploiting rotational symmetry, we present an end-to-end learning framework that is able to de-render the world's revolutionary artefacts. We conduct experiments on a real vase dataset and demonstrate compelling decomposition results, allowing for applications including free-viewpoint rendering and relighting.

Note the name itself is a "revolutionary" palindrome.




Video




Results

De-rendering from a single image

Decomposition results 1 Decomposition results 2 Decomposition results 3 Decomposition results 4 Decomposition results 5 Decomposition results 6 Decomposition results 7 Decomposition results 8 Decomposition results labels

Novel view rendering and relighting from a single image

images from The Metropolitan Museum of Art Collection



images from Open Images Dataset




Paper

Paper thumbnail.

De-rendering the World's Revolutionary Artefacts

Shangzhe Wu, Ameesh Makadia, Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa

In CVPR 2021

@InProceedings{wu2021derender,
    author={Shangzhe Wu and Ameesh Makadia and Jiajun Wu and Noah Snavely and Richard Tucker and Angjoo Kanazawa},
    title={De-rendering the World's Revolutionary Artefacts},
    booktitle = {CVPR},
    year = {2021}
}



Acknowledgements

We would like to thank Christian Rupprecht, Soumyadip Sengupta, Manmohan Chandraker and Andrea Vedaldi for insightful discussions. The webpage template was adapted from Richard Zhang's and Jason Zhang's templates.