Chuankun Zheng
Skip Abstract Section
Skip Supplemental Material SectionAbstract
This article presents object-oriented neural light transfer (NeLT), a novel neural representation of the dynamic light transportation between an object and the environment. Our method disentangles the global illumination of a scene into individual objects’ light transportation represented via neural networks, then composes them explicitly. It therefore enables flexible rendering with dynamic lighting, cameras, materials, and objects. Our rendering features various important global illumination effects, such as diffuse illumination, glossy illumination, dynamic shadowing, and indirect illumination, which completes the capability of existing neural object representation. Experiments show that NeLT does not require path tracing or shading results as input but achieves rendering quality comparable to state-of-the-art rendering frameworks, including the recent deep learning based denoisers.
Supplemental Material
Available for Download
zip
tog-22-0118-supp.zip (159.1 MB)
Supplementary material
- . 2020. Neural point-based graphics. In Computer Vision—ECCV 2020. Lecture Notes in Computer Science, Vol. 12367. Springer, 696–712.Google Scholar
- . 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics 36, 4 (2017), Article 97, 1–14.Google ScholarDigital Library
- . 2008. A precomputed polynomial representation for interactive BRDF editing with global illumination. ACM Transactions on Graphics 27, 2 (2008), 1–13.Google ScholarDigital Library
- . 2006. Real-time BRDF editing in complex lighting. ACM Transactions on Graphics 25, 3 (2006), 945–954.Google ScholarDigital Library
- . 2016. Reduced aggregate scattering operators for path tracing. Computer Graphics Forum 35 (2016), 461–473.Google Scholar
- . 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics 36, 4 (2017), Article 98, 1–12.Google ScholarDigital Library
- , Yuchi Huo, and Sung-Eui Yoon. 2021. Weakly-supervised contrastive learning in path manifold for monte carlo image reconstruction. ACM Trans. Graph. 40, 4 (2021), 38–1.Google Scholar
- . 2005. Reflective shadow maps. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games. 203–231.Google ScholarDigital Library
- . 2007. Implicit visibility and antiradiance for interactive global illumination. ACM Transactions on Graphics 26, 3 (2007), 61–es.Google ScholarDigital Library
- . 2022. Neural shadow mapping. In Proceedings of the ACM SIGGRAPH 2022 Conference (SIGGRAPH’22). 1–9.Google ScholarDigital Library
- . 2022. Active exploration for neural global illumination of variable scenes. ACM Transactions on Graphics 41, 5 (2022), Article 171, 18 pages.Google ScholarDigital Library
- . 2018. Neural scene representation and rendering. Science 360, 6394 (2018), 1204–1210.Google ScholarCross Ref
- . 2021. Real-time Monte Carlo denoising with weight sharing kernel prediction network. Computer Graphics Forum 40 (2021), 15–27.Google Scholar
- . 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics 38, 4 (2019), 1–12.Google ScholarDigital Library
- . 2020. Compositional neural scene representations for shading inference. ACM Transactions on Graphics 39, 4 (2020), Article 135, 13 pages.Google ScholarDigital Library
- . 2021. Neural scene graph rendering. ACM Transactions on Graphics 40, 4 (2021), 1–11.Google ScholarDigital Library
- . 2019. GradNet: Unsupervised deep screened Poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics 38, 6 (2019), 1–13.Google ScholarDigital Library
- . 2020. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503 (2020).Google Scholar
- . 2017. Hyper networks. In Proceedings of the 5th International Conference on Learning Representations: Conference Track (ICLR’17).Google Scholar
- . 2021. Neural radiosity. ACM Transactions on Graphics 40, 6 (2021), 1–11.Google ScholarDigital Library
- . 2019. Deep-learning the latent space of light transport. Computer Graphics Forum 38 (2019), 207–217).Google Scholar
- . 2020. Adaptive incident radiance field sampling and reconstruction using deep reinforcement learning. ACM Transactions on Graphics 39, 1 (2020), 1–17.Google ScholarDigital Library
- . 2021. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media 7 (2021), 169–185.Google ScholarCross Ref
- . 2017. Deep scattering: Rendering atmospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics 36, 6 (2017), 1–11.Google ScholarDigital Library
- . 2014. High-quality temporal supersampling. In Proceedings of the ACM SIGGRAPH 2014 Conference: Advances in Real-Time Rendering in Games(SIGGRAPH’14).Google Scholar
- . 2004. Hemispherical rasterization for self-shadowing of dynamic objects. In Proceedings of the 15th Eurographics Conference on Rendering Techniques (EGSR’04). 179–184.Google Scholar
- . 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR’15): Conference Track.Google Scholar
- . 2005. Ambient occlusion fields. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games. 41–48.Google ScholarDigital Library
- . 2005. Precomputed local radiance transfer for real-time lighting design. ACM Transactions on Graphics 24, 3 (2005), 1208–1215.Google ScholarDigital Library
- . 2019. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics 38, 4 (July 2019), Article 65, 14 pages.Google ScholarDigital Library
- . 2011. Modular radiance transfer. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1–10.Google ScholarDigital Library
- . 2012. Delta radiance transfer. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. 191–196.Google ScholarDigital Library
- , Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65, 1 (2021), 99–106.Google Scholar
- . 2021. Tiny CUDA Neural Network Framework. Retrieved May 18, 2023 from https://github.com/nvlabs/tiny-cuda-nn.Google Scholar
- . 2021. Real-time neural radiance caching for path tracing. ACM Transactions on Graphics 40, 4 (2021), 1–16.Google ScholarDigital Library
- . 2017. Deep shading: Convolutional neural networks for screen space shading. Computer Graphics Forum 36 (2017), 65–78.Google Scholar
- . 2021. GIRAFFE: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’21).Google ScholarCross Ref
- . 2021. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2856–2865.Google ScholarCross Ref
- . 2010. Optix: A general purpose ray tracing engine. ACM Transactions on Graphics 29, 4 (2010), 1–13.Google ScholarDigital Library
- . 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS’19). 8026–8037.Google Scholar
- . 2020. Physically Based Rendering: From Theory to Implementation (4th ed). MIT Press, Cambridge, MA. https://github.com/mmp/pbrt-v4.Google Scholar
- . 2020. Neural voxel renderer: Learning an accurate and controllable rendering tool. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5417–5427.Google ScholarCross Ref
- . 2013. Global illumination with radiance regression functions. ACM Transactions on Graphics 32, 4 (2013), Article 130, 12 pages.Google ScholarDigital Library
- . 2009. Approximating dynamic global illumination in image space. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games. 75–82.Google ScholarDigital Library
- . 2020. Learning neural light transport. arXiv preprint arXiv:2006.03427 (2020).Google Scholar
- . 2007. Hardware accelerated ambient occlusion techniques on GPUs. In Proceedings of the 2007 Symposium on Interactive 3D Graphics and Games. 73–80.Google ScholarDigital Library
- . 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’02). 527–536.Google ScholarDigital Library
- . 2007. Interactive relighting with dynamic BRDFs. ACM Transactions on Graphics 26, 3 (2007), 27–es.Google ScholarDigital Library
- . 2016. Multi-view 3D models from single images with a convolutional network. In Proceedings of the European Conference on Computer Vision. 322–337.Google ScholarCross Ref
- . 2019. A learned shape-adaptive subsurface scattering model. ACM Transactions on Graphics 38, 4 (2019), 1–15.Google ScholarDigital Library
- . 2018. Denoising with kernelprediction and asymmetric loss functions. ACM Transactions on Graphics 37, 4 (2018), 1–15.Google ScholarDigital Library
- . 2005. Lightcuts: A scalable approach to illumination. ACM Transactions on Graphics 24, 3 (2005), 1098–1107.Google Scholar
- . 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR’07). 195–206.Google Scholar
- , Shaokun Zheng, Kun Xu, and Ling-Qi Yan. 2022. Lightweight bilateral convolutional neural networks for interactive single-bounce diffuse indirect illumination. IEEE Transactions on Visualization and Computer Graphics 28, 4 (2022), 1824–1834.Google Scholar
- . 2019. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Transactions on Graphics 38, 6 (2019), Article 224, 12 pages.Google ScholarDigital Library
- . 2021. Monte Carlo denoising via auxiliary feature guided self-attention. ACM Transactions on Graphics 40, 6 (2021), Article 273, 13 pages.Google ScholarDigital Library
- . 2021. A compact representation of measured BRDFs using neural processes. ACM Transactions on Graphics 41, 2 (2021), 1–15.Google ScholarDigital Library
- . 2005. Precomputed shadow fields for dynamic scenes. ACM Transactions on Graphics 24, 3 (2005), 1196–1201.Google ScholarDigital Library
- . 2021. Neural complex luminaires: Representation and rendering. ACM Transactions on Graphics 40, 4 (2021), 1–12.Google ScholarDigital Library
Index Terms
- NeLT: Object-Oriented Neural Light Transfer
Recommendations
Deferred neural lighting: free-viewpoint relighting from unstructured photographs
We present deferred neural lighting, a novel method for free-viewpoint relighting from unstructured photographs of a scene captured with handheld devices. Our method leverages a scene-dependent neural rendering network for relighting a rough geometric ...
Neural Light Transport for Relighting and View Synthesis
The light transport (LT) of a scene describes how it appears under different lighting conditions from different viewing directions, and complete knowledge of a scene’s LT enables the synthesis of novel views under arbitrary lighting. In this article, we ...
Deep Reflectance Volumes: Relightable Reconstructions from Multi-view Photometric Images
Computer Vision – ECCV 2020AbstractWe present a deep learning approach to reconstruct scene appearance from unstructured images captured under collocated point lighting. At the heart of Deep Reflectance Volumes is a novel volumetric scene representation consisting of opacity, ...
Comments