Chris Careaga
Abstract
Intrinsic decomposition is a fundamental mid-level vision problem that plays a crucial role in various inverse rendering and computational photography pipelines. Generating highly accurate intrinsic decompositions is an inherently under-constrained task that requires precisely estimating continuous-valued shading and albedo. In this work, we achieve high-resolution intrinsic decomposition by breaking the problem into two parts. First, we present a dense ordinal shading formulation using a shift- and scale-invariant loss in order to estimate ordinal shading cues without restricting the predictions to obey the intrinsic model. We then combine low- and high-resolution ordinal estimations using a second network to generate a shading estimate with both global coherency and local details. We encourage the model to learn an accurate decomposition by computing losses on the estimated shading as well as the albedo implied by the intrinsic model. We develop a straightforward method for generating dense pseudo ground truth using our model’s predictions and multi-illumination data, enabling generalization to in-the-wild imagery. We present exhaustive qualitative and quantitative analysis of our predicted intrinsic components against state-of-the-art methods. Finally, we demonstrate the real-world applicability of our estimations by performing otherwise difficult editing tasks such as recoloring and relighting.
Supplemental Material
Available for Download
Supplementary material
- . 2018a. Joint learning of intrinsic images and semantic segmentation. In Proc. ECCV.Google Scholar
- . 2018b. CNN based learning using reflection and Retinex models for intrinsic image decomposition. In Proc. CVPR.Google Scholar
- . 2014. Intrinsic images in the wild. ACM Trans. Graph. 33, 4 (2014), 1–12.Google ScholarDigital Library
- . 2018. Deep hybrid real and synthetic training for intrinsic decomposition. In Proc. EGSR.Google Scholar
- . 2017. Intrinsic decompositions for image editing. Comput. Graph. Forum 36, 2 (2017).Google ScholarCross Ref
- . 2012. A naturalistic open source movie for optical flow evaluation. In Proc. ECCV.Google Scholar
- . 2015. ShapeNet: An Information-Rich 3D Model Repository.
Technical Report arXiv:1512.03012 [cs.GR]. Stanford University — Princeton University — Toyota Technological Institute at Chicago.Google Scholar - . 2018. Intrinsic image transformation via scale space decomposition. In Proc. CVPR.Google Scholar
- . 2022. PIE-Net: Photometric invariant edge guided network for intrinsic image decomposition. In Proc. CVPR.Google Scholar
- . 2021. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3D scans. In Proc. ICCV.Google Scholar
- . 2018. Revisiting deep intrinsic image decompositions. In Proc. CVPR.Google Scholar
- . 2012. Intrinsic images by clustering. Comput. Graph. Forum 31, 4 (2012), 1415–1424.Google ScholarDigital Library
- . 2022. A survey on intrinsic images: Delving deep into Lambert and beyond. Int. J. Comput. Vision (2022).Google ScholarDigital Library
- . 2009. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In Proc. ICCV.Google Scholar
- . 2017. Self-supervised intrinsic image decomposition. In Proc. NeurIPS.Google Scholar
- . 2017. Shading annotations in the wild. Proc. CVPR.Google Scholar
- . 2018. Free supervision from video games. In Proc. CVPR.Google Scholar
- . 2021. EDEN: Multimodal synthetic dataset of enclosed garden scenes. In Proc. WACV.Google Scholar
- . 2018a. DARN: A deep adversarial residual network for intrinsic image decomposition. Proc. WACV.Google Scholar
- . 2018b. Unsupervised deep single-image intrinsic decomposition using illumination-varying image sequences. Comput. Graph. Forum 37, 7 (2018), 409–419.Google ScholarCross Ref
- . 2020. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image. In Proc. CVPR.Google Scholar
- . 2018a. CGIntrinsics: Better intrinsic image decomposition through physically-based rendering. In Proc. ECCV.Google Scholar
- . 2018b. Learning intrinsic image decomposition from watching the world. In Proc. CVPR.Google Scholar
- . 2018c. MegaDepth: Learning single-view depth prediction from Internet photos. In Proc. CVPR.Google Scholar
- . 2021. OpenRooms: An open framework for photorealistic indoor scene datasets. Proc. CVPR.Google Scholar
- . 2017. RefineNet: Multi-path refinement networks for high-resolution semantic segmentation. In Proc. CVPR.Google Scholar
- . 2020. Unsupervised learning for intrinsic image decomposition from a single image. In Proc. CVPR.Google Scholar
- . 2020. NIID-Net: Adapting surface normal knowledge for intrinsic image decomposition in indoor scenes. IEEE Trans. Vis. Comp. Graph. (2020).Google ScholarCross Ref
- . 2018. Single image intrinsic decomposition without a single intrinsic image. In Proc. ECCV.Google Scholar
- . 2018. LIME: Live intrinsic material estimation. In Proc. CVPR.Google Scholar
- . 2021. Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging. In Proc. CVPR.Google Scholar
- . 2019. A multi-illumination dataset of indoor object appearance. In Proc. ICCV.Google Scholar
- . 2015. Learning lightness from human judgement on relative reflectance. In Proc. CVPR.Google Scholar
- . 2017. Reflectance adaptive filtering improves intrinsic image estimation. In Proc. CVPR.Google Scholar
- . 2003. Poisson image editing. In ACM SIGGRAPH. 313–318.Google Scholar
- . 2020. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Mach. Intell. (2020).Google Scholar
- . 2021. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proc. ICCV.Google Scholar
- . 2019. Neural inverse rendering of an indoor scene from a single image. In Proc. ICCV.Google Scholar
- . 2011. Intrinsic images using optimization. In Proc. CVPR.Google Scholar
- . 2017. Learning non-Lambertian object intrinsics across ShapeNet categories. In Proc. CVPR.Google Scholar
- . 2015. Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In Proc. ICCV.Google Scholar
- . 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proc. ICML.Google Scholar
- . 2020. Structure-guided ranking loss for single image depth prediction. In Proc. CVPR.Google Scholar
- . 2017. Aggregated residual transformations for deep neural networks. In Proc. CVPR.Google Scholar
- . 2014. Intrinsic video and applications. ACM Trans. Graph. 33, 4 (2014).Google ScholarDigital Library
- . 2012. A closed-form solution to Retinex with nonlocal texture constraints. IEEE Trans. Pattern Anal. Mach. Intell. 34, 7 (2012), 1437–1444.Google ScholarDigital Library
- . 2019. GLoSH: Global-local spherical harmonics for intrinsic image decomposition. In Proc. ICCV.Google Scholar
- . 2015. Learning data-driven reflectance priors for intrinsic image decomposition. In Proc. ICCV.Google Scholar
- . 2022. IRISformer: Dense vision transformers for single-image inverse rendering in indoor scenes. In Proc. CVPR.Google Scholar
- . 2015. Learning ordinal relationships for mid-level vision. In Proc. ICCV.Google Scholar
Index Terms
- Intrinsic Image Decomposition via Ordinal Shading
Recommendations
Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniquesIn this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...
SOL-NeRF: Sunlight Modeling for Outdoor Scene Decomposition and Relighting
SA '23: SIGGRAPH Asia 2023 Conference PapersOutdoor scenes often involve large-scale geometry and complex unknown lighting conditions, making it difficult to decompose them into geometry, reflectance and illumination. Recently researchers made attempts to decompose outdoor scenes using Neural ...
Technical Section: Reflectance modeling for a textured object under uncontrolled illumination from high dynamic range maps
During the past several years, considerable work has been presented on the methods for measuring and modeling the observed reflectance properties of materials. However, most of these works have been done under controlled lighting configurations, and ...
Comments