Skip to main content
SpringerLink
Log in

Light Flickering Guided Reflection Removal

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

When photographing through a piece of glass, reflections usually degrade the quality of captured images or videos. In this paper, by exploiting periodically varying light flickering, we investigate the problem of removing strong reflections from contaminated image sequences or videos with a unified capturing setup. We propose a learning-based method that utilizes short-term and long-term observations of mixture videos to exploit one-side contextual clues in fluctuant components and brightness-consistent clues in consistent components for achieving layer separation and flickering removal, respectively. A dataset containing synthetic and real mixture videos with light flickering is built for network training and testing. The effectiveness of the proposed method is demonstrated by the comprehensive evaluation on synthetic and real data, the application for video flickering removal, and the exploratory experiment on high-speed scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Data availability

Datasets used in this study are available from co-first authors on reasonable request.

Notes

  1. In this paper, when a method is not explicitly named, we adopt the convention of using “the initials of the surnames of the first two authors + year” as synonyms of it.

  2. Since the cycle of \(\sin (2\pi i+\epsilon )\) is 1, when \(f=100\) and \(\tau =1/30\), the cycle of \(\sin (20/3\pi i+\epsilon )\) becomes 3.

  3. https://www.edmundoptics.com/p/CM3-U3-50S5C-CS-2-3inch-Chameleon3-Color-Camera/37032.

References

  • Aksoy, Y., Kim, C., Kellnhofer, P., Paris, S., Elgharib, M., Pollefeys, M., & Matusik, W. (2018). A dataset of flash and ambient illumination pairs from the crowd. In Proceedings of European conference on computer vision.

  • Chandran, S., Hold-Geoffroy, Y., Sunkavalli, K., Shu, Z., & Jayasuriya, S. (2022). Temporally consistent relighting for portrait videos. In Proceedings of winter conference on applications of computer vision.

  • Chang, Y., Jung, C., Sun, J., & Wang, F. (2020). Siamese dense network for reflection removal with flash and no-flash image pairs. International Journal of Computer Vision, 128(6), 1673–1698.

    Article  Google Scholar 

  • Chang, Y., Zhou, C., Hong, Y., Hu, L., Xu, C., Huang, T., & Shi, B. (2023). 1000 fps HDR video with a spike-RGB hybrid camera. In Proceedings of computer vision and pattern recognition.

  • Cooley, J. W., & Tukey, J. W. (1965). An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation, 19(90), 297–301.

    Article  MathSciNet  Google Scholar 

  • Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017). Deformable convolutional networks. In Proceedings of computer vision and pattern recognition.

  • Diamant, Y., & Schechner, Y. Y. (2008). Overcoming visual reverberations. In Proceedings of computer vision and pattern recognition.

  • Dong, Z., Xu, K., Yang, Y., Bao, H., Xu, W., & Lau, R. W. (2021). Location-aware single image reflection removal. In Proceedings of international conference on computer vision.

  • Fan, Q., Yang, J., Hua, G., Chen, B., & Wipf, D. (2017). A generic deep architecture for single image reflection removal and image smoothing. In Proceedings of international conference on computer vision.

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Proceedings of advances in neural information processing systems.

  • Hong, Y., Lyu, Y., Li, S., Cao, G., & Shi, B. (2023a). Reflection removal with NIR and RGB image feature fusion. IEEE Transactions on Multimedia, 25, 7101–7112.

    Article  Google Scholar 

  • Hong, Y., Lyu, Y., Li, S., & Shi, B. (2020). Near-infrared image guided reflection removal. In: Proceedings of international conference on multimedia and expo.

  • Hong, Y., Zheng, Q., Zhao, L., Jiang, X., Kot, A. C., & Shi, B. (2021). Panoramic image reflection removal. In Proceedings of computer vision and pattern recognition.

  • Hong, Y., Zheng, Q., Zhao, L., Jiang, X., Kot, A. C., & Shi, B. (2023b). PAR\(^2\)Net: End-to-end panoramic image reflection removal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 12192–12205.

    Article  Google Scholar 

  • Huang, T., Zheng, Y., Yu, Z., Chen, R., Li, Y., Xiong, R., Ma, L., Zhao, J., Dong, S., & Zhu, L., et al. (2022). 1000\(\times \) faster camera and machine vision with ordinary devices. Engineering.

  • Huynh-Thu, Q., & Ghanbari, M. (2008). Scope of validity of PSNR in image/video quality assessment. Electronics Letters, 44(13), 800–801.

    Article  Google Scholar 

  • Kong, N., Tai, Y. W., & Shin, S. Y. (2012). A physically-based approach to reflection separation. In Proceedings of computer vision and pattern recognition.

  • Lei, C., & Chen, Q. (2021). Robust reflection removal with reflection-free flash-only cues. In Proceedings of computer vision and pattern recognition.

  • Lei, C., Huang, X., Zhang, M., Yan, Q., Sun, W., & Chen, Q. (2020). Polarized reflection removal with perfect alignment in the wild. In Proceedings of computer vision and pattern recognition.

  • Levin, A., & Weiss, Y. (2007). User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(9), 1647–1654.

    Article  Google Scholar 

  • Li, C., Yang, Y., He, K., Lin, S., & Hopcroft, J. E. (2020a). Single image reflection removal through cascaded refinement. In Proceedings of computer vision and pattern recognition.

  • Li, R., Qiu, S., Zang, G., & Heidrich, W. (2020b). Reflection separation via multi-bounce polarization state tracing. In Proceedings of European conference on computer vision.

  • Li, Y., & Brown, M. S. (2013). Exploiting reflection change for automatic reflection removal. In Proceedings of international conference on computer vision.

  • Li, Y., & Brown, M. S. (2014). Single image layer separation using relative smoothness. In Proceedings of computer vision and pattern recognition.

  • Liu, Y. L., Lai, W. S., Yang, M. H., Chuang, Y. Y., & Huang, J. B. (2020). Learning to see through obstructions. In Proceedings of computer vision and pattern recognition.

  • Lyu, Y., Cui, Z., Li, S., Pollefeys, M., & Shi, B. (2019). Reflection separation using a pair of unpolarized and polarized images. In Proceedings of advances in neural information processing systems.

  • Lyu, Y., Cui, Z., Li, S., Pollefeys, M., & Shi, B. (2023). Physics-guided reflection separation from a pair of unpolarized and polarized images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2), 2151–2165.

    Article  Google Scholar 

  • Ma, D., Wan, R., Shi, B., Kot, A. C., & Duan, L. Y. (2019). Learning to jointly generate and separate reflections. In Proceedings of international conference on computer vision.

  • Nandoriya, A., Elgharib, M., Kim, C., Hefeeda, M., & Matusik, W. (2017). Video reflection removal through spatio-temporal optimization. In Proceedings of international conference on computer vision.

  • Nayar, S. K., Fang, X. S., & Boult, T. (1997). Separation of reflection components using color and polarization. International Journal of Computer Vision, 21(3), 163–186.

    Article  Google Scholar 

  • Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., & Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Proceedings of advances in neural information processing systems.

  • Schechner, Y. Y., Kiryati, N., & Basri, R. (2000). Separation of transparent layers using focus. International Journal of Computer Vision, 39(1), 25–39.

    Article  Google Scholar 

  • Sheinin, M., Schechner, Y. Y., & Kutulakos, K. N. (2017). Computational imaging on the electric grid. In Proceedings of computer vision and pattern recognition.

  • Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of computer vision and pattern recognition.

  • Shi, X., Chen, Z., Wang, H., Yeung, D. Y., Wong, W. K., & Woo, W. C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proceedings of advances in neural information processing systems.

  • Shih, Y., Krishnan, D., Durand, F., & Freeman, W. T. (2015). Reflection removal using ghosting cues. In Proceedings of computer vision and pattern recognition.

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

  • Wan, R., Shi, B., Duan, L. Y., Tan, A. H., Gao, W., & Kot, A. C. (2018a). Region-aware reflection removal with unified content and gradient priors. IEEE Transactions on Image Processing, 27(6), 2927–2941.

    Article  MathSciNet  Google Scholar 

  • Wan, R., Shi, B., Duan, L. Y., Tan, A. H., & Kot, A. C. (2018b). CRRN: Multi-scale guided concurrent reflection removal network. In Proceedings of computer vision and pattern recognition.

  • Wan, R., Shi, B., Li, H., Duan, L. Y., Tan, A. H., & Kot, A. C. (2019). CoRRN: Cooperative reflection removal network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(12), 2969–2982.

    Article  Google Scholar 

  • Wan, R., Shi, B., Li, H., Hong, Y., Duan, L. Y., & Kot, A. C. (2023). Benchmarking single-image reflection removal algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45, 1424–1441.

  • Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2003). Multiscale structural similarity for image quality assessment. In The thrity-seventh Asilomar conference on signals, systems & computers.

  • Wei, K., Yang, J., Fu, Y., Wipf, D., & Huang, H. (2019). Single image reflection removal exploiting misaligned training data and network enhancements. In Proceedings of computer vision and pattern recognition.

  • Wells, E. F., Bernstein, G. M., Scott, B. W., Bennett, P. J., & Mendelson, J. R. (2001). Critical flicker frequency responses in visual cortex. Experimental Brain Research, 139(1), 106–110.

    Article  Google Scholar 

  • Wu, S., Xu, J., Tai, Y. W., & Tang, C. K. (2018). Deep high dynamic range imaging with large foreground motions. In Proceedings of European conference on computer vision.

  • Yan, Q., Gong, D., Shi, Q., Hengel, Avd, Shen, C., Reid, I., & Zhang, Y. (2019). Attention-guided network for ghost-free high dynamic range imaging. In Proceedings of computer vision and pattern recognition.

  • Yang, Y., Ma, W., Zheng, Y., Cai, J. F., & Xu, W. (2019). Fast single image reflection suppression via convex optimization. In Proceedings of computer vision and pattern recognition.

  • Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018a). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of computer vision and pattern recognition.

  • Zhang, X., Ng, R., & Chen, Q. (2018b). Single image reflection separation with perceptual losses. In Proceedings of computer vision and pattern recognition.

  • Zheng, Q., Shi, B., Chen, J., Jiang, X., Duan, L. Y., & Kot, A. C. (2021). Single image reflection removal with absorption effect. In Proceedings of computer vision and pattern recognition.

  • Zhong, H., Hong, Y., Weng, S., Liang, J., & Shi, B. (2024). Language-guided image reflection separation. arXiv preprint arXiv:2402.11874.

Download references

Acknowledgements

This work was supported by National Science and Technology Major Project (Grant No. 2021ZD0109803), National Natural Science Foundation of China under Grant No. 62301009, 62088102, and 62136001.

Author information

Author notes

  1. Yuchen Hong and Yakun Chang have contributed equally to this work.

Authors and Affiliations

  1. National Key Laboratory for Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China

    Yuchen Hong, Jinxiu Liang, Tiejun Huang & Boxin Shi

  2. Institute of Information Science, Beijing Jiaotong University, Beijing, China

    Yakun Chang

  3. Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing, China

    Yakun Chang

  4. National Biomedical Imaging Center, College of Future Technology, Peking University, Beijing, China

    Lei Ma

  5. Beijing Academy of Artificial Intelligence, Beijing, China

    Lei Ma

Authors
  1. Yuchen Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yakun Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinxiu Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tiejun Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Boxin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Boxin Shi.

Additional information

Communicated by Rynson W. H. Lau.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 24134 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, Y., Chang, Y., Liang, J. et al. Light Flickering Guided Reflection Removal. Int J Comput Vis (2024). https://doi.org/10.1007/s11263-024-02073-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11263-024-02073-z

Keywords

Search

Navigation