这是indexloc提供的服务，不要输入任何密码
Skip to main content
Springer Nature Link
Log in

Deep Unrolled Weighted Graph Laplacian Regularization for Depth Completion

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Depth completion aims to estimate dense depth images from sparse depth measurements with RGB image guidance. However, previous approaches have not fully considered sparse input fidelity, resulting in inconsistency with sparse input and poor robustness to input corruption. In this paper, we propose the deep unrolled Weighted Graph Laplacian Regularization (WGLR) for depth completion which enhances input fidelity and noise robustness by enforcing input constraints in the network design. Specifically, we assume graph Laplacian regularization as the prior for depth completion optimization and derive the WGLR solution by interpreting the depth map as the discrete counterpart of continuous manifold, enabling analysis in continuous domain and enforcing input consistency. Based on its anisotropic diffusion interpretation, we unroll the WGLR solution into iterative filtering for efficient implementation. Furthermore, we integrate the unrolled WGLR into deep learning framework to develop high-performance yet interpretable network, which diffuses the depth in a hierarchical manner to ensure global smoothness while preserving visually salient details. Experimental results demonstrate that the proposed scheme improves consistency with depth measurements and robustness to input corruption for depth completion, outperforming competing schemes on the NYUv2, KITTI-DC and TetrasRGBD datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  • Barron, J. T., & Malik, J. (2013). Intrinsic scene properties from a single rgb-d image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 17–24).

  • Barron, J. T., & Poole, B. (2016). The fast bilateral solver. In European Conference on Computer Vision (ECCV) (pp. 617–632). Springer

  • Bishop, C. M., & Nasrabadi, N. M. (2006). Pattern Recognition and Machine Learning (pp. 161–162). New York: Springer.

    MATH  Google Scholar 

  • Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017). Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV) (pp. 667–676).

  • Chen, H., Yang, H., Zhang, Y., et al. (2022). Depth completion using geometry-aware embedding. In International Conference on Robotics and Automation (ICRA) (pp. 8680–8686). IEEE.

  • Cheng, X., Wang, P., & Yang, R. (2018). Depth estimation via affinity learned with convolutional spatial propagation network. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 103–119).

  • Cheng, X., Wang, P., & Yang, R. (2019). Learning depth with convolutional spatial propagation network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10), 2361–2379.

    Article  MATH  Google Scholar 

  • Cheung, G., Magli, E., Tanaka, Y., & Ng, M. K. (2018). Graph spectral image processing. Proceedings of the IEEE, 106(5), 907–930.

    Article  Google Scholar 

  • Chodosh, N., Wang, C., & Lucey, S. (2018). Deep convolutional compressed sensing for lidar depth completion. In Asian Conference on Computer Vision (ACCV) (pp. 499–513). Springer.

  • Cong, R., Lei, J., Zhang, C., Huang, Q., Cao, X., & Hou, C. (2016). Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Processing Letters, 23(6), 819–823.

    Article  MATH  Google Scholar 

  • Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (pp. 248–255). Ieee.

  • Du, R., Turner, E., Dzitsiuk, M., Prasso, L., Duarte, I., Dourgarian, J., Afonso, J., Pascoal, J., Gladstone, J., Cruces, N., et al. (2020). Depthlab: Real-time 3d interaction with depth maps for mobile augmented reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (pp. 829–843).

  • Eldesokey, A., Felsberg, M., Holmquist, K., & Persson, M. (2020). Uncertainty-aware cnns for depth completion: Uncertainty from beginning to end. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 12014–12023).

  • Eldesokey, A., Felsberg, M., & Khan, F. S. (2019). Confidence propagation through CNNS for guided sparse depth regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10), 2423–2436.

    Article  MATH  Google Scholar 

  • Farid, M. S., Lucenteforte, M., & Grangetto, M. (2015). Blind depth quality assessment using histogram shape analysis. In 2015 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON) (pp. 1–5). IEEE

  • Feng, Z., Jing, L., Yin, P., Tian, Y., & Li, B. (2022). Advancing self-supervised monocular depth learning with sparse lidar. In Conference on Robot Learning (pp. 685–694). PMLR.

  • Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., & Bischof, H. (2013). Image guided depth upsampling using anisotropic total generalized variation. In International Conference on Computer Vision (ICCV) (pp. 993–1000).

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (pp. 3354–3361). IEEE.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778).

  • Hu, W., Li, X., Cheung, G., & Au, O. (2013). Depth map denoising using graph-based transform and group sparsity. In IEEE International Workshop on Multimedia Signal Processing (pp. 001–006). IEEE.

  • Hu, M., Wang, S., Li, B., Ning, S., Fan, L., & Gong, X. (2021). Penet: Towards precise and efficient image guided depth completion. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 13656–13662). IEEE.

  • Huang, Z., Fan, J., Cheng, S., Yi, S., Wang, X., & Li, H. (2019). Hms-net: Hierarchical multi-scale sparsity-invariant network for sparse depth completion. IEEE Transactions on Image Processing, 29, 3429–3441.

    Article  MATH  Google Scholar 

  • Li, Y., Yu, A. W., Meng, T., Caine, B., Ngiam, J., Peng, D., Shen, J., Lu, Y., Zhou, D., Le, Q.V., et al. (2022). Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 17182–17191).

  • Li, A., Yuan, Z., Ling, Y., Chi, W., Zhang, C., et al. (2020). A multi-scale guided cascade hourglass network for depth completion. In IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 32–40).

  • Lin, Y., Cheng, T., Zhong, Q., Zhou, W., & Yang, H. (2022). Dynamic spatial propagation network for depth completion. In Proceedings of the AAAI Conference on Artificial Intelligence,36, 1638–1646.

  • Li, Z., Shi, Z., & Sun, J. (2017). Point integral method for solving poisson-type equations on manifolds from point clouds with convergence guarantees. Communications in Computational Physics, 22(1), 228–258.

    Article  MathSciNet  MATH  Google Scholar 

  • Liu, S., De Mello, S., Gu, J., Zhong, G., Yang, M.-H., & Kautz, J. (2017). Learning affinity via spatial propagation networks. Advances in Neural Information Processing Systems,30.

  • Liu, X., Shao, X., Wang, B., Li, Y., & Wang, S. (2022). Graphcspn: Geometry-aware depth completion via dynamic gcns. In European Conference on Computer Vision (ECCV) (pp. 90–107). Springer

  • Liu, L.-K., Chan, S. H., & Nguyen, T. Q. (2015). Depth reconstruction from sparse samples: Representation, algorithm, and sampling. IEEE Transactions on Image Processing, 24(6), 1983–1996.

    Article  MathSciNet  MATH  Google Scholar 

  • Liu, L., Song, X., Sun, J., Lyu, X., Li, L., Liu, Y., & Zhang, L. (2023). Mff-net: Towards efficient monocular depth completion with multi-modal feature fusion. IEEE Robotics and Automation Letters, 8(2), 920–927.

    Article  MATH  Google Scholar 

  • López-Randulfe, J., Veiga, C., Rodríguez-Andina, J. J., & Farina, J. (2017). A quantitative method for selecting denoising filters, based on a new edge-sensitive metric. In 2017 IEEE International Conference on Industrial Technology (ICIT) (pp. 974–979). IEEE

  • Lopez-Rodriguez, A., Busam, B., & Mikolajczyk, K. (2022). Project to adapt: Domain adaptation for depth completion from noisy and sparse sensor data. International Journal of Computer Vision, 1–17.

  • Ma, F., & Karaman, S. (2018). Sparse-to-dense: Depth prediction from sparse depth samples and a single image. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 4796–4803). IEEE.

  • Ma, X., Liu, S., Xia, Z., Zhang, H., Zeng, X., & Ouyang, W. (2020). Rethinking pseudo-lidar representation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16 (pp. 311–327). Springer.

  • Milanfar, P. (2012). A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE Signal Processing Magazine, 30(1), 106–128.

    Article  MATH  Google Scholar 

  • Monga, V., Li, Y., & Eldar, Y. C. (2021). Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2), 18–44.

    Article  MATH  Google Scholar 

  • Mufti, F., & Mahony, R. (2011). Statistical analysis of signal measurement in time-of-flight cameras. ISPRS Journal of Photogrammetry and Remote Sensing, 66(5), 720–731.

    Article  MATH  Google Scholar 

  • Ortega, A., Frossard, P., Kovačević, J., Moura, J. M., & Vandergheynst, P. (2018). Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE, 106(5), 808–828.

    Article  MATH  Google Scholar 

  • Osher, S., Shi, Z., & Zhu, W. (2017). Low dimensional manifold model for image processing. SIAM Journal on Imaging Sciences, 10(4), 1669–1690.

    Article  MathSciNet  MATH  Google Scholar 

  • Pang, J., & Zeng, J. (2021). Graph spectral image restoration. Graph Spectral Image Processing,133.

  • Pang, J., & Cheung, G. (2017). Graph Laplacian regularization for image denoising: Analysis in the continuous domain. IEEE Transactions on Image Processing, 26(4), 1770–1785.

    Article  MathSciNet  MATH  Google Scholar 

  • Park, J., Joo, K., Hu, Z., Liu, C.-K., & So Kweon, I. (2020). Non-local spatial propagation network for depth completion. In European Conference on Computer Vision (ECCV) (pp. 120–136). Springer.

  • Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., & Lerer, A. (2017). Automatic differentiation in pytorch.

  • Perona, P., & Malik, J. (1990). Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on pattern analysis and machine intelligence, 12(7), 629–639.

    Article  MATH  Google Scholar 

  • Qiu, J., Cui, Z., Zhang, Y., Zhang, X., Liu, S., Zeng, B., & Pollefeys, M. (2019). Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 3313–3322).

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention (pp. 234–241). Springer.

  • Shewchuk, J. R., et al. (1994). An introduction to the conjugate gradient method without the agonizing pain. Department of Computer Science, Carnegie-Mellon University.

  • Shi, Z., Sun, J., & Tian, M. (2018). Harmonic extension on the point cloud. Multiscale Modeling & Simulation, 16(1), 215–247.

    Article  MathSciNet  MATH  Google Scholar 

  • Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012). Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision (ECCV) (pp. 746–760). Springer.

  • Strong, D. M., & Chan, T. F. (1996). Spatially and scale adaptive total variation based regularization and anisotropic diffusion in image processing. In Diusion in Image Processing, UCLA Math Department CAM Report. Citeseer.

  • Sun, W., Zhu, Q., Li, C., Feng, R., Zhou, S., Jiang, J., Yang, Q., Loy, C. C., Gu, J., Hou, D., et al. (2023). Mipi 2022 challenge on rgb+ tof depth completion: Dataset and report. In European Conference on Computer Vision (ECCV) Workshop (pp. 3–20). Springer.

  • Uhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A . (2017). Sparsity invariant cnns. In International Conference on 3D Vision (3DV) (pp. 11–20). IEEE

  • Van Gansbeke, W., Neven, D., De Brabandere, B., & Van Gool, L. (2019). Sparse and noisy lidar completion with rgb guidance and uncertainty. In International Conference on Machine Vision Applications (MVA) (pp. 1–6). IEEE.

  • Xu, Y., Zhu, X., Shi, J., Zhang, G., Bao, H., & Li, H. (2019). Depth completion from sparse lidar data with depth-normal constraints. In International Conference on Computer Vision (ICCV) (pp. 2811–2820).

  • Yan, Z., Wang, K., Li, X., Zhang, Z., Li, J., & Yang, J. (2022). Rignet: Repetitive image guided network for depth completion. In European Conference on Computer Vision (pp. 214–230). Springer.

  • You, Y., Wang, Y., Chao, W.-L., Garg, D., Pleiss, G., Hariharan, B., Campbell, M., & Weinberger, K. Q. (2020). Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In International Conference on Learning Representations (ICLR).

  • Zeng, J., Tong, Y., Huang, Y., Yan, Q., Sun, W., Chen, J., & Wang, Y. (2019). Deep surface normal estimation with hierarchical rgb-d fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6153–6162).

  • Zhang, Y., Guo, X., Poggi, M., Zhu, Z., Huang, G., & Mattoccia, S. (2023). Completionformer: Depth completion with convolutions and vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18527–18536).

  • Zhao, S., Gong, M., Fu, H., & Tao, D. (2021). Adaptive context-aware multi-modal network for depth completion. IEEE Transactions on Image Processing, 30, 5264–5276.

    Article  MATH  Google Scholar 

  • Zhou, W., Yan, X., Liao, Y., Lin, Y., Huang, J., Zhao, G., Cui, S., & Li, Z. (2023). Bev@ dc: Bird’s-eye view assisted training for depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9233–9242).

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China under Grant 62201389, and in part by Shanghai Sailing Program under Grant 22YF1451200.

Author information

Authors and Affiliations

  1. School of Software Engineering, Tongji University, Shanghai, China

    Jin Zeng, Tongxuan Tian, Lin Zhang & Shengjie Zhao

  2. SenseTime Research, Shenzhen, China

    Qingpeng Zhu & Wenxiu Sun

Authors
  1. Jin Zeng
    View author publications

    Search author on:PubMed Google Scholar

  2. Qingpeng Zhu
    View author publications

    Search author on:PubMed Google Scholar

  3. Tongxuan Tian
    View author publications

    Search author on:PubMed Google Scholar

  4. Wenxiu Sun
    View author publications

    Search author on:PubMed Google Scholar

  5. Lin Zhang
    View author publications

    Search author on:PubMed Google Scholar

  6. Shengjie Zhao
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Qingpeng Zhu.

Additional information

Communicated by Yasuyuki Matsushita.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, J., Zhu, Q., Tian, T. et al. Deep Unrolled Weighted Graph Laplacian Regularization for Depth Completion. Int J Comput Vis 133, 190–210 (2025). https://doi.org/10.1007/s11263-024-02188-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s11263-024-02188-3

Keywords