这是indexloc提供的服务，不要输入任何密码
Skip to main content
Springer Nature Link
Log in

DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Hyperspectral Images (HSIs) provide detailed scene insights using extensive spectral bands, crucial for material discrimination and earth observation with substantial costs and low spatial resolution. Recently, Convolutional Neural Networks (CNNs) are common choice for Spectral Super-Resolution (SSR) from Multispectral Images (MSIs). However, they often fail to simultaneously exploit pixel-level noise degradation of MSIs and complex contextual spatial-spectral characteristics of HSIs. In this paper, a Deep Local Residual Attention Network with Contextual Refinement Network (DLRA-Net) is proposed to integrate local low-rank spectral and global contextual priors for improved SSR. Specifically, SSR is unfolded into Contextual-attention Refinement Module (CRM) and Dual Local Residual Attention Module (DLRAM). CRM is proposed to adaptively learn complex contextual priors to guide the convolution layer weights for improved spatial restorations. While DLRAM captures deep refined texture details to enhance contextual priors representations for recovering HSIs. Moreover, lateral fusion strategy is designed to integrate the obtained priors among DLRAMs for faster network convergence. Experimental results on natural-scene datasets with practical noise patterns confirm exceptional DLRA-Net performance with relatively small model size. DLRA-Net demonstrates Maximum Relative Improvements (MRI) between 9.71 and 58.58% in Mean Relative Absolute Error (MRAE) with reduced parameters between 52.18 and 85.85%. Besides, a practical RS-HSI dataset is generated for evaluations showing MRI between 8.64 and 50.56% in MRAE. Furthermore, experiments with HSI classifiers indicate improved performance of reconstructed RS-HSIs compared to RS-MSIs, with MRI in Overall Accuracy (OA) between 7.10 and 15.27%. Lastly, a detailed ablation study assesses model complexity and runtime.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data Availability

The data and materials utilized in this study can be made available by the corresponding author upon reasonable request.

Code Availability

The code utilized in this study can be made available by the corresponding author upon reasonable request.

References

  • Aly, H., & Dubois, E. (2005). Image up-sampling using total-variation regularization with a new observation model. IEEE Transactions on Image Processing, 14(10), 1647–1659. https://doi.org/10.1109/TIP.2005.851684

    Article  MATH  Google Scholar 

  • Aly, H., & Dubois, E. (2005). Specification of the observation model for regularized image up-sampling. IEEE Transactions on Image Processing, 14(5), 567–576. https://doi.org/10.1109/TIP.2005.846019

    Article  MathSciNet  MATH  Google Scholar 

  • Aly, H. A., & Sharma, G. (2014). A regularized model-based optimization framework for pan-sharpening. IEEE Transactions on Image Processing, 23(6), 2596–2608. https://doi.org/10.1109/TIP.2014.2316641

    Article  MathSciNet  MATH  Google Scholar 

  • Arad, B. , Ben-Shahar, O. , Timofte, R., & Van Gool, L. (2018). NTIRE 2018 challenge on single image super-resolution: Methods and results. in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 965–96511.

  • Arad, B. , Timofte, R. , Ben-Shahar, O. , Lin, Y. T., & Finlayson, G.D. (2020). NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image. in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1806–1822. Los Alamitos, CA, USA: IEEE Computer Society.

  • Arad, B. , Timofte, R. , Yahel, R. , Morag, N. , Bernat, A. , Cai, Y. et al. (2022). NTIRE 2022 spectral recovery challenge and data set. in 2022 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp. 862–880.

  • Arad, B. , Timofte, R. , Yahel, R. , Morag, N. , Bernat, A. , Wu, Y. et al. (2022). NTIRE 2022 spectral demosaicing challenge and data set. in 2022 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp. 881–895.

  • Arad, B., & Ben-Shahar, O. (2016). Sparse recovery of hyperspectral signal from natural RGB images. In B. Leibe, J. Matas, N. Sebe, & M. Welling (Eds.), Computer vision-ECCV 2016 (pp. 19–34). Cham: Springer International Publishing.

    Google Scholar 

  • Awad, M., Elliethy, A., & Aly, H. A. (2020). Adaptive near-infrared and visible fusion for fast image enhancement. IEEE Transactions on Computational Imaging, 6, 408–418. https://doi.org/10.1109/TCI.2019.2956873

    Article  MATH  Google Scholar 

  • Bioucas-Dias, J. M., Plaza, A., Camps-Valls, G., Scheunders, P., Nasrabadi, N., & Chanussot, J. (2013). Hyperspectral remote sensing data analysis and future challenges. IEEE Geoscience and remote sensing magazine, 1(2), 6–36. https://doi.org/10.1109/MGRS.2013.2244672

    Article  Google Scholar 

  • Borengasser, M., Hungate, W. S., & Watkins, R. (2007). Hyperspectral remote sensing: Principles and applications. CRC Press.

    MATH  Google Scholar 

  • Breuer, M., & Albertz, J. (2000). Geometric correction of airborne whiskbroom scanner imagery using hybrid auxiliary data. International Archives of Photogrammetry and Remote Sensing, 33(B3/1; PART 3), 93–100.

    MATH  Google Scholar 

  • Cai, Y. , Lin, J. , Hu, X. , Wang, H. , Yuan, X. , Zhang, Y., & Van Gool, L. (2022). Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17502–17511. https://doi.org/10.1109/CVPR52688.2022.01698

  • Calin, M. A., Parasca, S. V., Savastru, D., & Manea, D. (2014). Hyperspectral imaging in the medical field: Present and future. Applied Spectroscopy Reviews, 49(6), 435–447. https://doi.org/10.1080/05704928.2013.838678

    Article  Google Scholar 

  • Chen, X. , Wang, X. , Zhou, J. , Qiao, Y., & Dong, C. (2023). Activating more pixels in image super-resolution transformer. in 2023 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 22367–22377.

  • Deng, L., Sun, J., Chen, Y., Lu, H., Duan, F., Zhu, L., & Fan, T. (2021). M2H-Net: A reconstruction method for hyperspectral remotely sensed imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 173, 323–348. https://doi.org/10.1016/j.isprsjprs.2021.01.019

    Article  Google Scholar 

  • Dian, R., Li, S., & Kang, X. (2021). Regularizing hyperspectral and multispectral image fusion by CNN denoiser. IEEE Transactions on Neural Networks and Learning Systems, 32(3), 1124–1135. https://doi.org/10.1109/TNNLS.2020.2980398

    Article  MATH  Google Scholar 

  • Dian, R., Shan, T., He, W., & Liu, H. (2023). Spectral super-resolution via model-guided cross-fusion network. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2023.3238506

    Article  MATH  Google Scholar 

  • Dong, W., Fu, F., Shi, G., Cao, X., Wu, J., Li, G., & Li, X. (2016). Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing, 25(5), 2337–2352. https://doi.org/10.1109/TIP.2016.2542360

    Article  MathSciNet  MATH  Google Scholar 

  • Elizar, E., Zulkifley, M. A., Muharar, R., Zaman, M. H. M., & Mustaza, S. M. (2022). A review on multiscale-deep-learning applications. Sensors, 22(19), 7384. https://doi.org/10.3390/s22197384

    Article  MATH  Google Scholar 

  • Elliethy, A., & Aly, H.A. (2017). Fast near infrared fusion-based adaptive enhancement of visible images. in 2017 IEEE global conference on signal and information processing (globalsip), pp. 156–160.

  • Feng, K., Zeng, H., Zhao, Y., Kong, S. G., & Bu, Y. (2024). Unsupervised spectral demosaicing with lightweight spectral attention networks. IEEE Transactions on Image Processing, 33, 1655–1669. https://doi.org/10.1109/TIP.2024.3364064

    Article  MATH  Google Scholar 

  • Fu, Q. , Souza, M. , Choi, E. , Shin, S. , Baek, S. H., & Heidrich, W. (2024). Limitations of data-driven spectral reconstruction—optics-aware analysis and mitigation.

  • Galliani, S. , Lanaras, C. , Marmanis, D. , Baltsavias, E., & Schindler, K. (2017). Learned spectral super-resolution. arXiv:1703.09470,

  • Han, X., Yu, J., Luo, J., & Sun, W. (2019). Reconstruction from multispectral to hyperspectral image using spectral library-based dictionary learning. IEEE Transactions on Geoscience and Remote Sensing, 57(3), 1325–1335. https://doi.org/10.1109/TGRS.2018.2866054

    Article  MATH  Google Scholar 

  • He, K. , Zhang, X. , Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. in 2015 IEEE international conference on computer vision (ICCV), pp. 1026–1034.

  • He, K. , Zhang, X. , Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. in 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp. 770–778.

  • Hu, W., Huang, Y., Wei, L., Zhang, F., & Li, H. (2015). Deep convolutional neural networks for hyperspectral image classification. Journal of Sensors, 2015, 1–12. https://doi.org/10.1155/2015/258619

    Article  MATH  Google Scholar 

  • Jia, Y. , Zheng, Y. , Gu, L. , Subpa-Asa, A. , Lam, A. , Sato, Y., Sato, I. (2017). From RGB to spectrum for natural scenes via manifold-based mapping. in 2017 IEEE international conference on computer vision (ICCV), pp. 4715–4723.

  • Jiaojiao Li., S., Du , Wu, C. , Leng, Y. , Song, R., & Li, Y. (2022). DRCR Net: Dense residual channel re-calibration network with non-local purification for spectral super resolution. in 2022 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp. 1258–1267.

  • Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. CoRRabs/1412.6980

  • Li, J. , Wu, C. , Song, R. , Li, Y., & Liu, F. (2020). Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images. in 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp. 1894–1903.

  • Li, J., Du, Q., Li, Y., & Li, W. (2018). Hyperspectral image classification with imbalanced data based on orthogonal complement subspace projection. IEEE Transactions on Geoscience and Remote Sensing, 56(7), 3838–3851. https://doi.org/10.1109/TGRS.2018.2813366

    Article  MATH  Google Scholar 

  • Li, T., & Gu, Y. (2021). Progressive spatial-spectral joint network for hyperspectral image reconstruction. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–14. https://doi.org/10.1109/TGRS.2021.3079969

    Article  MATH  Google Scholar 

  • Li, J., Huang, X., & Tu, L. (2022). WHU-OHS: A benchmark dataset for large-scale hersepctral image classification. International Journal of Applied Earth Observation and Geoinformation, 113, 103022. https://doi.org/10.1016/j.jag.2022.103022

    Article  Google Scholar 

  • Li, J., Wu, C., Song, R., Xie, W., Ge, C., Li, B., & Li, Y. (2021). Hybrid 2-D-3-D deep residual attentional network with structure tensor constraints for spectral super-resolution of RGB images. IEEE Transactions on Geoscience and Remote Sensing, 59(3), 2321–2335. https://doi.org/10.1109/TGRS.2020.3004934

    Article  MATH  Google Scholar 

  • Mahesh, S., Jayas, D., Paliwal, J., & White, N. (2015). Hyperspectral imaging to classify and monitor quality of agricultural materials. Journal of Stored Products Research, 61, 17–26. https://doi.org/10.1016/j.jspr.2015.01.006

    Article  Google Scholar 

  • Mei, S., Geng, Y., Hou, J., & Du, Q. (2022). Learning hyperspectral images from RGB images via a coarse-to-fine CNN. Science China Information Sciences, 65(5), 1–14. https://doi.org/10.1007/s11432-020-3102-9

    Article  MATH  Google Scholar 

  • Monno, Y., Teranaka, H., Yoshizaki, K., Tanaka, M., & Okutomi, M. (2019). Single-sensor RGB-NIR imaging: High-quality system design and prototype implementation. IEEE Sensors Journal, 19(2), 497–507. https://doi.org/10.1109/JSEN.2018.2876774

    Article  MATH  Google Scholar 

  • Pan, Z., Healey, G., Prasad, M., & Tromberg, B. (2003). Face recognition in hyperspectral images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12), 1552–1560. https://doi.org/10.1109/TPAMI.2003.1251148

    Article  MATH  Google Scholar 

  • Paoletti, M. E., Haut, J. M., Fernandez-Beltran, R., Plaza, J., Plaza, A. J., & Pla, F. (2019). Deep pyramidal residual networks for spectral-spatial hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 57(2), 740–754. https://doi.org/10.1109/TGRS.2018.2860125

    Article  MATH  Google Scholar 

  • Poli, D., & Toutin, T. (2012). Review of developments in geometric modelling for high resolution satellite pushbroom sensors. The Photogrammetric Record, 27(137), 58–73. https://doi.org/10.1111/j.1477-9730.2011.00665.x

    Article  Google Scholar 

  • Ravikanth, L., Jayas, D. S., White, N. D., Fields, P. G., & Sun, D. W. (2017). Extraction of spectral information from hyperspectral data and application of hyperspectral imaging for food and agricultural products. Food and Bioprocess Technology, 10(1), 1–33. https://doi.org/10.1007/s11947-016-1817-8

    Article  Google Scholar 

  • Roy, S. K., Manna, S., Song, T., & Bruzzone, L. (2021). Attention-based adaptive spectral-spatial kernel resnet for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 59(9), 7831–7843. https://doi.org/10.1109/TGRS.2020.3043267

    Article  Google Scholar 

  • Shi, Z. , Chen, C. , Xiong, Z. , Liu, D., & Wu, F. (2018). HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. in 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp. 1052–10528.

  • Song, W., Li, S., Fang, L., & Lu, T. (2018). Hyperspectral image classification with deep feature fusion network. IEEE Transactions on Geoscience and Remote Sensing, 56(6), 3173–3184. https://doi.org/10.1109/TGRS.2018.2794326

    Article  MATH  Google Scholar 

  • Wald, L. (2002). Data Fusion: Definitions and architectures: Fusion of images of different spatial resolutions. Presses des MINES.

    MATH  Google Scholar 

  • Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  MATH  Google Scholar 

  • Wei, Y., Yuan, Q., Shen, H., & Zhang, L. (2017). Boosting the accuracy of multispectral image Pansharpening by learning a deep residual network. IEEE Geoscience and Remote Sensing Letters, 14(10), 1795–1799. https://doi.org/10.1109/LGRS.2017.2736020

    Article  MATH  Google Scholar 

  • Wu, J. , Aeschbacher, J., & Timofte, R. (2017). In Defense of Shallow Learned Spectral Reconstruction from RGB Images. in 2017 IEEE international conference on computer vision workshops (ICCVW), pp. 471–479.

  • Yokoya, N., & Iwasaki, A. (2014). Airborne unmixing-based hyperspectral super-resolution using RGB imagery. in 2014 IEEE geoscience and remote sensing symposium, pp. 2653–2656.

  • Yuhas, R.H. , Goetz, A.F.H., & Boardman, J.W. (1992). Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (sam) algorithm. Jpl, summaries of the third annual jpl airborne geoscience workshop. volume 1: Aviris workshop. https://api.semanticscholar.org/CorpusID:126879175

  • Zhang, J. , Su, R. , Ren, W. , Fu, Q., & Nie, Y. (2021). Learnable reconstruction methods from RGB images to hyperspectral imaging: A survey. arXiv:2106.15944

  • Zhang, L., Lang, Z., Wang, P., Wei, W., Liao, S., Shao, L., & Zhang, Y. (2020). Pixel-aware deep function-mixture network for spectral super-resolution. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 12821–12828. https://doi.org/10.1609/aaai.v34i07.6978

    Article  Google Scholar 

  • Zhang, L., Wei, W., Bai, C., Gao, Y., & Zhang, Y. (2018). Exploiting clustering manifold structure for hyperspectral imagery super-resolution. IEEE Transactions on Image Processing, 27(12), 5969–5982. https://doi.org/10.1109/TIP.2018.2862629

    Article  MathSciNet  MATH  Google Scholar 

  • Zhao, Y. , Po, L. M. , Yan, Q. , Liu, W., & Lin, T. (2020). Hierarchical regression network for spectral reconstruction from RGB images. in 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp. 1695–1704.

  • Zheng, Z., Zhong, Y., Ma, A., & Zhang, L. (2020). FPGA: Fast patch-free global learning framework for fully end-to-end hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 58(8), 5612–5626. https://doi.org/10.1109/TGRS.2020.2967821

    Article  MATH  Google Scholar 

  • Zhong, Z., Li, J., Luo, Z., & Chapman, M. (2018). Spectral-spatial residual network for hyperspectral image classification: A 3-D deep learning framework. IEEE Transactions on Geoscience and Remote Sensing, 56(2), 847–858. https://doi.org/10.1109/TGRS.2017.2755542

    Article  MATH  Google Scholar 

  • Zhu, L., Chen, Y., Ghamisi, P., & Benediktsson, J. A. (2018). Generative adversarial networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 56(9), 5046–5063. https://doi.org/10.1109/TGRS.2018.2805286

    Article  Google Scholar 

Download references

Funding

No funding is used for this work.

Author information

Author notes

  1. Hussein A. Aly, Tarek S. Ghoniemy, and Mohamed A. Elshafey have contributed equally to this work.

Authors and Affiliations

  1. Technical Research Center of Egyptian Armed Forces, Cairo, 11765, Egypt

    Ahmed R. El-gabri

  2. Department of Computer Engineering and A.I., Military Technical College, Kobry Elkobbah, Cairo, 11765, Egypt

    Hussein A. Aly, Tarek S. Ghoniemy & Mohamed A. Elshafey

Authors
  1. Ahmed R. El-gabri
    View author publications

    Search author on:PubMed Google Scholar

  2. Hussein A. Aly
    View author publications

    Search author on:PubMed Google Scholar

  3. Tarek S. Ghoniemy
    View author publications

    Search author on:PubMed Google Scholar

  4. Mohamed A. Elshafey
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Ahmed R. El-gabri: Methodology, Software, Data Curation, Writing—Original Draft, Visualization, Formal analysis. Tarek S. Ghoniemy: Validation, Writing—Review & Editing, Supervision. Mohamed A. Elshafey: Validation, Investigation, Resources, Writing—Review, Conceptualization & Editing, Supervision. Hussein A. Aly: Conceptualization, Validation, Formal analysis, Investigation, Resources, Writing—Review & Editing, Supervision, Project administration. All authors reviewed and approved the final version of the manuscript before submission for publication.

Corresponding author

Correspondence to Ahmed R. El-gabri.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Ethics Approval

Not applicable.

Additional information

Communicated by Ayan Chakrabarti.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 388 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

El-gabri, A.R., Aly, H.A., Ghoniemy, T.S. et al. DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution. Int J Comput Vis 133, 1499–1531 (2025). https://doi.org/10.1007/s11263-024-02238-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s11263-024-02238-w

Keywords

Profiles

  1. Ahmed R. El-gabri View author profile