基于注意力残差网络的快照式多光谱相机图像重构

闫纲琦; 梁宗林; 宋延嵩; 董科研; 张博; 刘天赐; 张雷; 王岩柏

doi:10.37188/CO.2023-0196

Article Contents

Chinese Optics > 2024 > Accepted Manuscript

YAN Gang-qi, LIANG Zong-lin, SONG Yan-song, DONG Ke-yan, ZHANG Bo, LIU Tian-ci, ZHANG LEI, WANG Yan-bo. Reconstruction of snapshot multispectral camera images based on an attention residual network[J]. Chinese Optics. doi: 10.37188/CO.2023-0196

Citation:

PDF( 8459 KB)

Reconstruction of snapshot multispectral camera images based on an attention residual network

doi: 10.37188/CO.2023-0196

1.
School of Opto-Electronic Engineering, Changchun University of Science and Technology, Changchun 130022, China
2.
Institute of Space Optoelectronics Technology, Changchun University of Science and Technology, Changchun 130022, China
3.
Peng Cheng Laboratory, Shenzhen 518052, China

Funds: Supported by National key research and development program (No. 2022YFB3902500); National key research and development program (No. 2021YFA0718804); National Science Foundation of China (No. 62305032)

More Information

Corresponding author: songyansong2006@126.com
Available Online: 01 Feb 2024

Abstract

Abstract

With the rapid advancement of spectral imaging technology, the use of multispectral filter array (MSFA) to collect the spatial and spectral information of multispectral images has become a research hotspot. The uses of the original data are limited because of its low sampling rate and strong spectral inter-correlation for reconstruction. Therefore, this paper proposes a multi-branch attention residual network model for spatial-spectral association based on an 8-band 4 × 4 MSFA with all-pass bands. First, the multi-branch model was used to learn the image features after interpolation in each band; second, the feature information of the eight bands and the all-pass band were united by the spatial channel attention model designed in this paper, and the application of multi-layer convolution and the convolutional attention module and the use of residual compensation effectively compensated the color difference of each band and enriched the edge texture-related feature information; finally, the preliminary interpolated full-pass band and the rest of the band feature information were used in feature learning by residual dense blocks without batch normalization on the spatial and spectral correlation of multispectral images to match the spectral information of each band. Experimental results show that the peak signal-to-noise ratio, structural similarity, and spectral angular similarity of the test image under the D65 light source outperform the state-of-the-art deep learning method by 3.46%, 0.27%, and 6%, respectively; in conclusion, this method not only reduces artifacts but also obtains more texture details.
- multispectral filter array,
- image reconstruction,
- spatial-spectral combination,
- residual network,
- deep learning

FullText(HTML)

References(38)

References

[1]	JUNIOR J D D, BACKES A R, ESCARPINATI M C. Detection of control points for UAV-multispectral sensed data registration through the combining of feature descriptors[C]. Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, SciTePress.
[2]	MANGAI U G, SAMANTA S, DAS S, et al. A hierarchical multi-classifier framework for landform segmentation using multi-spectral satellite images - a case study over the Indian subcontinent[C]. 2010 Fourth Pacific-Rim Symposium on Image and Video Technology, IEEE, 2010: 306-313.
[3]	LIU CH H, LIU W, LU X ZH, et al. Application of multispectral imaging to determine quality attributes and ripeness stage in strawberry fruit[J]. PLoS One, 2014, 9(2): e87818. doi: 10.1371/journal.pone.0087818
[4]	CHEN I T, LIN H Y. Detection, counting and maturity assessment of cherry tomatoes using multi-spectral images and machine learning techniques[C]. Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, SciTePress, 2020: 759-766.
[5]	CHANG K, LI H X, TAN Y F, et al. A two-stage convolutional neural network for joint demosaicking and super-resolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(7): 4238-4254. doi: 10.1109/TCSVT.2021.3129201
[6]	SHINODA K, OGAWA S, YANAGI Y, et al. Multispectral filter array and demosaicking for pathological images[C]. 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), IEEE, 2015: 697-703.
[7]	ZENTENO O, TREUILLET S, LUCAS Y. 3D cylinder pose estimation by maximization of binary masks similarity: a simulation study for multispectral endoscopy image registration[C]. Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, SciTePress, 2019: 857-864.
[8]	李云辉. 压缩光谱成像系统中物理实现架构研究综述[J]. 中国光学(中英文),2022,15(5):929-945. doi: 10.37188/CO.2022-0104 LI Y H. Review of physical implementation architecture in compressive spectral imaging system[J]. Chinese Optics, 2022, 15(5): 929-945. doi: 10.37188/CO.2022-0104
[9]	ZHUANG L N, NG M K, FU X Y, et al. Hy-demosaicing: hyperspectral blind reconstruction from spectral subsampling[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-15. doi: 10.1109/TGRS.2021.3102136
[10]	ZHANG J CH, CHEN J L, YU H W, et al. Polarization image demosaicking via nonlocal sparse tensor factorization[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-10. doi: 10.1109/TGRS.2021.3093903
[11]	RATHI V, GOYAL P. Multispectral image demosaicking based on novel spectrally localized average images[J]. IEEE Signal Processing Letters, 2022, 29: 449-453. doi: 10.1109/LSP.2021.3139581
[12]	RATHI V, GOYAL P. Generic multispectral demosaicking using spectral correlation between spectral bands and pseudo-panchromatic image[J]. Signal Processing:Image Communication, 2023, 110: 116893. doi: 10.1016/j.image.2022.116893
[13]	LIU SH M, ZHANG Y G, CHEN J, et al. A deep joint network for multispectral demosaicking based on pseudo-panchromatic images[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(4): 622-635. doi: 10.1109/JSTSP.2022.3172865
[14]	ZHANG Y, SUN W J, CHEN ZH ZH. Joint image demosaicking and denoising with mutual guidance of color channels[J]. Signal Processing, 2022, 200: 108674. doi: 10.1016/j.sigpro.2022.108674
[15]	AGGARWAL H K, MAJUMDAR A. Single-sensor multi-spectral image demosaicing algorithm using learned interpolation weights[C]. 2014 IEEE Geoscience and Remote Sensing Symposium, IEEE, 2014: 2011-2014.
[16]	CHINI M, CHIANCONE A, STRAMONDO S. Scale Object Selection (SOS) through a hierarchical segmentation by a multi-spectral per-pixel classification[J]. Pattern Recognition Letters, 2014, 49: 214-223. doi: 10.1016/j.patrec.2014.07.012
[17]	MIHOUBI S, LOSSON O, MATHON B, et al. Multispectral demosaicing using pseudo-panchromatic image[J]. IEEE Transactions on Computational Imaging, 2017, 3(4): 982-995. doi: 10.1109/TCI.2017.2691553
[18]	齐海超, 宋延嵩, 张博, 等. 基于改进引导滤波器的多光谱去马赛克方法[J]. 中国光学(中英文),2023,16(5):1056-1065. doi: 10.37188/CO.2022-0231 QI H CH, SONG Y S, ZHANG B, et al. Multispectral demosaicing method based on an improved guided filter[J]. Chinese Optics, 2023, 16(5): 1056-1065. doi: 10.37188/CO.2022-0231
[19]	CHEN L K, ZHAO Y Q, CHAN J C W, et al. Histograms of oriented mosaic gradients for snapshot spectral image description[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 183: 79-93. doi: 10.1016/j.isprsjprs.2021.10.018
[20]	XIONG F CH, ZHOU J, QIAN Y T. Material based object tracking in hyperspectral videos: benchmark and algorithms[Z]. arXiv: 1812.04179, 2019.
[21]	WANG X H, CHEN J, WEI Q, et al. Hyperspectral image super-resolution via deep prior regularization with parameter estimation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(4): 1708-1723. doi: 10.1109/TCSVT.2021.3078559
[22]	HU J, JIA X P, LI Y S, et al. Hyperspectral image super-resolution via intrafusion network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(10): 7459-7471. doi: 10.1109/TGRS.2020.2982940
[23]	KUMAR S P P, PETER K J, KINGSLY C S. De-noising and demosaicking of Bayer image using deep convolutional attention residual learning[J]. Multimedia Tools and Applications, 2023, 82(13): 20323-20342. doi: 10.1007/s11042-023-14334-z
[24]	KURNIAWAN E, PARK Y, LEE S. Noise-resistant demosaicing with deep image prior network and random RGBW color filter array[J]. Sensors, 2022, 22(5): 1767. doi: 10.3390/s22051767
[25]	SHINODA K, YOSHIBA S, HASEGAWA M. Deep demosaicking for multispectral filter arrays[Z]. arXiv: 1808.08021, 2018.
[26]	PAN ZH H, LI B P, BAO Y Z, et al. Deep panchromatic image guided residual interpolation for multispectral image demosaicking[C]. 2019 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), IEEE, 2019: 1-5.
[27]	FENG K, ZHAO Y Q, CHAN J C W, et al. Mosaic convolution-attention network for demosaicing multispectral filter array images[J]. IEEE Transactions on Computational Imaging, 2021, 7: 864-878. doi: 10.1109/TCI.2021.3102052
[28]	DIJKSTRA K, VAN DE LOOSDRECHT J, SCHOMAKER L R B, et al. Hyperspectral demosaicking and crosstalk correction using deep learning[J]. Machine Vision and Applications, 2019, 30(1): 1-21. doi: 10.1007/s00138-018-0965-4
[29]	WANG ZH, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/TIP.2003.819861
[30]	KRUSE F A, LEFKOFF A B, BOARDMAN J W, et al. The spectral image processing system (SIPS)-interactive visualization and analysis of imaging spectrometer data[J]. AIP Conference Proceedings, 1993, 283(1): 192-201.
[31]	WANG X T, YU K, WU SH X, et al. ESRGAN: Enhanced super-resolution generative adversarial networks[C]. European Conference on Computer Vision, Springer, 2018: 63-79.
[32]	ZHAO H, GALLO O, FROSIO I, et al. Loss functions for image restoration with neural networks[J]. IEEE Transactions on Computational Imaging, 2017, 3(1): 47-57. doi: 10.1109/TCI.2016.2644865
[33]	KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2016.
[34]	KINGMA D P, BA L J. Adam: A method for stochastic optimization[C]. 3rd International Conference on Learning Representations (ICLR), ICLR, 2017.
[35]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]. FERRARI V, HEBERT M, SMINCHISESCU C, et al. Computer Vision – ECCV 2018. Cham: Springer, 2018: 3-19.
[36]	YASUMA F, MITSUNAGA T, ISO D, et al. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum[J]. IEEE Transactions on Image Processing, 2010, 19(9): 2241-2253. doi: 10.1109/TIP.2010.2046811
[37]	MONNO Y, KIKUCHI S, TANAKA M, et al. A practical one-shot multispectral imaging system using a single image sensor[J]. IEEE Transactions on Image Processing, 2015, 24(10): 3048-3059. doi: 10.1109/TIP.2015.2436342
[38]	HU J, SHEN L, ALBANIE S, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023. doi: 10.1109/TPAMI.2019.2913372