Super-resolution reconstruction for colorectal endoscopic images based on a residual network
-
摘要:
针对结直肠镜图像分辨率偏低、纹理信息偏少和细节模糊等缺点，提出了一种基于残差注意力网络的图像超分辨率重建算法SMRAN，选取结直肠息肉内窥镜图像数据集PolypsSet中的部分图像作为原始数据进行实验。首先，使用卷积网络提取低分辨率图像的浅层特征；其次，设计Res-Sobel结构对图像边缘特征进行增强；然后，通过引入不同大小的卷积核，设计多尺度特征融合模块(Multi-Scale feature Extraction Block, MEB)，自适应地提取不同尺度的特征，从而得到有效的图像信息，并通过残差注意力网络将Res-Sobel模块和多尺度特征融合模块MEB进行连接；最后，通过亚像素卷积层对图像进行重建，得到最终的高分辨率图像。在尺度因子为×4时，网络在测试集上的测试结果如下: 峰值信噪比PSNR为34.25 dB，结构相似性SSIM为0.8675。实验结果表明，与传统的双三次插值算法及常用的SRCNN、RCAN等深度学习算法相比，本文提出的SMRAN对结直肠内窥镜图像具有更好的超分辨率重建效果。
Abstract:In this paper, an image super-resolution reconstruction multi-scale algorithm based on a residual attention network (SMRAN) is proposed to solve the problems caused by low resolutions, less texture information and blurred details in colorectal endoscopic images. Images from the colorectal polyp endoscope image dataset PolypsSet are selected as the raw data for these experiments. A convolutional network is built to extract the shallow features of the low-resolution image and a Res-Sobel block is designed to enhance its edge features. A multi-scale feature fusion block MEB is designed by introducing convolution kernels of different sizes to adaptively extract image features of different scales and obtain effective image information. The Res-Sobel block and multi-scale feature fusion module block MEB are connected through the residual attention network. Finally, a high-resolution image is reconstructed at the sub-pixel convolution layer. When the amplification factor is ×4, the performance of the proposed algorithm on the test set are as follows: the peak signal-to-noise ratio (PSNR) is 34.25 dB and the structural similarity (SSIM) is 0.8675. Compared with the traditional bicubic interpolation algorithm and commonly used deep learning algorithms such as SRCNN and RCAN, the proposed SMRAN algorithm shows better super-resolution reconstruction results on colorectal endoscopic images.
-
图 8 采用不同超分辨率算法的结直肠息肉内窥图像的重建效果对比图
Figure 8. Comparison of reconstruction effects of endoscopic images of colorectal polyps using different super-resolution algorithms
表 1 测试集上不同算法的PSNR值
Table 1. PSNR values of different algorithms on the testing set
(Unit: dB) 算法 PSNR(dB) ×2 ×3 ×4 Bicubic 33.85 31.94 29.91 SRCNN 36.70 34.53 32.04 FSRCNN 37.63 35.22 32.23 EDSR 37.34 35.25 32.13 ESPCN 36.75 34.78 31.38 RCAN 39.04 35.63 33.86 本文算法 39.69 36.92 34.25 
下载: 导出CSV
表 2 测试集上不同算法的SSIM值
Table 2. SSIM values of different algorithms on the testing set
算法 SSIM ×2 ×3 ×4 Bicubic 0.9121 0.8824 0.8103 SRCNN 0.9400 0.8983 0.8642 FSRCNN 0.9382 0.9132 0.8660 EDSR 0.9325 0.9158 0.8401 ESPCN 0.9392 0.9003 0.8566 RCAN 0.9483 0.9182 0.8667 本文算法 0.9559 0.9249 0.8675
下载: 导出CSV
表 3 不同损失函数的PSNR和SSIM值
Table 3. PSNR and SSIM values for different loss functions
损失函数 PSNR(dB) SSIM ${L_1}$ 34.27 0.8664 ${L_1}$+MS_SSIM 34.25 0.8675
下载: 导出CSV
表 4 各模块对性能的影响
Table 4. The impact of each module on performance
Res-Sobel Block MEB CBAM PSNR(dB)/SSIM √ √ 33.33/0.8577 √ √ 33.39/0.8601 √ √ 33.47/0.8609 √ √ √ 34.25/0.8675
下载: 导出CSV
表 5 Kvasir-SEG数据集上不同算法的PSNR值
Table 5. PSNR values of different algorithms on the Kvasir-SEG dataset
(Unit: dB) 算法 PSNR(dB) ×2 ×3 ×4 Bicubic 37.35 33.86 31.78 SRCNN 39.98 35.98 33.23 FSRCNN 40.62 36.57 33.68 EDSR 40.94 36.69 33.99 ESPCN 39.71 35.91 33.01 RCAN 41.58 37.62 34.23 本文算法 41.80 37.81 34.56
下载: 导出CSV
表 6 Kvasir-SEG数据集上不同算法的SSIM值
Table 6. SSIM values of different algorithms on the Kvasir-SEG dataset
算法 SSIM ×2 ×3 ×4 Bicubic 0.9776 0.9469 0.9079 SRCNN 0.9833 0.9658 0.9242 FSRCNN 0.9859 0.9672 0.9256 EDSR 0.9867 0.9669 0.9219 ESPCN 0.9886 0.9710 0.9396 RCAN 0.9859 0.9702 0.9447 本文算法 0.9884 0.9714 0.9456
下载: 导出CSV
-
[1] 刘爽, 田兆星, 李浩然, 等. 一种基于改进YOLOv5s网络的结直肠腺瘤实时检测方法[J]. 河北大学学报(自然科学版),2022,42(3):327-336.LIU SH, TIAN ZH X, LI H R, et al. A real-time method for colorectal adenoma detection based on an improved YOLOv5s network[J]. Journal of Hebei University (Natural Science Edition), 2022, 42(3): 327-336. (in Chinese) [2] 白瑞峰, 江山, 孙海江, 等. 基于编码解码结构的微血管减压图像实时语义分割[J]. 中国光学(中英文),2022,15(5):1055-1065. doi: 10.37188/CO.2022-0120BAI R F, JIANG SH, SUN H J, et al. Real-time semantic segmentation of microvascular decompression images based on encoder-decoder structure[J]. Chinese Optics, 2022, 15(5): 1055-1065. (in Chinese) doi: 10.37188/CO.2022-0120 [3] 陈晓冬, 艾大航, 张佳琛, 等. Gabor滤波融合卷积神经网络的路面裂缝检测方法[J]. 中国光学,2020,13(6):1293-1301. doi: 10.37188/CO.2020-0041CHEN X D, AI D H, ZHANG J CH, et al. Gabor filter fusion network for pavement crack detection[J]. Chinese Optics, 2020, 13(6): 1293-1301. (in Chinese) doi: 10.37188/CO.2020-0041 [4] 左超, 陈钱. 分辨率、超分辨率与空间带宽积拓展——从计算光学成像角度的一些思考[J]. 中国光学(中英文),2022,15(6):1105-1166. doi: 10.37188/CO.2022-0105ZUO CH, CHEN Q. Resolution, super-resolution and spatial bandwidth product expansion——some thoughts from the perspective of computational optical imaging[J]. Chinese Optics, 2022, 15(6): 1105-1166. (in Chinese) doi: 10.37188/CO.2022-0105 [5] GU X G, ZHOU F X, CHEN R F, et al. . Endoscopic single image super-resolution based on transformer and convolutional neural network[C]. 7th International Conference on Intelligent Life System Modelling, Image Processing and Analysis, Springer, 2021: 24-32. [6] YANG X R, CHEN Y, TAO R, et al. . Endoscopic image deblurring and super-resolution reconstruction based on deep learning[C]. 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE), IEEE, 2020: 168-172. [7] TURAN M. A generative adversarial network based super-resolution approach for capsule endoscopy images[J]. Medicine Science, 2021, 10(3): 1002-1007. doi: 10.5455/medscience.2021.06.218 [8] TAŞ M, YILMAZ B. Super resolution convolutional neural network based pre-processing for automatic polyp detection in colonoscopy images[J]. Computers &Electrical Engineering, 2021, 90: 106959. [9] DONG CH, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. doi: 10.1109/TPAMI.2015.2439281 [10] 雷茂, 郭锋, 秦明伟. 基于改进POCS算法的太赫兹图像超分辨率重建[J]. 传感器与微系统,2022,41(3):122-125. doi: 10.13873/J.1000-9787(2022)03-0122-04LEI M, GUO F, QIN M W. Super-resolution reconstruction for terahertz images based on improved POCS algorithm[J]. Transducer and Microsystem Technologies, 2022, 41(3): 122-125. (in Chinese) doi: 10.13873/J.1000-9787(2022)03-0122-04 [11] 孙琰玥, 何小海, 陈为龙. 小波局部适应插值的图像超分辨率重建[J]. 计算机工程,2010,36(13):183-185. doi: 10.3969/j.issn.1000-3428.2010.13.065SUN Y Y, HE X H, CHEN W L. Image super-resolution reconstruction of wavelet local-adaptation interpolation[J]. Computer Engineering, 2010, 36(13): 183-185. (in Chinese) doi: 10.3969/j.issn.1000-3428.2010.13.065 [12] 庞博, 张旭东, 徐小红. 自适应图像插值在超分辨率图像重建中的应用[J]. 合肥工业大学学报(自然科学版),2006,29(7):825-829.PANG B, ZHANG X D, XU X H. Super-resolution image reconstruction using adaptive interpolation method[J]. Journal of Hefei University of Technology (Natural Science), 2006, 29(7): 825-829. (in Chinese) [13] 陈楠, 张标. 多尺度半耦合卷积稀疏编码的遥感影像超分辨率重建[J]. 计算机辅助设计与图形学学报,2022,34(3):382-391.CHEN N, ZHANG B. Multi-scale semi-coupled convolutional sparse coding for the super-resolution reconstruction of remote sensing image[J]. Journal of Computer-Aided Design &Computer Graphics, 2022, 34(3): 382-391. (in Chinese) [14] 李方玗, 贾晓芬, 赵佰亭, 等. 高效多注意力特征融合的图像超分辨率重建算法[J/OL]. 小型微型计算机系统: 1-9[2022-11-25]. http://kns.cnki.net/kcms/detail/21.1106.TP.20221111.1852.006.html.LI F Y, JIA X F, ZHAO B T, et al. . Efficient multi-attention feature fusion for image super-resolution reconstruction algorithms[J/OL]. Journal of Chinese Computer Systems: 1-9[2022-11-25]. http://kns.cnki.net/kcms/detail/21.1106.TP.20221111.1852.006.html. (in Chinese) [15] 林敏强, 赵曈, 祝明欣. 基于残差注意力的合成孔径雷达图像超分辨率算法[J]. 电子产品世界,2022,29(11):33-34.LIN M Q, ZHAO T, ZHU M X. SAR image super-resolution algorithm based on residual attention[J]. Electronic Engineering &Product World, 2022, 29(11): 33-34. (in Chinese) [16] 须颖, 刘帅, 邵萌, 等. 一种多尺度GAN的低剂量CT超分辨率重建方法[J]. 西安电子科技大学学报,2022,49(2):228-236. doi: 10.19665/j.issn1001-2400.2022.02.026XU Y, LIU SH, SHAO M, et al. Multi-scale generation antagonistic network for the low-dose CT images super-resolution reconstruction algorithm[J]. Journal of Xidian University, 2022, 49(2): 228-236. (in Chinese) doi: 10.19665/j.issn1001-2400.2022.02.026 [17] LIM B, SON S, KIM H, et al. . Enhanced deep residual networks for single image super-resolution[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, IEEE, 2017: 1132-1140. [18] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016: 1646-1654. [19] ZHANG Y L, LI K P, LI K, et al. . Image super-resolution using very deep residual channel attention networks[C]. Proceedings of the 15th European Conference on Computer Vision, Springer, 2018: 294-310. [20] LEDIG C, THEIS L, HUSZÁR F, et al. . Photo-realistic single image super-resolution using a generative adversarial network[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2017: 105-114. [21] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]. 3rd International Conference on Learning Representations, ICLR, 2015. [22] 王玉玺, 陈健美. 基于自适应Canny算子和多方向Sobel算子的虹膜边缘检测算法[J]. 计算机与数字工程,2020,48(11):2744-2749. doi: 10.3969/j.issn.1672-9722.2020.11.040WANG Y X, CHEN J M. Iris edge detection algorithm based on adaptive canny operator and multi-directional Sobel operator[J]. Computer &Digital Engineering, 2020, 48(11): 2744-2749. (in Chinese) doi: 10.3969/j.issn.1672-9722.2020.11.040 [23] LIU S R, TANG X Y, WANG D. Facial expression recognition based on Sobel operator and improved CNN-SVM[C]. 2020 IEEE 3rd International Conference on Information Communication and Signal Processing (ICICSP), IEEE, 2020: 236-240. [24] HAO F, XU D SH, CHEN D L, et al. Sobel operator enhancement based on eight-directional convolution and entropy[J]. International Journal of Information Technology, 2021, 13(5): 1823-1828. doi: 10.1007/s41870-021-00770-3 [25] WOO S, PARK J, LEE J Y, et al. . CBAM: convolutional block attention module[C]. Proceedings of the 15th European Conference on Computer Vision (ECCV), Springer, 2018: 3-19. [26] SHI W ZH, CABALLERO J, HUSZÁR F, et al. . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016: 1874-1883. [27] LI K D, FATHAN M I, PATEL K, et al. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations[J]. PLoS One, 2021, 16(8): e0255809. doi: 10.1371/journal.pone.0255809 [28] JHA D, SMEDSRUD P H, RIEGLER M A, et al. . Kvasir-SEG: a segmented polyp dataset[C]. 26th International Conference on Multimedia Modeling, Springer, 2020: 451-462. -
下载:
计量
- 文章访问数: 524
- HTML全文浏览量: 364
- PDF下载量: 217
- 被引次数: 0
