留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于残差网络的结直肠内窥镜图像超分辨率重建方法

郑跃坤 葛明锋 常智敏 董文飞

郑跃坤, 葛明锋, 常智敏, 董文飞. 基于残差网络的结直肠内窥镜图像超分辨率重建方法[J]. 中国光学(中英文), 2023, 16(5): 1022-1033. doi: 10.37188/CO.2022-0247
引用本文: 郑跃坤, 葛明锋, 常智敏, 董文飞. 基于残差网络的结直肠内窥镜图像超分辨率重建方法[J]. 中国光学(中英文), 2023, 16(5): 1022-1033. doi: 10.37188/CO.2022-0247
ZHENG Yue-kun, GE Ming-feng, CHANG Zhi-min, DONG Wen-fei. Super-resolution reconstruction for colorectal endoscopic images based on a residual network[J]. Chinese Optics, 2023, 16(5): 1022-1033. doi: 10.37188/CO.2022-0247
Citation: ZHENG Yue-kun, GE Ming-feng, CHANG Zhi-min, DONG Wen-fei. Super-resolution reconstruction for colorectal endoscopic images based on a residual network[J]. Chinese Optics, 2023, 16(5): 1022-1033. doi: 10.37188/CO.2022-0247

基于残差网络的结直肠内窥镜图像超分辨率重建方法

基金项目: 国家重点研发计划(No. 2021YFB3602200);苏州市科技计划项目(No. SZS201903)
详细信息
    作者简介:

    郑跃坤(1998—),男,广东汕头人,硕士研究生,2020年于中山大学获得学士学位,主要从事图像处理、深度学习等方面的研究。E-mail:645352858@qq.com

    葛明锋(1987—),男,江苏南通人,博士,副研究员,硕士生导师,主要从事高光谱、荧光显微成像方面研究。E-mail:gemf@sibet.ac.cn

    董文飞(1975—),男,吉林长春人,博士,研究员,博士生导师,1999年于中国科学院长春应用化学研究所获得高分子物理化学专业硕士学位,2004年于德国马普胶体界面所和波兹坦大学获得自然科学博士学位,主要从事纳米材料和技术在生物医用光子学领域的应用基础研究。E-mail: wenfeidong@126.com

  • 中图分类号: TP391.41

Super-resolution reconstruction for colorectal endoscopic images based on a residual network

Funds: Supported by the National Key R & D Program of China (No. 2021YFB3602200); Suzhou Science and Technology Plan Project (No.SZS201903)
More Information
  • 摘要:

    针对结直肠镜图像分辨率偏低、纹理信息偏少和细节模糊等缺点,提出了一种基于残差注意力网络的图像超分辨率重建算法SMRAN,选取结直肠息肉内窥镜图像数据集PolypsSet中的部分图像作为原始数据进行实验。首先,使用卷积网络提取低分辨率图像的浅层特征;其次,设计Res-Sobel结构对图像边缘特征进行增强;然后,通过引入不同大小的卷积核,设计多尺度特征融合模块(Multi-Scale feature Extraction Block, MEB),自适应地提取不同尺度的特征,从而得到有效的图像信息,并通过残差注意力网络将Res-Sobel模块和多尺度特征融合模块MEB进行连接;最后,通过亚像素卷积层对图像进行重建,得到最终的高分辨率图像。在尺度因子为×4时,网络在测试集上的测试结果如下: 峰值信噪比PSNR为34.25 dB,结构相似性SSIM为0.8675。实验结果表明,与传统的双三次插值算法及常用的SRCNN、RCAN等深度学习算法相比,本文提出的SMRAN对结直肠内窥镜图像具有更好的超分辨率重建效果。

     

  • 图 1  SMRAN架构

    Figure 1.  Architecture of SMRAN

    图 2  Res-Sobel模块

    Figure 2.  Res-Sobel Block

    图 3  多尺度特征融合模块(MEB)

    Figure 3.  Multi-scale feature extraction block(MEB)

    图 4  CBAM注意力机制

    Figure 4.  CBAM attention mechanism

    图 5  SMRAB结构

    Figure 5.  SMRAB Structure

    图 6  验证集的PSNR曲线

    Figure 6.  PSNR curve of validation set

    图 7  验证集的SSIM曲线

    Figure 7.  SSIM curve of validation set

    图 8  采用不同超分辨率算法的结直肠息肉内窥图像的重建效果对比图

    Figure 8.  Comparison of reconstruction effects of endoscopic images of colorectal polyps using different super-resolution algorithms

    图 9  SMRAN模型对光学分辨率的提升效果

    Figure 9.  Improvement of optical resolution by SMRAN model

    表  1  测试集上不同算法的PSNR值

    Table  1.   PSNR values of different algorithms on the testing set (Unit: dB)

    算法PSNR(dB)
    ×2×3×4
    Bicubic33.8531.9429.91
    SRCNN36.7034.5332.04
    FSRCNN37.6335.2232.23
    EDSR37.3435.2532.13
    ESPCN36.7534.7831.38
    RCAN39.0435.6333.86
    本文算法39.6936.9234.25
    下载: 导出CSV

    表  2  测试集上不同算法的SSIM值

    Table  2.   SSIM values of different algorithms on the testing set

    算法SSIM
    ×2×3×4
    Bicubic0.91210.88240.8103
    SRCNN0.94000.89830.8642
    FSRCNN0.93820.91320.8660
    EDSR0.93250.91580.8401
    ESPCN0.93920.90030.8566
    RCAN0.94830.91820.8667
    本文算法0.95590.92490.8675
    下载: 导出CSV

    表  3  不同损失函数的PSNR和SSIM值

    Table  3.   PSNR and SSIM values for different loss functions

    损失函数PSNR(dB)SSIM
    ${L_1}$34.270.8664
    ${L_1}$+MS_SSIM34.250.8675
    下载: 导出CSV

    表  4  各模块对性能的影响

    Table  4.   The impact of each module on performance

    Res-Sobel BlockMEBCBAMPSNR(dB)/SSIM
    33.33/0.8577
    33.39/0.8601
    33.47/0.8609
    34.25/0.8675
    下载: 导出CSV

    表  5  Kvasir-SEG数据集上不同算法的PSNR值

    Table  5.   PSNR values of different algorithms on the Kvasir-SEG dataset (Unit: dB)

    算法PSNR(dB)
    ×2×3×4
    Bicubic37.3533.8631.78
    SRCNN39.9835.9833.23
    FSRCNN40.6236.5733.68
    EDSR40.9436.6933.99
    ESPCN39.7135.9133.01
    RCAN41.5837.6234.23
    本文算法41.8037.8134.56
    下载: 导出CSV

    表  6  Kvasir-SEG数据集上不同算法的SSIM值

    Table  6.   SSIM values of different algorithms on the Kvasir-SEG dataset

    算法SSIM
    ×2×3×4
    Bicubic0.97760.94690.9079
    SRCNN0.98330.96580.9242
    FSRCNN0.98590.96720.9256
    EDSR0.98670.96690.9219
    ESPCN0.98860.97100.9396
    RCAN0.98590.97020.9447
    本文算法0.98840.97140.9456
    下载: 导出CSV
  • [1] 刘爽, 田兆星, 李浩然, 等. 一种基于改进YOLOv5s网络的结直肠腺瘤实时检测方法[J]. 河北大学学报(自然科学版),2022,42(3):327-336.

    LIU SH, TIAN ZH X, LI H R, et al. A real-time method for colorectal adenoma detection based on an improved YOLOv5s network[J]. Journal of Hebei University (Natural Science Edition), 2022, 42(3): 327-336. (in Chinese)
    [2] 白瑞峰, 江山, 孙海江, 等. 基于编码解码结构的微血管减压图像实时语义分割[J]. 中国光学(中英文),2022,15(5):1055-1065. doi: 10.37188/CO.2022-0120

    BAI R F, JIANG SH, SUN H J, et al. Real-time semantic segmentation of microvascular decompression images based on encoder-decoder structure[J]. Chinese Optics, 2022, 15(5): 1055-1065. (in Chinese) doi: 10.37188/CO.2022-0120
    [3] 陈晓冬, 艾大航, 张佳琛, 等. Gabor滤波融合卷积神经网络的路面裂缝检测方法[J]. 中国光学,2020,13(6):1293-1301. doi: 10.37188/CO.2020-0041

    CHEN X D, AI D H, ZHANG J CH, et al. Gabor filter fusion network for pavement crack detection[J]. Chinese Optics, 2020, 13(6): 1293-1301. (in Chinese) doi: 10.37188/CO.2020-0041
    [4] 左超, 陈钱. 分辨率、超分辨率与空间带宽积拓展——从计算光学成像角度的一些思考[J]. 中国光学(中英文),2022,15(6):1105-1166. doi: 10.37188/CO.2022-0105

    ZUO CH, CHEN Q. Resolution, super-resolution and spatial bandwidth product expansion——some thoughts from the perspective of computational optical imaging[J]. Chinese Optics, 2022, 15(6): 1105-1166. (in Chinese) doi: 10.37188/CO.2022-0105
    [5] GU X G, ZHOU F X, CHEN R F, et al. . Endoscopic single image super-resolution based on transformer and convolutional neural network[C]. 7th International Conference on Intelligent Life System Modelling, Image Processing and Analysis, Springer, 2021: 24-32.
    [6] YANG X R, CHEN Y, TAO R, et al. . Endoscopic image deblurring and super-resolution reconstruction based on deep learning[C]. 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE), IEEE, 2020: 168-172.
    [7] TURAN M. A generative adversarial network based super-resolution approach for capsule endoscopy images[J]. Medicine Science, 2021, 10(3): 1002-1007. doi: 10.5455/medscience.2021.06.218
    [8] TAŞ M, YILMAZ B. Super resolution convolutional neural network based pre-processing for automatic polyp detection in colonoscopy images[J]. Computers &Electrical Engineering, 2021, 90: 106959.
    [9] DONG CH, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. doi: 10.1109/TPAMI.2015.2439281
    [10] 雷茂, 郭锋, 秦明伟. 基于改进POCS算法的太赫兹图像超分辨率重建[J]. 传感器与微系统,2022,41(3):122-125. doi: 10.13873/J.1000-9787(2022)03-0122-04

    LEI M, GUO F, QIN M W. Super-resolution reconstruction for terahertz images based on improved POCS algorithm[J]. Transducer and Microsystem Technologies, 2022, 41(3): 122-125. (in Chinese) doi: 10.13873/J.1000-9787(2022)03-0122-04
    [11] 孙琰玥, 何小海, 陈为龙. 小波局部适应插值的图像超分辨率重建[J]. 计算机工程,2010,36(13):183-185. doi: 10.3969/j.issn.1000-3428.2010.13.065

    SUN Y Y, HE X H, CHEN W L. Image super-resolution reconstruction of wavelet local-adaptation interpolation[J]. Computer Engineering, 2010, 36(13): 183-185. (in Chinese) doi: 10.3969/j.issn.1000-3428.2010.13.065
    [12] 庞博, 张旭东, 徐小红. 自适应图像插值在超分辨率图像重建中的应用[J]. 合肥工业大学学报(自然科学版),2006,29(7):825-829.

    PANG B, ZHANG X D, XU X H. Super-resolution image reconstruction using adaptive interpolation method[J]. Journal of Hefei University of Technology (Natural Science), 2006, 29(7): 825-829. (in Chinese)
    [13] 陈楠, 张标. 多尺度半耦合卷积稀疏编码的遥感影像超分辨率重建[J]. 计算机辅助设计与图形学学报,2022,34(3):382-391.

    CHEN N, ZHANG B. Multi-scale semi-coupled convolutional sparse coding for the super-resolution reconstruction of remote sensing image[J]. Journal of Computer-Aided Design &Computer Graphics, 2022, 34(3): 382-391. (in Chinese)
    [14] 李方玗, 贾晓芬, 赵佰亭, 等. 高效多注意力特征融合的图像超分辨率重建算法[J/OL]. 小型微型计算机系统: 1-9[2022-11-25]. http://kns.cnki.net/kcms/detail/21.1106.TP.20221111.1852.006.html.

    LI F Y, JIA X F, ZHAO B T, et al. . Efficient multi-attention feature fusion for image super-resolution reconstruction algorithms[J/OL]. Journal of Chinese Computer Systems: 1-9[2022-11-25]. http://kns.cnki.net/kcms/detail/21.1106.TP.20221111.1852.006.html. (in Chinese)
    [15] 林敏强, 赵曈, 祝明欣. 基于残差注意力的合成孔径雷达图像超分辨率算法[J]. 电子产品世界,2022,29(11):33-34.

    LIN M Q, ZHAO T, ZHU M X. SAR image super-resolution algorithm based on residual attention[J]. Electronic Engineering &Product World, 2022, 29(11): 33-34. (in Chinese)
    [16] 须颖, 刘帅, 邵萌, 等. 一种多尺度GAN的低剂量CT超分辨率重建方法[J]. 西安电子科技大学学报,2022,49(2):228-236. doi: 10.19665/j.issn1001-2400.2022.02.026

    XU Y, LIU SH, SHAO M, et al. Multi-scale generation antagonistic network for the low-dose CT images super-resolution reconstruction algorithm[J]. Journal of Xidian University, 2022, 49(2): 228-236. (in Chinese) doi: 10.19665/j.issn1001-2400.2022.02.026
    [17] LIM B, SON S, KIM H, et al. . Enhanced deep residual networks for single image super-resolution[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, IEEE, 2017: 1132-1140.
    [18] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016: 1646-1654.
    [19] ZHANG Y L, LI K P, LI K, et al. . Image super-resolution using very deep residual channel attention networks[C]. Proceedings of the 15th European Conference on Computer Vision, Springer, 2018: 294-310.
    [20] LEDIG C, THEIS L, HUSZÁR F, et al. . Photo-realistic single image super-resolution using a generative adversarial network[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2017: 105-114.
    [21] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]. 3rd International Conference on Learning Representations, ICLR, 2015.
    [22] 王玉玺, 陈健美. 基于自适应Canny算子和多方向Sobel算子的虹膜边缘检测算法[J]. 计算机与数字工程,2020,48(11):2744-2749. doi: 10.3969/j.issn.1672-9722.2020.11.040

    WANG Y X, CHEN J M. Iris edge detection algorithm based on adaptive canny operator and multi-directional Sobel operator[J]. Computer &Digital Engineering, 2020, 48(11): 2744-2749. (in Chinese) doi: 10.3969/j.issn.1672-9722.2020.11.040
    [23] LIU S R, TANG X Y, WANG D. Facial expression recognition based on Sobel operator and improved CNN-SVM[C]. 2020 IEEE 3rd International Conference on Information Communication and Signal Processing (ICICSP), IEEE, 2020: 236-240.
    [24] HAO F, XU D SH, CHEN D L, et al. Sobel operator enhancement based on eight-directional convolution and entropy[J]. International Journal of Information Technology, 2021, 13(5): 1823-1828. doi: 10.1007/s41870-021-00770-3
    [25] WOO S, PARK J, LEE J Y, et al. . CBAM: convolutional block attention module[C]. Proceedings of the 15th European Conference on Computer Vision (ECCV), Springer, 2018: 3-19.
    [26] SHI W ZH, CABALLERO J, HUSZÁR F, et al. . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016: 1874-1883.
    [27] LI K D, FATHAN M I, PATEL K, et al. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations[J]. PLoS One, 2021, 16(8): e0255809. doi: 10.1371/journal.pone.0255809
    [28] JHA D, SMEDSRUD P H, RIEGLER M A, et al. . Kvasir-SEG: a segmented polyp dataset[C]. 26th International Conference on Multimedia Modeling, Springer, 2020: 451-462.
  • 加载中
图(9) / 表(6)
计量
  • 文章访问数:  473
  • HTML全文浏览量:  318
  • PDF下载量:  208
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-11-29
  • 修回日期:  2022-12-23
  • 录用日期:  2023-03-15
  • 网络出版日期:  2023-04-04

目录

    /

    返回文章
    返回