留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

多尺度注意力融合的图像超分辨率重建

陈纯毅 吴欣怡 胡小娟 于海洋

陈纯毅, 吴欣怡, 胡小娟, 于海洋. 多尺度注意力融合的图像超分辨率重建[J]. 中国光学(中英文), 2023, 16(5): 1034-1044. doi: 10.37188/CO.2023-0020
引用本文: 陈纯毅, 吴欣怡, 胡小娟, 于海洋. 多尺度注意力融合的图像超分辨率重建[J]. 中国光学(中英文), 2023, 16(5): 1034-1044. doi: 10.37188/CO.2023-0020
CHEN Chun-yi, WU Xin-yi, HU Xiao-juan, YU Hai-yang. Image super-resolution reconstruction with multi-scale attention fusion[J]. Chinese Optics, 2023, 16(5): 1034-1044. doi: 10.37188/CO.2023-0020
Citation: CHEN Chun-yi, WU Xin-yi, HU Xiao-juan, YU Hai-yang. Image super-resolution reconstruction with multi-scale attention fusion[J]. Chinese Optics, 2023, 16(5): 1034-1044. doi: 10.37188/CO.2023-0020

多尺度注意力融合的图像超分辨率重建

基金项目: 国家自然科学基金项目(No. U19A2063);吉林省科技发展计划项目(No. 20230201080GX)
详细信息
    作者简介:

    陈纯毅(1981—),男,重庆人,博士,教授,博士生导师,2009年于长春理工大学获得博士学位,主要从事计算光学成像、计算机仿真等方面的研究。E-mail:chenchunyi@hotmail.com

  • 中图分类号: TP391

Image super-resolution reconstruction with multi-scale attention fusion

Funds: Supported by the National Natural Science Foundation of China (No. U19A2063); Science and Technology Development Project of Jilin Province (No. 20230201080GX)
More Information
  • 摘要:

    光学成像分辨率受衍射极限、探测器尺寸等诸多因素限制。为了获得细节更丰富、纹理更清晰的超分辨率图像,本文提出了一种多尺度特征注意力融合残差网络。首先,使用一层卷积提取图像的浅层特征,之后,通过级联的多尺度特征提取单元提取多尺度特征,多尺度特征提取单元中引入通道注意力模块自适应地校正特征通道的权重,以提高对高频信息的关注度。将网络中的浅层特征和每个多尺度特征提取单元的输出作为全局特征融合重建的层次特征。最后,利用残差分支引入浅层特征和多级图像特征,重建出高分辨率图像。算法使用Charbonnier损失函数使训练更加稳定,收敛速度更快。在国际基准数据集上的对比实验表明:该模型的客观指标优于大多数最先进的方法。尤其在Set5数据集上,4倍重建结果的PSNR指标提升了0.39 dB,SSIM指标提升至0.8992,且算法主观视觉效果更好。

     

  • 图 1  多尺度注意力残差网络

    Figure 1.  Multi-scale attention residual network

    图 2  多尺度特征提取单元

    Figure 2.  Multi-scale feature extraction unit

    图 3  特征融合重建层

    Figure 3.  Feature fusion reconstruction layer

    图 4  用于比较的模块

    Figure 4.  Modules for comparison

    图 5  Set14数据集中“zebra”3×的视觉效果图

    Figure 5.  Comparison of the results of "zebra" 3× in the Set14 dataset

    图 6  B100数据集中“148026”放大倍数为4×的结果对比

    Figure 6.  Comparison of the results of "148026" 4× in the B100 dataset

    图 7  Urban100数据集中“img012”放大倍数4×的结果对比

    Figure 7.  Comparison of the results of "img012" 4× in the Urban100 dataset

    图 8  不同模型在Set5(×4)上的PSNR以及参数量

    Figure 8.  PSNR and parameters of different models on the Set5(×4) dataset

    表  1  多尺度特征提取单元参数

    Table  1.   Parameters of the multi-scale feature extraction units

    所属
    模块
    组件名卷积核
    大小
    输入尺寸输出尺寸
    第一级Conv11×1H×W×64H×W×32
    Conv33×3H×W×32H×W×32
    第二级Conv33×3H×W×64H×W×64
    通道注意力Fusion1×1H×W×192H×W×64
    PoolingH×W×641×1×64
    Conv1-11×11×1×641×1×4
    Conv1-21×11×1×41×1×64
    下载: 导出CSV

    表  2  不同模块的有效性验证

    Table  2.   Validation of different modules

    模型名字CARBFFRLPSNR/SSIM/TIME
    MSARNSC××27.62/0.7682/ 0.11s
    MSARNDB××27.67/0.7751/0.16s
    MSARNIB××27.78/0.7767/0.13s
    MSARNFFRL-×28.26/0.7789/0.15s
    MSARN28.64 /0.7840/0.14s
    下载: 导出CSV

    表  3  残差分支与通道注意力有效性验证

    Table  3.   Validation of residual branch and channel attention

    模块名字CARBFFRLPSNR/SSIM
    MSARNRB-×28.57/0.7802
    MSARNCA-×28.35/0.7778
    MSARN28.64 /0.7840
    下载: 导出CSV

    表  4  不同损失函数的PSNR比较

    Table  4.   PSNR comparison of different loss functions

    放大比例损失函数Set5Set14
    ×2L237.8433.50
    Charbonnier38.1333.89
    ×3L233.9130.03
    Charbonnier34.0530.40
    ×4L231.5328.26
    Charbonnier31.6728.41
    下载: 导出CSV

    表  5  不同超分辨率模型重建PSNR/SSIM比较

    Table  5.   PSNR/SSIM comparison of different super-resolution models

    放大比例方法Set5Set14BSD100Urban100
    ×2Bicubic33.68/0.926530.24/0.869129.56/0.843526.88/0.8405
    SRCNN36.66/0.954232.45/0.906731.56/0.887929.51/0.8946
    VDSR37.52/0.958733.05/0.912731.90/0.896030.77/0.9141
    DRRN37.74/0.959733.23/0.913632.05/0.897331.23/0.9188
    IDN37.83/0.960033.30/0.914832.08/0.898531.27/0.9196
    MSRN38.08/0.960533.74/0.917032.23/0.901332.22/0.9326
    PAN38.00/0.960533.59/0.918132.18/0.899732.01/0.9273
    EFDN38.00/0.960433.57/0.917932.18/0.899832.05/0.9275
    本文38.43/0.962634.05/0.921332.32/0.902832.28/0.9338
    ×3Bicubic30.40/0.868627.54/0.774127.21/0.738924.46/0.7349
    SRCNN32.75/0.909029.29/0.821528.41/0.786326.24/0.7991
    VDSR33.66/0.921329.78/0.831828.83/0.797627.14/0.8279
    DRRN34.03/0.924429.96/0.834928.95/0.800427.53/0.8377
    IDN34.11/0.925329.99/0.835428.95/0.801327.42/0.8359
    MSRN34.38/0.926230.34/0.839529.08/0.804128.08/0.8554
    PAN34.40/0.927130.36/0.842329.11/0.805028.11/0.8511
    本文34.61/0.928430.33/0.848029.25/0.807628.39/0.8607
    ×4Bicubic28.43/0.810926.00/0.702325.96/0.667823.14/0.6574
    SRCNN30.48/0.862827.50/0.751326.90/0.710324.52/0.7226
    VDSR31.35/0.883828.02/0.767827.29/0.725225.18/0.7525
    DRRN31.68/0.888828.21/0.772027.38/0.728425.44/0.7638
    IDN31.82/0.890328.25/0.773027.41/0.729725.41/0.7632
    MSRN32.07/0.890328.60/0.775127.52/0.727326.04/0.7896
    PAN32.13/0.894828.61/0.782227.59/0.736326.11/0.7854
    EFDN32.08/0.893128.58/0.780927.56/0.735426.00/0.7815
    本文32.52/0.899228.85/0.784027.70/0.741026.21/0.7866
    下载: 导出CSV
  • [1] 左超, 陈钱. 分辨率、超分辨率与空间带宽积拓展—从计算光学成像角度的一些思考[J]. 中国光学(中英文),2022,15(6):1105-1166.

    ZUO CH, CHEN Q. Resolution, super-resolution and spatial bandwidth product expansion——some thoughts from the perspective of computational optical imaging[J]. Chinese Optics, 2022, 15(6): 1105-1166. (in Chinese)
    [2] 吴靖, 叶晓晶, 黄峰, 等. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报,2022,50(9):2265-2294.

    WU J, YE X J, HUANG F, et al. A review of single image super-resolution reconstruction based on deep learning[J]. Acta Electronica Sinica, 2022, 50(9): 2265-2294. (in Chinese)
    [3] 李洪安, 郑峭雪, 陶若霖, 等. 基于深度学习的图像超分辨率研究综述[J]. 图学学报,2023,44(1):1-15.

    LI H A, ZHENG Q X, TAO R L, et al. Review of image super-resolution based on deep learning[J]. Journal of Graphics, 2023, 44(1): 1-15. (in Chinese)
    [4] 毕勇, 潘鸣奇, 张硕, 等. 三维点云数据超分辨率技术[J]. 中国光学(中英文),2022,15(2):210-223.

    BI Y, PAN M Q, ZHANG SH, et al. Overview of 3D point cloud super-resolution technology[J]. Chinese Optics, 2022, 15(2): 210-223. (in Chinese)
    [5] 王溢琴, 董云云, 刘慧玲. 基于GoogLeNet和空间谱变换的高光谱图像超分辨率方法[J]. 光学技术,2022,48(1):93-101. doi: 10.3321/j.issn.1002-1582.2022.1.gxjs202201015

    WANG Y Q, DONG Y Y, LIU H L. Super-resolution method of hyperspectral image based on GoogLeNet and spatial spectrum transformation[J]. Optical Technique, 2022, 48(1): 93-101. (in Chinese) doi: 10.3321/j.issn.1002-1582.2022.1.gxjs202201015
    [6] 曲海成, 王雅萱, 申磊. 多感受野特征与空谱注意力结合的高光谱图像超分辨率算法[J]. 自然资源遥感,2022,34(1):43-52.

    QU H CH, WANG Y X, SHEN L. Hyperspectral super-resolution combining multi-receptive field features with spectral-spatial attention[J]. Remote Sensing for Natural Resources, 2022, 34(1): 43-52. (in Chinese)
    [7] 柯舒婷, 陈明惠, 郑泽希, 等. 生成对抗网络对OCT视网膜图像的超分辨率重建[J]. 中国激光,2022,49(15):1507203.

    KE SH T, CHEN M H, ZHENG Z X, et al. Super-resolution reconstruction of optical coherence tomography retinal images by generating adversarial network[J]. Chinese Journal of Lasers, 2022, 49(15): 1507203. (in Chinese)
    [8] 左艳, 黄钢, 聂生东. 深度学习在医学影像智能处理中的应用与挑战[J]. 中国图象图形学报,2021,26(2):305-315. doi: 10.11834/jig.190470

    ZUO Y, HUANG G, NIE SH D. Application and challenges of deep learning in the intelligent processing of medical images[J]. Journal of Image and Graphics, 2021, 26(2): 305-315. (in Chinese) doi: 10.11834/jig.190470
    [9] 王一宁, 赵青杉, 秦品乐, 等. 基于轻量密集神经网络的医学图像超分辨率重建算法[J]. 计算机应用,2022,42(8):2586-2592.

    WANG Y N, ZHAO Q SH, QIN P L, et al. Super-resolution reconstruction algorithm of medical image based on lightweight dense neural network[J]. Journal of Computer Applications, 2022, 42(8): 2586-2592. (in Chinese)
    [10] 耿铭昆, 吴凡路, 王栋. 轻量化火星遥感影像超分辨率重建网络[J]. 光学 精密工程,2022,30(12):1487-1498. doi: 10.37188/OPE.20223012.1487

    GENG M K, WU F L, WANG, D. Lightweight Mars remote sensing image super-resolution reconstruction network[J]. Optics and Precision Engineering, 2022, 30(12): 1487-1498. (in Chinese) doi: 10.37188/OPE.20223012.1487
    [11] ZHANG J ZH, XU T F, LI J N, et al. Single-image super resolution of remote sensing images with real-world degradation modeling[J]. Remote Sensing, 2022, 14(12): 2895. doi: 10.3390/rs14122895
    [12] 倪若婷, 周莲英. 基于卷积神经网络的人脸图像超分辨率重建方法[J]. 计算机与数字工程,2022,50(1):195-200. doi: 10.3969/j.issn.1672-9722.2022.01.037

    NI R T, ZHOU L Y. Face image super-resolution reconstruction method based on convolutional neural network[J]. Computer &Digital Engineering, 2022, 50(1): 195-200. (in Chinese) doi: 10.3969/j.issn.1672-9722.2022.01.037
    [13] 卢峰, 周琳, 蔡小辉. 面向安防监控场景的低分辨率人脸识别算法研究[J]. 计算机应用研究,2021,38(4):1230-1234. doi: 10.19734/j.issn.1001-3695.2020.01.0074

    LU F, ZHOU L, CAI X H. Research on low-resolution face recognition algorithm for security surveillance scene[J]. Application Research of Computers, 2021, 38(4): 1230-1234. (in Chinese) doi: 10.19734/j.issn.1001-3695.2020.01.0074
    [14] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics,Speech,and Signal Processing, 1981, 29(6): 1153-1160. doi: 10.1109/TASSP.1981.1163711
    [15] 黄友文, 唐欣, 周斌. 结合双注意力和结构相似度量的图像超分辨率重建网络[J]. 液晶与显示,2022,37(3):367-375. doi: 10.37188/CJLCD.2021-0178

    HUANG Y W, TANG X, ZHOU B. Image super-resolution reconstruction network with dual attention and structural similarity measure[J]. Chinese Journal of Liquid Crystals and Displays, 2022, 37(3): 367-375. (in Chinese) doi: 10.37188/CJLCD.2021-0178
    [16] 周乐, 徐龙, 刘孝艳, 等. 基于梯度感知的单幅图像超分辨[J]. 液晶与显示,2022,37(10):1334-1344. doi: 10.37188/CJLCD.2022-0083

    ZHOU L, XU L, LIU X Y, et al. Gradient-aware based single image super-resolution[J]. Chinese Journal of Liquid Crystals and Displays, 2022, 37(10): 1334-1344. (in Chinese) doi: 10.37188/CJLCD.2022-0083
    [17] DONG CH, LOY C C, HE K M, et al.. Learning a deep convolutional network for image super-resolution[C]. Proceedings of the 13th European Conference on Computer Vision, Springer, 2014: 184-199.
    [18] DONG CH, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]. Proceedings of the 14th European Conference on Computer Vision, Springer, 2016: 391-407.
    [19] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2016: 1646-1654.
    [20] TAI Y, YANG J, LIU X M. Image super-resolution via deep recursive residual network[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2017: 2790-2798.
    [21] SHI W ZH, CABALLERO J, HUSZÁR F, et al.. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2016: 1874-1883.
    [22] ZHANG Y L, TIAN Y P, KONG Y, et al. . Residual dense network for image super-resolution[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2018: 2472-2481.
    [23] LIM B, SON S, KIM H, et al. . Enhanced deep residual networks for single image super-resolution[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, 2017: 1132-1140.
    [24] ZHANG Y L, LI K P, LI K, et al. . Image super-resolution using very deep residual channel attention networks[C]. Proceedings of the 15th European Conference on Computer Vision, Springer, 2018: 294-310.
    [25] LI J CH, FANG F M, MEI K F, et al. . Multi-scale residual network for image super-resolution[C]. Proceedings of the 15th European Conference on Computer Vision, Springer, 2018: 527-542.
    [26] ZHAO H Y, KONG X T, HE J W, et al. . Efficient image super-resolution using pixel attention[C]. Proceedings of the European Conference on Computer Vision, Springer, 2020: 56-72.
    [27] WANG Y. Edge-enhanced feature distillation network for efficient super-resolution[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, 2022: 776-784.
    [28] LAI W SH, HUANG J B, AHUJA N, et al.. Deep Laplacian pyramid networks for Fast and accurate super-resolution[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2017: 5835-5843.
    [29] SZEGEDY C, LIU W, JIA Y Q, et al.. Going deeper with convolutions[C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2015: 1-9.
    [30] HUI ZH, WANG X M, GAO X B. Fast and accurate single image super-resolution via information distillation network[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2018: 723-731.
  • 加载中
图(8) / 表(5)
计量
  • 文章访问数:  874
  • HTML全文浏览量:  362
  • PDF下载量:  279
  • 被引次数: 0
出版历程
  • 收稿日期:  2023-01-28
  • 修回日期:  2023-02-20
  • 录用日期:  2023-04-04
  • 网络出版日期:  2023-04-13

目录

    /

    返回文章
    返回