留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于编码解码结构的微血管减压图像实时语义分割

白瑞峰 江山 孙海江 刘心睿

白瑞峰, 江山, 孙海江, 刘心睿. 基于编码解码结构的微血管减压图像实时语义分割[J]. 中国光学(中英文), 2022, 15(5): 1055-1065. doi: 10.37188/CO.2022-0120
引用本文: 白瑞峰, 江山, 孙海江, 刘心睿. 基于编码解码结构的微血管减压图像实时语义分割[J]. 中国光学(中英文), 2022, 15(5): 1055-1065. doi: 10.37188/CO.2022-0120
BAI Rui-feng, JIANG Shan, SUN Hai-jiang, LIU Xin-rui. Real-time semantic segmentation of microvascular decompression images based on encoder-decoder structure[J]. Chinese Optics, 2022, 15(5): 1055-1065. doi: 10.37188/CO.2022-0120
Citation: BAI Rui-feng, JIANG Shan, SUN Hai-jiang, LIU Xin-rui. Real-time semantic segmentation of microvascular decompression images based on encoder-decoder structure[J]. Chinese Optics, 2022, 15(5): 1055-1065. doi: 10.37188/CO.2022-0120

基于编码解码结构的微血管减压图像实时语义分割

doi: 10.37188/CO.2022-0120
基金项目: 吉林省科技发展计划项目(No. 20200404155YY,No. 20200401091GX);白求恩医学工程与仪器中心(长春)项目(No. Bqegczx2019047)
详细信息
    作者简介:

    白瑞峰(1994—),男,甘肃通渭人,博士研究生,2017年于兰州交通大学获得学士学位,主要从事智能医学图像处理方面的研究。E-mail: bairuifeng_ucas@126.com

    江 山(1986—),男,吉林长春人,副研究员,硕士生导师,2010年、2013年于吉林大学分别获得学士、硕士学位,主要从事深度学习、高速目标跟踪处理方面的研究。E-mail: 617798169@qq.com

    孙海江(1980—),男,吉林辉南人,研究员,博士生导师,2012年于中科院长春光机所获得博士学位,主要从事目标识别与跟踪技术及高清视频图像增强显示方面的研究。E-mail: sunhaijiang@126.com

    刘心睿(1980—),男,吉林长春人,副教授,副主任医师,硕士生导师,2006年于吉林大学获得临床医学硕士学位,2018年于吉林大学获得神经外科博士学位,主要从事显微镜及内镜下复杂颅底入路手术、术中磁共振引导神经系统肿瘤的外科治疗、脑积水脑脊液循环重建、脑神经网络与脑功能研究。E-mail: liuxinr@jlu.edu.cn

  • 中图分类号: TP394.1;TH691.9

Real-time semantic segmentation of microvascular decompression images based on encoder-decoder structure

Funds: Supported by Jilin Province Science and Technology Development Plan Project (No. 20200404155YY, No. 20200401091GX); Bethune Center for Medical Engineering and Instrumentation (Changchun) (No. BQEGCZX2019047)
More Information
  • 摘要:

    针对真彩色微血管减压图像实时语义分割网络参数量大、语义分割精度低的问题,本文提出了一种适用于微血管减压场景的U型轻量级快速语义分割网络U-MVDNet (U-Shaped Microvascular Decompression Network),该网络由编码解码结构构成。在编码器中设计了轻型非对称瓶颈模块(LABM)对上下文特征进行编码,解码器中引入了特征融合模块(FFM),有效组合高级语义特征和低级空间细节。实验结果表明:对于微血管减压测试集,U-MVDNet在单NVIDIA GTX 2080Ti上的参数量只有0.66 M,平均交并比(mIoU)达到了76.29%,速度达到140 frame/s,且当输入图像尺寸为$640 \times 480$时,U-MVDNet在嵌入式平台 NVIDIA Jetson AGX Xavier上实现了实时(24 frame/s)语义分割。本文方法未使用任何的预训练模型,参数量少且推理速度快,语义分割性能优于其他对比方法,在分割精度和速度上做到了良好的平衡。同时,还可以方便地在嵌入式平台上开发和应用,性能优越,易于部署。

     

  • 图 1  U-MVDNet架构

    Figure 1.  Architecture of U-MVDNet

    图 2  (a)ResNet 瓶颈设计及(b)轻型非对称瓶颈模块

    Figure 2.  (a) ResNet bottleneck design and (b) LABM

    图 3  特征融合模块流程图

    Figure 3.  Flow chart of feature fusion module

    图 4  损失曲线图

    Figure 4.  Loss curves

    图 5  MVD验证集上的可视化对比结果

    Figure 5.  The visual comparison results of different methods on MVD validate set

    图 6  ISIC 2016 + PH2测试集上的可视化对比

    Figure 6.  The visual comparison results of different methods on ISIC 2016 + PH2 test set

    表  1  U-MVDNet架构细节

    Table  1.   Architecture details of proposed U-MVDNet

    LayerOperatorModeChannelOutput size
    1$3 \times 3$ Convstride 232$256 \times 256$
    2$3 \times 3$ Convstride 132$256 \times 256$
    3$3 \times 3$ Convstride 132$256 \times 256$
    4-5$n \times $LABMdilated 232$256 \times 256$
    6$3 \times 3$ Convstride 264$128 \times 128$
    7-8$m \times $LABMdilated 464$128 \times 128$
    9$3 \times 3$ Convstride 2128$64 \times 64$
    10-12$l \times $LABMdilated 8128$64 \times 64$
    131×FFM128$64 \times 64$
    141×FFM64$128 \times 128$
    151×FFM32$256 \times 256$
    161×1 Convstride 110$256 \times 256$
    17Bilinear interpolation$ \times 2$10$512 \times 512$
    下载: 导出CSV

    表  2  医学术语缩写及对应颜色

    Table  2.   Abbreviations of medical terms and corresponding color

    简称全称对应颜色
    cn5三叉神经
    cn7面神经
    cn9舌咽神经
    cn10迷走神经
    aica+cn7小脑前下动脉及面神经
    pica+cn7小脑后下动脉及面神经
    pica小脑后下动脉
    aica小脑前下动脉
    pv岩静脉
    下载: 导出CSV

    表  3  训练参数

    Table  3.   Training parameters

    Parameter nameParameter selection
    Learning ratePolicyInitializationPower
    poly0.160.9
    OptimizerPolicyMomentumWeight decay
    SGD0.9$1\times10 ^{- 4}$
    Enter picture size$768 \times 576$
    Batch size8
    下载: 导出CSV

    表  4  不同扩张率组合的LABM编码器结果

    Table  4.   Results of LABM encoder with different combinations of dilation rates

    NameDilation ratesmIoU(%)
    LABM_N2M2L42,4,872.35
    LABM_N2M2L44,8,1672.08
    下载: 导出CSV

    表  5  不同设置下的LABM编码器结果

    Table  5.   Results of LABM encoder with different settings

    ConcatenationParams(M)FLOPs(G)mIoU(%)
    0.302.8172.35
    0.544.0373.08
    下载: 导出CSV

    表  6  输入尺寸为512 × 512时,不同深度的编码器结果

    Table  6.   Results of encoder with different depths when the input size is 512 × 512

    nmlParams(M)FLOPs(G)mIoU(%)
    2220.523.9572.35
    2240.544.0373.08
    2440.554.1173.84
    4440.554.2073.37
    下载: 导出CSV

    表  7  不同构成要素的FFM解码器结果

    Table  7.   Results of FFM decoder with different components

    FFMPoolingmIoU(%)
    w/o73.84
    w77.11
    w77.34
    下载: 导出CSV

    表  8  U-MVDNet的扩张率对mIoU的影响

    Table  8.   Effect of dilation of U-MVDNet on mIoU

    ConcatenationmIoU(%)Params(M)
    U-MVDNet77.340.66
    U-MVDNet_w/o dilation75.610.66
    U-MVDNet_First $3 \times 3$ conv ($r = 2$)76.810.66
    下载: 导出CSV

    表  9  MVD测试集实验结果

    Table  9.   Experimental results on MVD test set

    MethodParams(M)Speed(frame·s−1)mIoU(%)cn5cn7cn9cn10aica+cn7pica+cn7picaaicapv
    CGNet[28]0.9487.471.9581.2682.971.2969.8571.6487.1667.3765.6650.42
    EDANet[29]0.6912574.5183.0384.0270.3177.2575.0987.9870.3768.1854.34
    ContextNet[30]0.88163.375.8182.1484.1574.9178.0876.6787.8472.0869.7756.65
    U-MVDNet0.66140.876.2982.2585.4574.876.9176.3287.8574.0869.8359.12
    下载: 导出CSV

    表  10  ISIC 2016 + PH2测试集实验结果

    Table  10.   Experimental results on ISIC 2016 + PH2 test set

    ModelParams
    (M)
    Speed
    (frame·s−1)
    DIC
    (%)
    JAC
    (%)
    ACC
    (%)
    SPE
    (%)
    SEN
    (%)
    DeepLabv3[31]58.298.788.681.291.989.195.9
    CA-Net[32]2.79130.388.780.593.291.396.9
    U-MVDNet0.66175.189.381.793.293.394.3
    下载: 导出CSV

    表  11  两种不同的硬件环境

    Table  11.   Two different hardware environments

    Jetson Xavier服务器
    GPUVoltaGTX 2080Ti
    CPU8核Carmel ARM8核i7-9700K
    显存32GB LPDDR4x11GB GDDR6
    显存带宽136.5 GB/s616 GB/s
    CUDA核心5124352
    下载: 导出CSV

    表  12  不同分辨率下的测试结果

    Table  12.   Test results by different methods with different resolutions

    MethodSizeTimes/msSpeed/frame·s−1mIoU/%
    CGNet[28]$640 \times 480$65.715.270.31
    $768 \times 576$69.214.471.95
    EDANet[29]$640 \times 480$42.323.673.2
    $768 \times 576$45.222.174.18
    ContextNet[30]$640 \times 480$34.528.974.81
    $768 \times 576$36.127.775.81
    U-MVDNet$640 \times 480$41.524.275.76
    $768 \times 576$43.622.976.29
    下载: 导出CSV
  • [1] BENNETTO L, PATEL N K, FULLER G. Trigeminal neuralgia and its management[J]. BMJ, 2007, 334(7586): 201-205. doi: 10.1136/bmj.39085.614792.BE
    [2] KIZILTAN M E, GUNDUZ A. Reorganization of sensory input at brainstem in hemifacial spasm and postparalytic facial syndrome[J]. Neurological Sciences, 2018, 39(2): 313-319. doi: 10.1007/s10072-017-3185-1
    [3] NAZIR A, CHEEMA M N, SHENG B, et al. OFF-eNET: an optimally fused fully end-to-end network for automatic dense volumetric 3D intracranial blood vessels segmentation[J]. IEEE Transactions on Image Processing, 2020, 29: 7192-7202. doi: 10.1109/TIP.2020.2999854
    [4] 吴红宇, 郑波. 脑动静脉畸形CTA、DSA的影像学表现及诊断的对照性研究[J]. 中国CT和MRI杂志,2021,19(1):36-37,52. doi: 10.3969/j.issn.1672-5131.2021.01.012

    WU H Y, ZHENG B. Analysis on imaging manifestations and diagnostic contrast of cerebral arteriovenous malformations in CTA and DSA[J]. Chinese Journal of CT and MRI, 2021, 19(1): 36-37,52. (in Chinese) doi: 10.3969/j.issn.1672-5131.2021.01.012
    [5] PATEL T R, PALIWAL N, JAISWAL P, et al. Multi-resolution CNN for brain vessel segmentation from cerebrovascular images of intracranial aneurysm: a comparison of U-Net and DeepMedic[J]. Proceedings of SPIE, 2020, 11314: 113142W.
    [6] 王华. 磁共振血管成像与三维动脉自旋标记脑灌注成像技术诊断缺血性脑血管疾病一致性比较[J]. 实用医院临床杂志,2020,17(1):36-39. doi: 10.3969/j.issn.1672-6170.2020.01.011

    WANG H. Comparison of consistency of MRA and 3D-ASL cerebral perfusion imaging in the diagnosisof ischemic cerebrovascular diseases[J]. Practical Journal of Clinical Medicine, 2020, 17(1): 36-39. (in Chinese) doi: 10.3969/j.issn.1672-6170.2020.01.011
    [7] 徐冰洁. 电子计算机断层扫描联合核磁共振血管成像对脑血管疾病的诊断价值[J]. 临床合理用药杂志,2021,14(32):177-178. doi: 10.15887/j.cnki.13-1389/r.2021.32.071

    XU B J. Diagnostic value of computed tomography combined with nuclear magnetic resonance angiography in cerebrovascular diseases[J]. Chinese Journal of Clinical Rational Drug Use, 2021, 14(32): 177-178. (in Chinese) doi: 10.15887/j.cnki.13-1389/r.2021.32.071
    [8] ZHANG H, XIA L K, SONG R, et al. . Cerebrovascular segmentation in mra via reverse edge attention network[C]. Proceedings of the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2020: 66-75.
    [9] GUO X Y, XIAO R X, LU Y Y, et al. Cerebrovascular segmentation from TOF-MRA based on multiple-U-net with focal loss function[J]. Computer Methods and Programs in Biomedicine, 2021, 202: 105998. doi: 10.1016/j.cmpb.2021.105998
    [10] HILBERT A, MADAI V I, AKAY E M, et al. BRAVE-NET: fully automated arterial brain vessel segmentation in patients with cerebrovascular disease[J]. Frontiers in Artificial Intelligence, 2020, 3: 552258. doi: 10.3389/frai.2020.552258
    [11] 陈星, 宋智洋, 周明全, 等. 面向脑血管分割的改进型非局部均值滤波算法研究[J]. 中国光学,2014,7(4):572-580.

    CHEN X, SONG ZH Y, ZHOU M Q, et al. An improved non-local mean filter algorithm facing the cerebrovascular segmentation[J]. Chinese Optics, 2014, 7(4): 572-580. (in Chinese)
    [12] WANG R, LI CH, WANG J, et al. Threshold segmentation algorithm for automatic extraction of cerebral vessels from brain magnetic resonance angiography images[J]. Journal of Neuroscience Methods, 2015, 241: 30-36. doi: 10.1016/j.jneumeth.2014.12.003
    [13] BHUIYAN A, NATH B, CHUA J. An adaptive region growing segmentation for blood vessel detection from retinal images[C]. Visapp: Second International Conference on Computer Vision Theory and Applications, 2007: 404-409.
    [14] 王醒策, 张美霞, 武仲科, 等. 基于全局LBF水平集模型的脑血管层次粗分割[J]. 光学 精密工程,2013,21(12):3283-3297. doi: 10.3788/OPE.20132112.3283

    WANG X C, ZHANG M X, WU ZH K, et al. Level coarse brain vessel segmentation based on global LBF model[J]. Optics and Precision Engineering, 2013, 21(12): 3283-3297. (in Chinese) doi: 10.3788/OPE.20132112.3283
    [15] WANG J X, ZHAO SH F, LIU Z F, et al. An active contour model based on adaptive threshold for extraction of cerebral vascular structures[J]. Computational and Mathematical Methods in Medicine, 2016, 2016: 6472397.
    [16] 陈晓冬, 艾大航, 张佳琛, 等. Gabor滤波融合卷积神经网络的路面裂缝检测方法[J]. 中国光学,2020,13(6):1293-1301. doi: 10.37188/CO.2020-0041

    CHEN X D, AI D H, ZHANG J CH, et al. Gabor filter fusion network for pavement crack detection[J]. Chinese Optics, 2020, 13(6): 1293-1301. (in Chinese) doi: 10.37188/CO.2020-0041
    [17] 王春哲, 安军社, 姜秀杰, 等. 基于卷积神经网络的候选区域优化算法[J]. 中国光学,2019,12(6):1348-1361. doi: 10.3788/co.20191206.1348

    WANG CH ZH, AN J SH, JIANG X J, et al. Region proposal optimization algorithm based on convolutional neural networks[J]. Chinese Optics, 2019, 12(6): 1348-1361. (in Chinese) doi: 10.3788/co.20191206.1348
    [18] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]. Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2015: 3431-3440.
    [19] SHVETS A A, IGLOVIKOV V I, RAKHLIN A, et al. . Angiodysplasia detection and localization using deep convolutional neural networks[C]. Proceedings of 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, 2018: 612-617.
    [20] ZHANG M, ZHANG CH, WU X, et al. A neural network approach to segment brain blood vessels in digital subtraction angiography[J]. Computer Methods Programs in Biomedicine, 2020, 185: 105159. doi: 10.1016/j.cmpb.2019.105159
    [21] XIA L K, XIE Y X, WANG Q W, et al. A nested parallel multiscale convolution for cerebrovascular segmentation[J]. Medical Physics, 2021, 48(12): 7971-7983. doi: 10.1002/mp.15280
    [22] MENG C, SUN K, GUAN SH Y, et al. Multiscale dense convolutional neural network for DSA cerebrovascular segmentation[J]. Neurocomputing, 2020, 373: 123-134. doi: 10.1016/j.neucom.2019.10.035
    [23] HUANG G, LIU ZH, VAN DER MAATEN L, et al. . Densely connected convolutional networks[C]. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2017: 2261-2269.
    [24] ALVAREZ J, PETERSSON L. DecomposeMe: simplifying convnets for end-to-end learning[J]. arXiv preprint arXiv:, 1606, 05426: 2016.
    [25] HE K M, ZHANG X Y, REN SH Q, et al. . Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[C]. Proceedings of 2015 IEEE International Conference on Computer Vision, IEEE, 2015: 1026-1034.
    [26] CODELLA N C F, GUTMAN D, CELEBI M E, et al. . Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC)[C]. Proceedings of the 15th International Symposium on Biomedical Imaging, IEEE, 2016.
    [27] MENDONÇA T, FERREIRA P M, MARQUES J S, et al. . PH2-A dermoscopic image database for research and benchmarking[C]. Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2013: 5437-5440.
    [28] WU T Y, TANG SH, ZHANG R, et al. CGNet: a light-weight context guided network for semantic segmentation[J]. IEEE Transactions on Image Processing, 2020, 30: 1169-1179.
    [29] LO S Y, HANG H M, CHAN SH W, et al. . Efficient dense modules of asymmetric convolution for real-time semantic segmentation[C]. Proceedings of the ACM Multimedia Asia, ACM, 2019: 1.
    [30] POUDEL R P K, BONDE U, LIWICKI S, et al. ContextNet: exploring context and detail for semantic segmentation in real-time[J]. arXiv preprint arXiv:, 0455, 4: 2018.
    [31] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[J]. arXiv preprint arXiv:, 1706, 05587: 2017.
    [32] GU R, WANG G T, SONG T, et al. CA-Net: comprehensive attention convolutional neural networks for explainable medical image segmentation[J]. IEEE Transactions on Medical Imaging, 2021, 40(2): 699-711. doi: 10.1109/TMI.2020.3035253
  • 加载中
图(6) / 表(12)
计量
  • 文章访问数:  457
  • HTML全文浏览量:  261
  • PDF下载量:  134
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-06-10
  • 修回日期:  2022-07-05
  • 网络出版日期:  2022-08-23

目录

    /

    返回文章
    返回

    重要通知

    2024年2月16日科睿唯安通过Blog宣布,2024年将要发布的JCR2023中,229个自然科学和社会科学学科将SCI/SSCI和ESCI期刊一起进行排名!《中国光学(中英文)》作为ESCI期刊将与全球SCI期刊共同排名!