Detection and segmentation method of surgical instruments based on improved YOLOv5s

MENG Xiao-liang; ZHAO Ji-kang; WANG Xiao-yu; ZHANG Li-ye; SONG Zheng

doi:10.37188/CJLCD.2023-0025

您当前的位置：

首页 >

文章列表页 >

Detection and segmentation method of surgical instruments based on improved YOLOv5s

Image Processing | 更新时间：2024-07-28

- Detection and segmentation method of surgical instruments based on improved YOLOv5s
- Chinese Journal of Liquid Crystals and Displays Vol. 38, Issue 12, Pages: 1698-1706(2023)
- 作者机构：
  
  山东理工大学计算机科学与技术学院，山东淄博 255000
- 作者简介：
- 基金信息：
  
  Supported by National Natural Science Foundation of China(62001272)
- DOI：10.37188/CJLCD.2023-0025
  CLC： TP391
- Received：31 January 2023，
  
  Revised：05 March 2023，
  
  Published：05 December 2023
- 稿件说明：
移动端阅览
MENG Xiao-liang, ZHAO Ji-kang, WANG Xiao-yu, et al. Detection and segmentation method of surgical instruments based on improved YOLOv5s[J]. Chinese journal of liquid crystals and displays, 2023, 38(12): 1698-1706.
DOI：

MENG Xiao-liang, ZHAO Ji-kang, WANG Xiao-yu, et al. Detection and segmentation method of surgical instruments based on improved YOLOv5s[J]. Chinese journal of liquid crystals and displays, 2023, 38(12): 1698-1706. DOI： 10.37188/CJLCD.2023-0025.

摘要

在内窥镜手术过程中，外科医师需实时掌握手术器械的位置信息。现有目标检测算法受反光和阴影等因素影响，其准确度和漏检率仍有优化的空间。本文提出一种基于改进YOLOv5s的手术器械检测与分割方法。首先，通过Gamma校正算法校正图像的亮度和对比度，以解决手术器械的反光和阴影遮挡等问题；其次，设计CBAM和动态卷积模块，增加重要特征信息的权重，以进一步提高目标检测的准确度并减少模型的漏检率；同时，优化空间金字塔池化模块以扩大感受野，从而更好地识别多尺度目标；最后，设计FPN语义分割头，以实现语义分割功能。在内窥镜手术数据集上的实验结果表明，本文目标检测的mAP@0.5为98.2%，语义分割的mIoU为94.0%。所提方法可辅助外科医师快速掌握手术器械的位置和类型，提高手术效率。

Abstract

In the process of endoscopic surgery， surgeons need to know the position information of surgical instruments in real time. The existing target detection algorithms are affected by factors such as reflection and shadow， and there is still optimization space for the accuracy and missed detection rate. This paper proposes a detection and segmentation method of surgical instruments based on improved YOLOv5s. Firstly， the brightness and contrast of images are corrected by Gamma correction algorithm to solve the problems of reflection and shadow occlusion of surgical instruments. Secondly， convolutional block attention module（CBAM） and dynamic convolution module are designed to increase the weight of important feature information， which further improves the accuracy of target detection and reduces the missed detection rate of the model. At the same time， the spatial pyramid pooling module is optimized to expand the receptive field， so as to better identify multi-scale targets. Finally， the feature pyramid networks （FPN） semantic segmentation head is designed to realize the semantic segmentation. Experimental results on endoscopic surgery dataset show that the mAP@0.5 of target detection in this paper is 98.2%， and the mIoU of semantic segmentation is 94.0%. The proposed method can assist surgeons to quickly grasp the position and type of surgical instruments， and improve the efficiency of surgery.

关键词

Keywords

references

RIVENSON Y ， WU Y C ， OZCAN A . Deep learning in holography and coherent imaging ［J］. Light： Science & Applications ， 2019 ， 8 ： 85 . doi: 10.1038/s41377-019-0196-0 http://dx.doi.org/10.1038/s41377-019-0196-0

ZUO C ， QIAN J M ， FENG S J ， et al . Deep learning in optical metrology： a review ［J］. Light： Science & Applications ， 2022 ， 11 （ 1 ）： 39 . doi: 10.1038/s41377-022-00714-x http://dx.doi.org/10.1038/s41377-022-00714-x

FU Z M ， JIN Z Y ， ZHANG C G ， et al . The future of endoscopic navigation： a review of advanced endoscopic vision technology ［J］. IEEE Access ， 2021 ， 9 ： 41144 - 41167 . doi: 10.1109/access.2021.3065104 http://dx.doi.org/10.1109/access.2021.3065104

JIN A ， YEUNG S ， JOPLING J ， et al . Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks ［C］// Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision . Lake Tahoe， USA ： IEEE ， 2018 ： 691 - 699 . doi: 10.1109/wacv.2018.00081 http://dx.doi.org/10.1109/wacv.2018.00081

MAHMOOD T ， CHO S W ， PARK K R . DSRD-Net： dual-stream residual dense network for semantic segmentation of instruments in robot-assisted surgery ［J］. Expert Systems with Applications ， 2022 ， 202 ： 117420 . doi: 10.1016/j.eswa.2022.117420 http://dx.doi.org/10.1016/j.eswa.2022.117420

LIU Y Y ， ZHAO Z J ， CHANG F L ， et al . An anchor-free convolutional neural network for real-time surgical tool detection in robot-assisted surgery ［J］. IEEE Access ， 2020 ， 8 ： 78193 - 78201 . doi: 10.1109/access.2020.2989807 http://dx.doi.org/10.1109/access.2020.2989807

郑腾辉，陶青川 . 基于全卷积神经网络的手术器械图像语义分割算法［J］. 现代计算机， 2019 （ 9 ）： 80 - 84 .

ZHENG T H ， TAO Q C . Semantic segmentation algorithm for images of surgical instruments based on fully convolutional networks ［J］. Modern Computer ， 2019 （ 9 ）： 80 - 84 . （in Chinese）

NI Z L ， BIAN G B ， ZHOU X H ， et al . RAUNet： residual attention U-Net for semantic segmentation of cataract surgical instruments ［C］// Proceedings of the 26th International Conference on Neural Information Processing . Sydney， Australia ： Springer ， 2019 ： 139 - 149 . doi: 10.1007/978-3-030-36711-4_13 http://dx.doi.org/10.1007/978-3-030-36711-4_13

NI Z L ， ZHOU X H ， WANG G A ， et al . SurgiNet： pyramid attention aggregation and class-wise self-distillation for surgical instrument segmentation ［J］. Medical Image Analysis ， 2022 ， 76 ： 102310 . doi: 10.1016/j.media.2021.102310 http://dx.doi.org/10.1016/j.media.2021.102310

徐昌佳，易见兵，曹锋，等 . 采用DoubleUNet网络的结直肠息肉分割算法［J］. 光学精密工程， 2022 ， 30 （ 8 ）： 970 - 983 . doi: 10.37188/ope.20223008.0970 http://dx.doi.org/10.37188/ope.20223008.0970

XU C J ， YI J B ， CAO F ， et al . Colorectal polyp segmentation algorithm using DoubleUNet network ［J］. Optics and Precision Engineering ， 2022 ， 30 （ 8 ）： 970 - 983 . （in Chinese） . doi: 10.37188/ope.20223008.0970 http://dx.doi.org/10.37188/ope.20223008.0970

JHA D ， RIEGLER M A ， JOHANSEN D ， et al . DoubleU-Net： a deep convolutional neural network for medical image segmentation ［C］// Proceedings of the 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems . Rochester， USA ： IEEE ， 2020 ： 558 - 564 . doi: 10.1109/cbms49503.2020.00111 http://dx.doi.org/10.1109/cbms49503.2020.00111

YANG M K ， YU K ， ZHANG C ， et al . DenseASPP for semantic segmentation in street scenes ［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE ， 2018 ： 3684 - 3692 . doi: 10.1109/cvpr.2018.00388 http://dx.doi.org/10.1109/cvpr.2018.00388

PAN J J ， LIU W M ， GE P ， et al . Real-time segmentation and tracking of excised corneal contour by deep neural networks for DALK surgical navigation ［J］. Computer Methods and Programs in Biomedicine ， 2020 ， 197 ： 105679 . doi: 10.1016/j.cmpb.2020.105679 http://dx.doi.org/10.1016/j.cmpb.2020.105679

樊新川，陈春梅 . 基于YOLO框架的轻量化高精度目标检测算法［J］. 液晶与显示， 2023 ， 38 （ 7 ）： 945 - 954 . doi: 10.37188/CJLCD.2022-0328 http://dx.doi.org/10.37188/CJLCD.2022-0328

FAN X C ， CHEN C M . Lightweight and high-precision object detection algorithm based on YOLO framework ［J］. Chinese Journal of Liquid Crystals and Displays ， 2023 ， 38 （ 7 ）： 945 - 954 . （in Chinese） . doi: 10.37188/CJLCD.2022-0328 http://dx.doi.org/10.37188/CJLCD.2022-0328

WOO S ， PARK J ， LEE J Y ， et al . CBAM： convolutional block attention module ［C］// Proceedings of the 15th European Conference on Computer Vision . Munich， Germany ： Springer ， 2018 ： 3 - 19 . doi: 10.1007/978-3-030-01234-2_1 http://dx.doi.org/10.1007/978-3-030-01234-2_1

OZAWA T ， HAYASHI Y ， ODA H ， et al . Synthetic laparoscopic video generation for machine learning-based surgical instrument segmentation from real laparoscopic video and virtual surgical instruments ［J］. Computer Methods in Biomechanics and Biomedical Engineering： Imaging & Visualization ， 2021 ， 9 （ 3 ）： 225 - 232 . doi: 10.1080/21681163.2020.1835560 http://dx.doi.org/10.1080/21681163.2020.1835560

ELFWING S ， UCHIBE E ， DOYA K . Sigmoid-weighted linear units for neural network function approximation in reinforcement learning ［J］. Neural Networks ， 2018 ， 107 ： 3 - 11 . doi: 10.1016/j.neunet.2017.12.012 http://dx.doi.org/10.1016/j.neunet.2017.12.012

ELHASSAN M A ， YANG C H ， HUANG C X ， et al . SPFNet： subspace pyramid fusion network for semantic segmentation ［J/OL］. arXiv ， 2022 ： 2204 . 01278 . doi: 10.1155/2021/5563875 http://dx.doi.org/10.1155/2021/5563875

LI C ， ZHOU A J ， YAO A B . Omni-dimensional dynamic convolution ［C］// Proceedings of the 10th International Conference on Learning Representations . Madrid， Spain ： OpenReview.net ， 2022 ， DOI： 10.‍48550 http://dx.doi.org/10.‍48550 .

WANG C Y ， BOCHKOVSKIY A ， LIAO H Y M . YOLOv7： trainable bag-of-freebies sets new state-of-the-art for real-time object detectors ［C］// Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Vancouver， Canada ： IEEE ， 2023 ： 7464 - 7475 . doi: 10.1109/cvpr52729.2023.00721 http://dx.doi.org/10.1109/cvpr52729.2023.00721

WANG C Y ， LIAO M H Y ， WU Y H ， et al . CSPNet： a new backbone that can enhance learning capability of CNN ［C］// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE ， 2020 ： 1571 - 1580 . doi: 10.1109/cvprw50498.2020.00203 http://dx.doi.org/10.1109/cvprw50498.2020.00203

YU L T ， WANG P C ， YU X Y ， et al . A holistically-nested U-Net： surgical instrument segmentation based on convolutional neural network ［J］. Journal of Digital Imaging ， 2020 ， 33 （ 2 ）： 341 - 347 . doi: 10.1007/s10278-019-00277-1 http://dx.doi.org/10.1007/s10278-019-00277-1

SUN Y W ， PAN B ， FU Y L . Lightweight deep neural network for real-time instrument semantic segmentation in robot assisted minimally invasive surgery ［J］. IEEE Robotics and Automation Letters ， 2021 ， 6 （ 2 ）： 3870 - 3877 . doi: 10.1109/lra.2021.3066956 http://dx.doi.org/10.1109/lra.2021.3066956

Views

241

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Semantic segmentation method for street images with multi-dimensional features

Infrared dim-small target detection under complex background based on attention mechanism

Semantic segmentation algorithm based on class feature attention mechanism fusion

Attention and feature fusion for aircraft target detection in optical remote sensing images

Related Author

ZHU Lei

CHE Chenjie

YAO Tongyu

PAN Yang

ZHANG Bo

LIU Ying

SUN Hai-jiang

ZHAO Yong-xian

Related Institution

College of Electronic Information， Xi'an Polytechnic University

Changchun Institute of Optics， Fine Mechanics and Physics， Chinese Academy of Sciences

University of Chinese Academy of Sciences

College of Big Data and Information Engineering， Guizhou University

School of Physics and Electronic and Electrical Engineering, Ningxia University

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China 130033 Postal code：130033
Tel：0431-86176059 Email：yjxs@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰