Abstract
Purpose
Automatic segmentation of surgical instruments in robot-assisted minimally invasive surgery plays a fundamental role in improving context awareness. In this work, we present an instance segmentation model based on refined Mask R-CNN for accurately segmenting the instruments as well as identifying their types.
Methods
We re-formulate the instrument segmentation task as an instance segmentation task. Then we optimize the Mask R-CNN with anchor optimization and improved Region Proposal Network for instrument segmentation. Moreover, we perform cross-dataset evaluation with different sampling strategies.
Results
We evaluate our model on a public dataset of the MICCAI 2017 Endoscopic Vision Challenge with two segmentation tasks, and both achieve new state-of-the-art performance. Besides, cross-dataset training improved the performance on both segmentation tasks compared with those tested on the public dataset.
Conclusion
Results demonstrate the effectiveness of the proposed instance segmentation network for surgical instruments segmentation. Cross-dataset evaluation shows our instance segmentation model presents certain cross-dataset generalization capability, and cross-dataset training can significantly improve the segmentation performance. Our empirical study also provides guidance on how to allocate the annotation cost for surgeons while labelling a new dataset in practice.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and material
The public dataset used during the current study is available from MICCAI 2017 EndoVis Challenge (https://endovissub2017-roboticinstrumentsegmentation.grand-challenge.org/). The in-house dataset is not applicable.
References
Allan M, Shvets A, Kurmann T, Zhang Z, Duggal R, Su YH, Rieke N, Laina I, Kalavakonda N, Bodenstedt S, Herrera L, Li W, Iglovikov V, Luo H, Yang J, Stoyanov D, Maier-Hein L, Speidel S, Azizian M (2019) 2017 robotic instrument segmentation challenge. arXiv:1902.06426
Bouget D, Benenson R, Omran M, Riffaud L, Schiele B, Jannin P (2015) Detecting surgical tools by modelling local appearance and global shape. IEEE Trans Med Imaging 34(12):2603–2617. https://doi.org/10.1109/TMI.2015.2450831
Bouget D, Allan M, Stoyanov D, Jannin P (2017) Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal 35:633–654. https://doi.org/10.1016/j.media.2016.09.003
Choi B, Jo K, Choi S, Choi J (2017) Surgical-tools detection based on convolutional neural network in laparoscopic robot-assisted surgery. In: 2017 39th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 1756–1759. https://doi.org/10.1109/EMBC.2017.8037183
Da Costa Rocha C, Padoy N, Rosa B (2019) Self-supervised surgical tool segmentation using kinematic information. In: 2019 international conference on robotics and automation (ICRA). IEEE, pp 8720–8726. https://doi.org/10.1109/ICRA.2019.8794334
Dou Q, Ouyang C, Chen C, Chen H, Heng PA (2019) Unsupervised domain adaptation of convnets for medical image segmentation via adversarial learning. In: Deep learning and convolutional neural networks for medical imaging and clinical informatics. Springer, Cham, pp 93–115. https://doi.org/10.1007/978-3-030-13969-8_5
Fuentes-Hurtado F, Kadkhodamohammadi A, Flouty E, Barbarisi S, Luengo I, Stoyanov D (2019) Easylabels: weak labels for scene segmentation in laparoscopic videos. Int J Comput Assist Radiol Surg 14(7):1247–1257. https://doi.org/10.1007/s11548-019-02003-2
Hasan SK, Linte CA (2019) U-netplus: a modified encoder-decoder u-net architecture for semantic and instance segmentation of surgical instruments from laparoscopic images. In: 2019 41st annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 7205–7211. https://doi.org/10.1109/EMBC.2019.8856791
He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969
Islam M, Atputharuban DA, Ramesh R, Ren H (2019) Real-time instrument segmentation in robotic surgery using auxiliary supervised deep adversarial learning. IEEE Robot Autom Lett 4(2):2188–2195. https://doi.org/10.1109/LRA.2019.2900854
Jin Y, Cheng K, Dou Q, Heng PA (2019) Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 440–448. https://doi.org/10.1007/978-3-030-32254-0_49
Jung AB, Wada K, Crall J, Tanaka S, Graving J, Reinders C, Yadav S, Banerjee J, Vecsei G, Kraft A, Rui Z, Borovec J, Vallentin C, Zhydenko S, Pfeiffer K, Cook B, Fernández I, De Rainville FM, Weng CH, Ayala-Acevedo A, Meudec R, Laporte M (2020) imgaug. https://github.com/aleju/imgaug. Accessed 01 Feb 2020
Kamnitsas K, Ledig C, Newcombe VF, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med Image Anal 36:61–78. https://doi.org/10.1016/j.media.2016.10.004
Pakhomov D, Premachandran V, Allan M, Azizian M, Navab N (2019) Deep residual learning for instrument segmentation in robotic surgery. In: International workshop on machine learning in medical imaging. Springer, pp 566–573. https://doi.org/10.1007/978-3-030-32692-0_65
Pezzementi Z, Voros S, Hager GD (2009) Articulated object tracking by rendering consistent appearance parts. In: 2009 IEEE international conference on robotics and automation. IEEE, pp 3940–3947. https://doi.org/10.1109/ROBOT.2009.5152374
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
Ross T, Zimmerer D, Vemuri A, Isensee F, Wiesenfarth M, Bodenstedt S, Both F, Kessler P, Wagner M, Müller B, Kenngott H, Speidel S, Kopp-Schneider A, Maier-Hein K, Len MH (2018) Exploiting the potential of unlabeled endoscopic video data with self-supervised learning. Int J Comput Assist Radiol Surg 13(6):925–933. https://doi.org/10.1007/s11548-018-1772-0
Shvets AA, Rakhlin A, Kalinin AA, Iglovikov VI (2018) Automatic instrument segmentation in robot-assisted surgery using deep learning. In: 2018 17th IEEE international conference on machine learning and applications (ICMLA). IEEE, pp 624–628. https://doi.org/10.1109/ICMLA.2018.00100
Wada K (2016) Labelme: image polygonal annotation with python. https://github.com/wkentaro/labelme. Accessed 02 Oct 2016
Wu A, Xu Z, Gao M, Buty M, Mollura DJ (2016) Deep vessel tracking: aD generalized probabilistic approach via deep learning. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, pp 1363–1367. https://doi.org/10.1109/ISBI.2016.7493520
Zlocha M, Dou Q, Glocker B (2019) Improving retinanet for ct lesion detection with dense masks from weak recist labels. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 402–410. https://doi.org/10.1007/978-3-030-32226-7_45
Funding
This work is supported by the Hong Kong Research Grants Council (No. T42-409/18-R), and a Grant from City University of Hong Kong (No. 9610443).
Author information
Authors and Affiliations
Contributions
Conceptualization: XK, YJ, QD; Methodology: XK, YJ, QD; Formal analysis and investigation: XK, YJ, QD; Data curation: ZW; Writing—original draft preparation: XK, YJ; Writing—review and editing: QD, ZW, BL, ED, DS; Funding acquisition: YL, DS; Resources: QD, ED, YL, DS; Supervision: ED, DS.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Code availability
Code will be publicly available with the publication of this work.
Ethical approval
For this type of study, formal consent is not required.
Informed consent
This study belongs to exception where it is not necessary to obtain consent.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kong, X., Jin, Y., Dou, Q. et al. Accurate instance segmentation of surgical instruments in robotic surgery: model refinement and cross-dataset evaluation. Int J CARS 16, 1607–1614 (2021). https://doi.org/10.1007/s11548-021-02438-6
Received:
Accepted:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1007/s11548-021-02438-6