Ann Hepatobiliary Pancreat Surg.  2024 Nov;28(4):466-473. 10.14701/ahbps.24-091.

Deep learning-based surgical phase recognition in laparoscopic cholecystectomy

• Yang HY
• Hong SS
• Yoon J
• Park B
• Yoon Y
• Han DH
• Choi GH
• Choi MK
• Kim SH

• KMID: 2561579
• DOI: http://doi.org/10.14701/ahbps.24-091

Abstract

Backgrounds/Aims
Artificial intelligence (AI) technology has been used to assess surgery quality, educate, and evaluate surgical performance using video recordings in the minimally invasive surgery era. Much attention has been paid to automating surgical workflow analysis from surgical videos for an effective evaluation to achieve the assessment and evaluation. This study aimed to design a deep learning model to automatically identify surgical phases using laparoscopic cholecystectomy videos and automatically assess the accuracy of recognizing surgical phases.
Methods
One hundred and twenty cholecystectomy videos from a public dataset (Cholec80) and 40 laparoscopic cholecystectomy videos recorded between July 2022 and December 2022 at a single institution were collected. These datasets were split into training and testing datasets for the AI model at a 2:1 ratio. Test scenarios were constructed according to structural characteristics of the trained model. No pre- or post-processing of input data or inference output was performed to accurately analyze the effect of the label on model training.
Results
A total of 98,234 frames were extracted from 40 cases as test data. The overall accuracy of the model was 91.2%. The most accurate phase was Calot’s triangle dissection (F1 score: 0.9421), whereas the least accurate phase was clipping and cutting (F1 score: 0.7761).
Conclusions
Our AI model identified phases of laparoscopic cholecystectomy with a high accuracy.

Keyword

Artificial intelligence; Surgical procedures, operative; Computer terminals; Laparoscopic cholecystectomy; Pattern recognition, automated

Figure

• 

  Fig. 1 Training paradigm of the 3D-ResNet architecture for surgical phase recognition in laparoscopic cholecystectomy. Weight updates are performed for each iteration using mini-batches of frame chunks extracted at 16 frames per chunk.

• 

  Fig. 2 Distribution of phase duration of enrolled cases.

• 

  Fig. 3 Normalized confusion matrix.

Reference

References

1. Kovid N, Han HS, Yoon YS, Cho JY. 2020; Advanced laparoscopic HPB surgery: experience in Seoul National University Bundang Hospital. Ann Gastroenterol Surg. 4:224–228. DOI: 10.1002/ags3.12323. PMID: 32490336. PMCID: PMC7240149.

2. Hegde SR, Namazi B, Iyengar N, Cao S, Desir A, Marques C, et al. 2024; Automated segmentation of phases, steps, and tasks in laparoscopic cholecystectomy using deep learning. Surg Endosc. 38:158–170. DOI: 10.1007/s00464-023-10482-3. PMID: 37945709.

3. Tominaga T, Nonaka T, Hashimoto Y, Hamasaki K, Arai J, Matsumoto K, et al. 2024; A surgical training system designed to help trainees acquire laparoscopic surgical skills at a rural Japanese institute. Surg Today. 54:145–151. DOI: 10.1007/s00595-023-02713-w. PMID: 37300751.

4. Chadebecq F, Lovat LB, Stoyanov D. 2023; Artificial intelligence and automation in endoscopy and surgery. Nat Rev Gastroenterol Hepatol. 20:171–182. DOI: 10.1038/s41575-022-00701-y. PMID: 36352158.

5. Rangwani S, Ardeshna DR, Rodgers B, Melnychuk J, Turner R, Culp S, et al. 2022; Application of artificial intelligence in the management of pancreatic cystic lesions. Biomimetics (Basel). 7:79. DOI: 10.3390/biomimetics7020079. PMID: 35735595. PMCID: PMC9221027.

6. Wu Y, Li Y, Xiong X, Liu X, Lin B, Xu B. 2023; Recent advances of pathomics in colorectal cancer diagnosis and prognosis. Front Oncol. 13:1094869. DOI: 10.3389/fonc.2023.1094869. PMID: 37538112. PMCID: PMC10396402.

7. Hashimoto DA, Rosman G, Rus D, Meireles OR. 2018; Artificial intelligence in surgery: promises and perils. Ann Surg. 268:70–76. DOI: 10.1097/SLA.0000000000002693. PMID: 29389679. PMCID: PMC5995666.

8. Spinelli A, Carrano FM, Laino ME, Andreozzi M, Koleth G, Hassan C, et al. 2023; Artificial intelligence in colorectal surgery: an AI-powered systematic review. Tech Coloproctol. 27:615–629. DOI: 10.1007/s10151-023-02772-8. PMID: 36805890.

9. Cheng K, You J, Wu S, Chen Z, Zhou Z, Guan J, et al. 2022; Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis. Surg Endosc. 36:3160–3168. DOI: 10.1007/s00464-021-08619-3. PMID: 34231066.

10. Golany T, Aides A, Freedman D, Rabani N, Liu Y, Rivlin E, et al. 2022; Artificial intelligence for phase recognition in complex laparoscopic cholecystectomy. Surg Endosc. 36:9215–9223. DOI: 10.1007/s00464-022-09405-5. PMID: 35941306. PMCID: PMC9652206.

11. Kitano S, Sugimachi K. 1993; Peritoneoscopic cholecystectomy has opened the door to minimally invasive surgery. J Gastroenterol Hepatol. 8:476–482. DOI: 10.1111/j.1440-1746.1993.tb01552.x. PMID: 8218998.

12. Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N. 2017; EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging. 36:86–97. DOI: 10.1109/TMI.2016.2593957. PMID: 27455522.

13. Hong S, Hong S, Jang J, Kim K, Hyung WJ, Choi MK. 2023; Amplifying action-context greater: image segmentation-guided intraoperative active bleeding detection. Comput Methods Biomech Biomed Eng Imaging Vis. 11:1261–1270. DOI: 10.1080/21681163.2022.2159533.

14. Hara K, Kataoka H, Satoh Y. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet? Paper presented at: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018 Jun 18-23; Salt Lake City, USA. p. 6546–6555. DOI: 10.1109/CVPR.2018.00685.

15. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al. PyTorch: an imperative style, high-performance deep learning library. Paper presented at: Advances in Neural Information Processing Systems 32 (NeurIPS 2019). 2019 Dec 8-14; Vancouver, Canada.

16. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Paper presented at: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016 Jun 27-30; Las Vegas, USA. p. 770–778. DOI: 10.1109/CVPR.2016.90.

17. Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S, et al. The kinetics human action video dataset. arXiv preprint. 2017.

18. Zolfaghari M, Singh K, Brox T. ECO: efficient convolutional network for online video understanding. Paper presented at: European Conference on Computer Vision (ECCV). 2018 Sep 8-14; Munich, Germany. p. 695–712. DOI: 10.1007/978-3-030-01216-8_43.

19. Bar O, Neimark D, Zohar M, Hager GD, Girshick R, Fried GM, et al. 2020; Impact of data on generalization of AI for surgical intelligence applications. Sci Rep. 10:22208. DOI: 10.1038/s41598-020-79173-6. PMID: 33335191. PMCID: PMC7747564.

20. Czempiel T, Paschali M, Keicher M, Simson W, Feussner H, Kim ST, et al. Martel AL, Abolmaesumi P, Stoyanov D, Mateus D, Zuluaga MA, Zhou SK, editors. 2020. TeCNO: surgical phase recognition with multi-stage temporal convolutional networks. Medical image computing and computer assisted intervention - MICCAI 2020. p. 343–352. Springer;DOI: 10.1007/978-3-030-59716-0_33.

21. Czempiel T, Paschali M, Ostler D, Kim ST, Busam B, Navab N. de Bruijne M, Cattin PC, Cotin S, Padoy N, Speidel S, Zheng Y, editors. 2021. OperA: attention-regularized transformers for surgical phase recognition. Springer;p. 604–614. Medical image computing and computer assisted intervention - MICCAI 2021. DOI: 10.1007/978-3-030-87202-1_58.

22. Yi F, Yang Y, Jiang T. Wang L, Gall J, Chin TJ, Sato I, Chellappa R, editors. 2023. Not end-to-end: explore multi-stage architecture for online surgical phase recognition. Computer vision - ACCV 2022. Springer;p. 417–432. DOI: 10.1007/978-3-031-26316-3_25.

23. Pucher PH, Brunt LM, Davies N, Linsk A, Munshi A, Rodriguez HA, et al. SAGES Safe Cholecystectomy Task Force. 2018; Outcome trends and safety measures after 30 years of laparoscopic cholecystectomy: a systematic review and pooled data analysis. Surg Endosc. 32:2175–2183. DOI: 10.1007/s00464-017-5974-2. PMID: 29556977. PMCID: PMC5897463.

24. Evans L, Sams E, Naguib A, Hajibandeh S, Hajibandeh S. 2022; Iatrogenic gallbladder perforation during laparoscopic cholecystectomy and outcomes: a systematic review and meta-analysis. Langenbecks Arch Surg. 407:937–946. DOI: 10.1007/s00423-022-02439-2. PMID: 35039923.

25. Tzovaras G, Dervenis C. 2006; Vascular injuries in laparoscopic cholecystectomy: an underestimated problem. Dig Surg. 23:370–374. DOI: 10.1159/000097951. PMID: 17164583.

26. Yoon J, Hong S, Hong S, Lee J, Shin S, Park B, et al. Wang L, Dou Q, Fletcher PT, Speidel S, Li S, editors. 2022. Surgical scene segmentation using semantic image synthesis with a virtual surgery environment. p. 551–561. Medical image computing and computer assisted intervention - MICCAI 2022. Springer;DOI: 10.1007/978-3-031-16449-1_53.

27. Strasberg SM, Brunt LM. 2010; Rationale and use of the critical view of safety in laparoscopic cholecystectomy. J Am Coll Surg. 211:132–138. DOI: 10.1016/j.jamcollsurg.2010.02.053. PMID: 20610259.

28. Hu YY, Mazer LM, Yule SJ, Arriaga AF, Greenberg CC, Lipsitz SR, et al. 2017; Complementing operating room teaching with video-based coaching. JAMA Surg. 152:318–325. DOI: 10.1001/jamasurg.2016.4619. PMID: 27973648.

29. Pugh CM, Hashimoto DA, Korndorffer JR Jr. 2021; The what? How? And who? Of video based assessment. Am J Surg. 221:13–18. DOI: 10.1016/j.amjsurg.2020.06.027. PMID: 32665080.

30. Gawria L, Rosenthal R, van Goor H, Dell-Kuster S. ClassIntra Study Group. 2022; Classification of intraoperative adverse events in visceral surgery. Surgery. 171:1570–1579. DOI: 10.1016/j.surg.2021.12.011. PMID: 35177252.

31. Wang DE, Bakshi C, Sugiyama G, Coppa G, Alfonso A, Chung P. 2023; Does operative time affect complication rate in laparoscopic cholecystectomy. Am Surg. 89:4479–4484. DOI: 10.1177/00031348221117032. PMID: 38050322.

32. Vega EA, Salehi O, Loewenthal JV, Kutlu OC, Vellayappan U, Freeman R, et al. 2022; Strategic response to bleeding in laparoscopic hepato-pancreato-biliary surgery: an intraoperative checklist. HPB (Oxford). 24:452–460. DOI: 10.1016/j.hpb.2021.08.944. PMID: 34598880.

33. Rosenthal R, Hoffmann H, Dwan K, Clavien PA, Bucher HC. 2015; Reporting of adverse events in surgical trials: critical appraisal of current practice. World J Surg. 39:80–87. DOI: 10.1007/s00268-014-2776-8. PMID: 25201471.

34. Lucocq J, Scollay J, Patil P. 2022; Evaluation of textbook outcome as a composite quality measure of elective laparoscopic cholecystectomy. JAMA Netw Open. 5:e2232171. DOI: 10.1001/jamanetworkopen.2022.32171. PMID: 36125810. PMCID: PMC9490496.

35. Bamba Y, Ogawa S, Itabashi M, Shindo H, Kameoka S, Okamoto T, et al. 2021; Object and anatomical feature recognition in surgical video images based on a convolutional neural network. Int J Comput Assist Radiol Surg. 16:2045–2054. DOI: 10.1007/s11548-021-02434-w. PMID: 34169465. PMCID: PMC8224261.

36. Madani A, Namazi B, Altieri MS, Hashimoto DA, Rivera AM, Pucher PH, et al. 2022; Artificial intelligence for intraoperative guidance: using semantic segmentation to identify surgical anatomy during laparoscopic cholecystectomy. Ann Surg. 276:363–369. DOI: 10.1097/SLA.0000000000004594. PMID: 33196488. PMCID: PMC8186165.

37. Kaoukabani G, Gokcal F, Fanta A, Liu X, Shields M, Stricklin C, et al. 2023; A multifactorial evaluation of objective performance indicators and video analysis in the context of case complexity and clinical outcomes in robotic-assisted cholecystectomy. Surg Endosc. 37:8540–8551. DOI: 10.1007/s00464-023-10432-z. PMID: 37789179.