Multi-level effective surgical workflow recognition in robotic left lateral sectionectomy with deep learning: Experimental research.

Researchers

Journal

Modalities

Models

Abstract

Automated surgical workflow recognition is the foundation for computational models of medical knowledge to interpret surgical procedures. The fine-grained segmentation of the surgical process and the improvement of the accuracy of surgical workflow recognition facilitate the realization of autonomous robotic surgery. This study aimed to construct a multi-granularity temporal annotation dataset of the standardized robotic left lateral sectionectomy (RLLS) and develop a deep learning-based automated model for multi-level overall and effective surgical workflow recognition.From Dec 2016 to May 2019, 45 cases of RLLS videos were enrolled in our dataset. All frames of RLLS videos in this study are labeled with temporal annotations. We defined those activities that truly contribute to the surgery as effective frames, while other activities are labeled as under-effective frames. Effective frames of all RLLS videos are annotated with three hierarchical levels of 4 steps, 12 tasks and 26 activities. A hybrid deep learning model were used for surgical workflow recognition of steps, tasks, activities and under-effective frames. Moreover, we also carried out multi-level effective surgical workflow recognition after removing under-effective frames.The dataset comprises 4,383,516 annotated RLLS video frames with multi-level annotation, of which 2,418,468 frames are effective. The overall accuracies of automated recognition for Steps, Tasks, Activities, and Under-effective frames are 0.82, 0.80, 0.79, and 0.85, respectively, with corresponding precision values of 0.81, 0.76, 0.60, and 0.85. In multi-level effective surgical workflow recognition, the overall accuracies were increased to 0.96, 0.88, and 0.82 for Steps, Tasks, and Activities, respectively, while the precision values were increased to 0.95, 0.80, and 0.68.In this study, we created a dataset of 45 RLLS cases with multi-level annotations and developed a hybrid deep learning model for surgical workflow recognition. We demonstrated a fairly higher accuracy in multi-level effective surgical workflow recognition when under-effective frames were removed. Our research could be helpful in the development of autonomous robotic surgery.Copyright © 2023 The Author(s). Published by Wolters Kluwer Health, Inc.