Evaluation Tool to Diagnose Faults and Discrepancy in Semi-Automated or Manual Annotations in Ultrasound Cine Loops (Videos).

Researchers

Journal

Modalities

Models

Abstract

Good quality (annotated) data is one of the most important aspects of supervised deep learning. Tasks such as semantic segmentation have a huge data requirement in exchange for only satisfactory performance. Large-scale annotations spread across multiple annotators tends to create inconsistencies, as there are various manual and semi-automated techniques involved. This mandates an external evaluator or expert to check and narrow down the problematic annotations. Studies have shown that even marking a few instances wrong in classification can lead to a significant performance drop in the model (mislabeling only 10% of one class can degrade the total performance of all classes by up to 10%). It has been noticed that fault localization by a medical expert is one of the most expensive and time-consuming processes. In this paper, we propose a novel framework for detecting the inconsistencies in the annotation of every object/anatomy in a specific image. We leverage the power of semi-supervised deep learning models (STCN) to help produce high-quality data for AI segmentation algorithms. Evaluation using this algorithm has been shown to reduce annotation review time by at least 5 hours for just 1000 images, and the quality of ground truth data improved thereby increasing the performance of the model by almost 3%.