External Validation of SpineNet, an Open-source Deep Learning Model for Grading Lumbar Disc Degeneration MRI Features, Using the Northern Finland Birth Cohort 1966.

Researchers

Journal

Modalities

Models

Abstract

This is a retrospective observational study to externally validate a deep learning image classification model.Deep learning models such as SpineNet offer the possibility of automating the process of disc degeneration (DD) classification from MRI. External validation is an essential step to their development. The aim of this study was to externally validate SpineNet predictions for DD using Pfirrmann classification and Modic changes (MC) on data from the Northern Finland Birth Cohort 1966 (NFBC1966).We validated SpineNet using data from 1331 NFBC1966 participants for whom both lumbar spine MRI data and consensus disc degeneration gradings were available.SpineNet returned Pfirrmann grade and MC presence from T2-weighted sagittal lumbar MRI sequences from NFBC1966, a dataset geographically and temporally separated from its training dataset. A range of agreement and reliability metrics were used to compare predictions with expert radiologists. Subsets of data that match SpineNet training data more closely were also tested.Balanced accuracy for DD was 78% (77-79%) and for MC 86% (85-86%). Inter-rater reliability for Pfirrmann grading was Lin’s CCC=0.86 (0.85-0.87) and Cohen’s κ=0.68 (0.67-0.69). In a low back pain subset these reliability metrics remained largely unchanged. In total, 20.83% of discs were rated differently by SpineNet compared to the human raters, but only 0.85% of discs had a grade difference greater than 1. Inter-rater reliability for MC detection was κ=0.74 (0.72-0.75). In the low back pain subset this metric was almost unchanged at κ=0.76 (0.73-0.79).In this study, SpineNet has been benchmarked against expert human raters in the research setting. It has matched human reliability and demonstrates robust performance despite the multiple challenges facing model generalizability.Copyright © 2022 Wolters Kluwer Health, Inc. All rights reserved.