Deep multi-instance learning using multi-modal data for diagnosis of lymphocytosis.

Researchers

Journal

Modalities

Models

Abstract

We investigate the use of recent advances in deep learning and propose an end-to-end trainable multi-instance convolutional neural network within a mixture-of-experts formulation that combines information from two types of data—images and clinical attributes—for the diagnosis of lymphocytosis. The convolutional network learns to extract meaningful features from images of blood cells using an embedding level approach and aggregates them in order to associate them with lymphocytosis, while the mixture-of-experts model combines information from these images as well as clinical attributes to form an end-to-end trainable pipeline for multi-modal data. Our results demonstrate that even the convolutional network by itself is able to discover meaningful associations between the images and the diagnosis, indicating the presence of important unexploited information in the images. A repeatability study is performed to demonstrate that the mixture-of-experts formulation is shown to be more robust without loss of performance. The proposed methods are compared with different methods from literature based both on conventional radiomics and machine learning, and on recent deep learning models based on attention mechanisms. Our method reports a balanced accuracy of 85.41% and outperfroms the radiomics-based and attention-based approaches as well that of biologists which scored 79.44%, 82.89% and 77.07% respectively. These results give insights on the potentials of the applicability of the proposed method in clinical practice.