Theory Driven Analysis of Natural Language Processing Measures of Thought Disorder using Generative Language Modeling.
Researchers
Journal
Modalities
Models
Abstract
Natural language processing (NLP) holds promise to transform psychiatric research and practice. A pertinent example is the success of NLP in automatic detection of speech disorganization in formal thought disorder (FTD). However, we lack an understanding of what precisely common NLP metrics measure and how they relate to theoretical accounts of FTD. We propose tackling these questions by using deep generative language models to simulate FTD-like narratives by perturbing computational parameters instantiating theory-based mechanisms of FTD.We simulated FTD-like narratives using GPT-2 by either increasing word selection stochasticity or limiting the model’s memory span. We then examined the sensitivity of common NLP measures of derailment (semantic distance between consecutive words or sentences) and ‘tangentiality’ (how quickly meaning drifts away from the topic) in detecting, and dissociating, the two underlying impairments.Both parameters led to narratives characterized by greater semantic distance between consecutive sentences. Conversely, semantic distance between words was increased by increasing stochasticity, but decreased by limiting memory span. An NLP measure of tangentiality was uniquely predicted by limited memory span. The effects of limited memory span were non-monotonic, in that ‘forgetting’ the global context resulted in sentences that were semantically closer to their local, intermediate, context. Finally, different methods for encoding the meaning of sentences varied dramatically in performance.This work validates a simulation-based approach as a valuable tool for hypothesis generation and mechanistic analysis of NLP markers in psychiatry. To facilitate dissemination of this approach, we accompany the paper with a hands-on Python tutorial.Copyright © 2023 Society of Biological Psychiatry. Published by Elsevier Inc. All rights reserved.