Investigating the Use of Large Language Models for the Detection of Trauma-Related Symptoms: An Exploratory Study Using Trauma Narratives

This study explores whether Large Language Models (LLMs) can analyze brief trauma narratives in ways that align with self-reported trauma-related symptoms and probable Post-Traumatic Stress Disorder (PTSD) based on a recommended cut-off. We investigate Gemini 1.5 Pro and GPT-4o's ability to score 1000 trauma narratives for severity of PTSD symptoms comparing various prompting strategies. Prompts ranged from basic instructions requesting a binary or graded (5-point) inference based solely on the narrative, to more complex prompts incorporating demographic information (age, gender), time since the traumatic event, and extended instructions detailing PTSD symptomatology. Accuracy of inferences was evaluated using Pearson correlations and Area Under the Curve (AUC) metrics computed between LLM inferences and self-reported measures. Results showed small-to-moderate positive correlations across all prompting strategies, with correlations as high as r = .42 with self-report symptoms. Graded inferences yielded stronger correlations with self-reported symptoms than binary inferences. The AUC for probable PTSD peaked at 0.713 using Gemini 1.5 Pro, with a sensitivity of 0.712 and a specificity of 0.662 in detecting PTSD symptoms. Of note, providing extended instructions on PTSD symptoms did not reliably improve performance beyond basic demographic and temporal context. The performance of LLMs was comparable to that achieved by traditional machine learning methods trained on large datasets but without requiring extensive training data; however, the emerging moderate associations underscore the need for continued refinement. Findings are limited by brief, self-selected English-language web narratives and self-reported symptoms and should be considered preliminary pending validation in clinically verified samples.