Leveraging Local-LLM for sentiment analysis to enhance text data quality assessment in information science research

Purpose This study explores how a privacy-preserving local large language model (Local-LLM) assists researchers in evaluating unstructured text data quality, early in the analysis process. It aims to accurately and effectively determine if a Local-LLM can evaluate the quality of open-ended survey responses compared to human coders during the initial analysis of unstructured survey data. Design/methodology/approach A Python tool using Llama 3.2 and 3.3 classified 604 survey responses. Human-coded sentiment labels served as a baseline; model performance was assessed with confusion matrices, F1, Cohen’s κ and Gwet’s AC1. All processing was offline to protect data privacy. Findings Llama 3.3 achieved top performance (F1 ≈ 0.97, AC1 ≈ 0.97), while Llama 3.2 also excelled on consumer hardware. Automated sentiment analysis reduced processing time from 4 h to 8 min and identified short responses that manual reviewers might miss, improving speed and data quality. Research limitations/implications The approach’s reliability beyond English or on longer narratives remains to be examined in future work. Practical implications The Local-LLMs allow researchers to rapidly filter and assess large volumes of unstructured text data before conducting deeper analysis. Originality/value This study demonstrates that Local-LLM’s semantic features can be used for sentiment analysis as a scalable, cost-effective and rapid method for evaluating unstructured text data, aligning with the research questions before analysis. It also illustrates how to use zero-shot prompting to interact with Ollama via Local-LLMs, simplifying AI integration for non-technical researchers.