SUS audit aided by natural language processing: A comparative evaluation of BERT models in the analysis of health news

Established in 1988, Brazil’s Unified Health System (SUS) faces significant challenges regarding funding, regional inequalities, and resource oversight, necessitating innovative management solutions. Current auditing processes remain inefficient, as manually detecting irregularities within vast volumes of documents is both time-consuming and costly. This study applies Named Entity Recognition (NER) and text classification techniques to analyze health news relevant to SUS audits. We compare the performance of BERT, BERT-CRF, and ModBERTBr models to identify the most effective approach for optimizing content selection, thereby aiding investigations and combating corruption. A controlled experimental design was employed, following a pipeline of tokenization, label alignment, supervised training, and statistical analysis. Models were evaluated using accuracy, recall, precision, F1-score, and Mean Training Time (MTT). In the NER task, BERT-CRF demonstrated superior performance, achieving the best results in recall (0.880), precision (0.855), and F1-score (0.860). Conversely, the standard BERT model achieved the best overall performance in text classification, significantly outperforming ModBERTBr across all metrics.