Embedding Confidence to Enhance Trust in AI Document Entity Extraction

Large Language Models (LLMs) are rapidly transforming document understanding by enabling automated extraction of structured data from unstructured sources like resumes and transcripts. Despite high accuracy and efficiency gains, LLMs suffer from a critical limitation: they may hallucinate plausible but incorrect outputs yet provide no explicit confidence measure, undermining reliability in high-stakes domains such as admissions and hiring. This paper presents a practical verification technique for LLM-based entity extraction pipelines, leveraging embedding vector classification to estimate the confidence of each output. We conducted a comparative study of confidence estimation techniques, including LLM self-critique and embeddingbased approaches, on a public resume dataset with synthesized extraction errors. Our findings indicate that embedding-based verification more accurately distinguishes correct from erroneous extractions (F1-score of 0.98), enabling selective flagging of low-confidence fields for human-in-the-loop review. This work advances the adoption of LLMs in sensitive document workflows by providing a scalable, reliable confidence framework.