Improving few-shot named entity recognition for large language models using structured dynamic prompting with retrieval augmented generation

Abstract Biomedical named entity recognition (NER) is a high-utility natural language processing task, and large language models (LLMs) show promise in few-shot settings. In this article, we address performance challenges for few-shot biomedical NER by investigating innovative prompting strategies involving retrieval-augmented generation. Using five biomedical NER datasets, we implemented and evaluated a systematically-structured multi-component static prompt and a dynamic prompt engineering technique, where the prompt is dynamically updated via retrieval with most relevant in-context examples based on the input texts. Static prompting with structured components increased average F 1 -scores by 12% for GPT-4, and 11% for GPT-3.5 and LLaMA 3-70B, relative to basic static prompting. Dynamic prompting further boosted performance and was evaluated on GPT-4, LLaMA 3-70B, and the recently released open-weight GPT-OSS-120B model, with TF-IDF based retrieval yielding the best results, improving average F 1 -scores by 8.8% and 6.3% in 5-shot and 10-shot settings, respectively. An ablation study on retrieval pool size demonstrated that strong performance can be achieved with relatively small number of annotated samples, reinforcing the annotation efficiency and scalability of our framework in real-world settings.