From CNN to Vision Foundation Models and LLMs: A Multimodal Framework for Pathology Image Retrieval and Auto-Captioning

Background: Pathology image retrieval is crucial for diagnostic support, yet current vision-only retrieval systems often lack semantic interpretability and may generate hallucinated or confabulated outputs, limiting their clinical reliability. Captioning models like MedGemma produce textual descriptions but frequently diverge from expert annotations. To address these issues, we developed a multimodal framework that integrates vision foundation models with large language models (LLMs), leveraging retrieval-driven summarization.