Real-time OCR-based grocery product recognition with orientation alignment and embedding-driven classification

Product recognition is vital for automatic checkouts, inventory management, and fraud detection in grocery stores. However, fine-grained visual differences, variations in packaging and product orientations, and different lighting conditions make recognition challenging. To address these challenges, this paper presents a novel approach for real-time grocery product recognition using open-source optical character recognition (OCR) readers. First, a captured image is passed through a product orientation model, which aligns the image for optimal OCR reading. Then, OCR text is extracted and classified using different text recognition methods, including embedding-based classification and fine-tuning strategies. Experiments on the ProductLeaflets and FineGrain-OCR datasets show that our approach requires minimal training yet achieves recognition performance comparable to a complete fine-tuned model. We also show that our approach enables real-time open-source OCR engines to have competitive performance with commercial cloud OCR engines in OCR text recognition. Finally, our approach can run at 43.3 FPS on the FineGrainOCR dataset while also being fitting for deployment on embedded devices in solutions like self-checkout fraud detection and automatic checkout.