Resilient Integration of Large Language Models in Microservices Using Circuit Breakers and Fallback Strategies

The integration of Large Language Models (LLMs) into microservice-based systems introduces new resilience challenges due to unpredictable latency, probabilistic outputs, rate limits, and external dependency failures.Traditional fault-tolerance mechanisms are insufficient to address the non-deterministic and resource-intensive nature of LLMbased services.This paper presents a resilience-driven architectural approach for integrating LLMs into microservices using circuit breakers, adaptive timeouts, and intelligent fallback strategies.The proposed design extends classical resilience patterns by incorporating LLM-aware failure detection, response degradation policies, and context-preserving fallbacks such as cached responses, lightweight models, or rulebased services.We analyze failure modes unique to LLM inference, including prompt amplification, cascading retries, and semantic degradation under partial failures.A reference architecture is proposed and evaluated through simulated workloads, demonstrating improved system availability, reduced tail latency, and controlled cost escalation under failure conditions.The results show that resilience-aware LLM integration is essential for production-grade, enterprise-scale microservices and provides a foundation for reliable AI-assisted distributed systems.