Reinforcement-Learning-Based Optimal Control of Hybrid Energy Storage Systems in Hybrid AC–DC Microgrids

In this paper, a reinforcement-learning-based online optimal (RL-OPT) control method is proposed for the hybrid energy storage system (HESS) in ac-dc microgrids involving photovoltaic systems and diesel generators (DGs). Due to the low system inertia, conventional unregulated charging and discharging (C&D) of energy storages in microgrids may introduce disturbances that degrade the power quality and the system performance, especially in fast C&D situations. Secondary and tertiary control levels can optimize the state of charge reference of HESS; however, they are lacking the direct controllability of regulating the transient performance. Additionally, the uncertainties in practical systems greatly limit the performance of conventional model based controllers. In this study, the optimal control theory is applied to optimize the C&D profile and to suppress the disturbances caused by integrating HESS. Neural networks are devised to estimate the nonlinear dynamics of HESS based on the input/output measurements, and to learn the optimal control input for bidirectional-converter-interfaced HESS using the estimated system dynamics. Because the proposed RL-OPT method is fully decentralized, which only requires the local measurements, the plug and play capability of HESS can be easily realized. Both islanded and grid-tied modes are considered. Extensive simulations and experiments are conducted to evaluate the effectiveness of the proposed method.