Design of a Reinforcement Learning PID controller

This paper addresses a design problem of a Proportional-Integral-Derivative (PID) controller with new adaptive updating rule based on Reinforcement Learning (RL) approach for nonlinear systems. A new design scheme that RL can be used to complement the conventional control technology PID is presented. In this study, a single Radial Basis Function (RBF) network is introduced to calculate the control policy function of Actor and the value function of Critic simultaneously. Regarding to the PID controller structure, the inputs of RBF network are system error, the difference of output as well as the second order difference of output, and they are defined as system states. The Temporal Difference (TD) error in this study is newly defined and involves the error criterion which is defined by the difference between one-step ahead prediction and the reference value. The gradient descent method is adopted based on TD error performance index, then the updating rules can be obtained. Therefore, the network weights and the kernel function can be calculated in an adaptive manner. Finally, the numerical simulations are conducted in nonlinear systems to illustrate the efficiency and robustness of the proposed scheme.