Distributed Multi-Cell Power Control with NAF Reinforcement Learning

In this paper, we investigate the power allocation problem to maximize the long-term average downlink sum-rate for orthogonal frequency division multiplexing (OFDM) based multi-cell networks. The traditional centralized training and centralized execution (CTCE) scheme is impractical to solve this complex problem, because it is hard for training center to obtain the global information, and the signaling overhead is also unbearable. To this end, a centralized training and distributed execution (CTDE) framework for wireless power control based on deep reinforcement learning (DRL) is proposed. Specifically, we first transform this problem into a sequential decision problem by designing appropriate state, action and reward. Then, a CTDE scheme, where each agent only requires its own local observations for power allocation, is devised through combining QMIX algorithm and Normalized Advantage Functions (NAF) algorithm. In particular, QMIX network is used to fit the total Q-value function and NAF network is deployed to handle continuous power output. Simulation results show that the proposed scheme converges fast and achieves better rate performance compared with conventional CTDE algorithms.