GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash\n Equilibrium

Generative Adversarial Networks (GANs) excel at creating realistic images\nwith complex models for which maximum likelihood is infeasible. However, the\nconvergence of GAN training has still not been proved. We propose a two\ntime-scale update rule (TTUR) for training GANs with stochastic gradient\ndescent on arbitrary GAN loss functions. TTUR has an individual learning rate\nfor both the discriminator and the generator. Using the theory of stochastic\napproximation, we prove that the TTUR converges under mild assumptions to a\nstationary local Nash equilibrium. The convergence carries over to the popular\nAdam optimization, for which we prove that it follows the dynamics of a heavy\nball with friction and thus prefers flat minima in the objective landscape. For\nthe evaluation of the performance of GANs at image generation, we introduce the\n"Fr\\'echet Inception Distance" (FID) which captures the similarity of generated\nimages to real ones better than the Inception Score. In experiments, TTUR\nimproves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP)\noutperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN\nBedrooms, and the One Billion Word Benchmark.\n