Adaptive Communication Strategies to Achieve the Best Error-Runtime\n Trade-off in Local-Update SGD

Large-scale machine learning training, in particular distributed stochastic\ngradient descent, needs to be robust to inherent system variability such as\nnode straggling and random communication delays. This work considers a\ndistributed training framework where each worker node is allowed to perform\nlocal model updates and the resulting models are averaged periodically. We\nanalyze the true speed of error convergence with respect to wall-clock time\n(instead of the number of iterations), and analyze how it is affected by the\nfrequency of averaging. The main contribution is the design of AdaComm, an\nadaptive communication strategy that starts with infrequent averaging to save\ncommunication delay and improve convergence speed, and then increases the\ncommunication frequency in order to achieve a low error floor. Rigorous\nexperiments on training deep neural networks show that AdaComm can take $3\n\\times$ less time than fully synchronous SGD, and still reach the same final\ntraining loss.\n