A DISTRIBUTED SOURCE CODING FRAMEWORK FOR BANDWIDTH-EFFICIENT NEURAL NETWORK PARAMETER UPDATES

Neural networks have helped solve numerous problems and create applications in various fields in the past few years. In order for the neural networks to perform well, the number of parameters has had to grow considerably in size. One problem that this introduces is when one fine-tunes the parameters, a large amount of bandwidth is needed to synchronize other users to use the same parameters. In this paper, we propose a Distributed Source Coding framework that significantly reduces the bandwidth needed to synchronize these parameters. The protocol incorporates a tree-based quantizer in conjunction with Cyclic Redundancy Check (CRC) codes and a significance map. We derive an optimal block size to maximize the bit-rate savings and conduct experiments to show that we can achieve up to 99% bit rate savings with our approach. We further demonstrate how Luby Transform (LT) codes can be used to save additional bandwidth when a sender broadcasts the parameters to many users.