Learning Time/Memory-Efficient Deep Architectures with Budgeted Super\n Networks

We propose to focus on the problem of discovering neural network\narchitectures efficient in terms of both prediction quality and cost. For\ninstance, our approach is able to solve the following tasks: learn a neural\nnetwork able to predict well in less than 100 milliseconds or learn an\nefficient model that fits in a 50 Mb memory. Our contribution is a novel family\nof models called Budgeted Super Networks (BSN). They are learned using gradient\ndescent techniques applied on a budgeted learning objective function which\nintegrates a maximum authorized cost, while making no assumption on the nature\nof this cost. We present a set of experiments on computer vision problems and\nanalyze the ability of our technique to deal with three different costs: the\ncomputation cost, the memory consumption cost and a distributed computation\ncost. We particularly show that our model can discover neural network\narchitectures that have a better accuracy than the ResNet and Convolutional\nNeural Fabrics architectures on CIFAR-10 and CIFAR-100, at a lower cost.\n