Fast approximate matrix multiplication for 8-bit neural networks using tree averaging

Deploying neural networks on resource-constrained devices (e.g., mobile and embedded systems) for computer vision and image processing tasks requires computationally efficient algorithms. For this goal, 8-bit quantization is widely adopted to accelerate neural network inference while maintaining accuracy. However, matrix multiplication, a core operation in deep learning, often faces performance bottlenecks due to limited hardware capabilities. This paper introduces a novel tree averaging accumulation method for 8-bit quantized matrix multiplication. The proposed approach leverages averaging instructions to perform all intermediate computations in 16-bit registers, avoiding overflow and reducing the need for 32-bit accumulators. Experimental evaluations demonstrate a consistent speedup of 20% across various matrix sizes, with minimal impact on numerical accuracy (less than 0.05% deviation). Experiments on LeNet-5 (MNIST), ResNet-20 and DenseNet-40 (CIFAR-10) show that replacing standard convolutional layers with the layers with tree averaging results in negligible accuracy degradation — less than 0.1% for LeNet-5 and under 1% for ResNet-20 and DenseNet-40. Moreover, after the short fine-tuning the accuracy is restored.