Investigating the Generalization of Image Classifiers with Augmented Test Sets

Adding prior knowledge about the task or domain being learned can greatly facilitate learning. Two of the most common examples of injecting prior knowledge into Deep Learning systems are architecture design and data augmentation. For example, the convolutional architecture biases the model to learn local features. This emphasis on local features has been a useful prior for image processing. Data augmentation is full of examples that utilize prior knowledge. For example, cropping an image and preserving the original label gives the inductive bias of local feature importance. In this study, we aim to see how the priors in architecture interplay with augmentations. We begin by showing the overall benefit of training with data augmentation, improving the Vision Transformer’s test accuracy from 74.4% to 84.3% and improving the ResNet’s test accuracy from 79.3% to 86.7%. We focus on the distinction between global and local priors such as the difference between the convolution and attention layers and cropping versus noise addition augmentations. These tests are not yet able to find complementing flaws in architectures and augmentations. We find that neither the ResNet or the Vision Transformer is robust to distribution shifts controlled with data augmentation. The performance of both models degrades heavily even with moderate augmentation strengths. Although remedied by explicitly training with the augmentation used to construct the test set, we still see a notable decrease in performance. This study illustrates the utility of generalization testing with data augmentation and the challenge of measuring the impact of global and local priors in architecture.