Self-attention in vision transformers using ridge regression and the kernel method

The Vision Transformer (ViT) effectively applies the Transformer architecture to computer vision, treating each small image patch as a token and learning their relationships with self-attention. However, the standard dot-product attention can yield numerically unstable and difficult-to-interpret weights. This research introduces Ridge Attention, which recasts the weight-estimation step as a ridge-regression problem by adding a small amount of <i>L</i>₂ regularization when computing token similarity. This modification leaves the ViT’s core architecture untouched while producing more stable and statistically meaningful attention weights. Experimental results on CIFAR-10 and CIFAR-100 show that replacing pure similarity scores with regularized regression improves performance without requiring additional parameters.