On Unstable Fixed Points in Modern Continuous Hopfield Networks

The recently introduced continuous Hopfield network [9] exhibits large memorization capabilities, which manifest as attractive fixed points of its update rule—a differentiable function consisting of two linear mappings composed with the scaled softmax function. The authors of [9] provide proofs for the existence and approximate position of such attractive fixed points. For the softmax function alone, the fixed point structure has been fully characterized in earlier work [10] , from which it turns out that for sufficiently large scaling factors there are exponentially more unstable fixed points than attractive ones. In this work, we complement the findings in [9] by showing that, under natural geometric conditions on the vectors defining the continuous Hopfield network, unstable fixed points must occur, analogous to the findings in [10] . Our results show that, under these geometric conditions, continuous Hopfield networks necessarily admit additional unstable fixed points associated with higher-dimensional faces of the pattern polytope.