Geometric deep learning and equivariant neural networks

Abstract We survey the mathematical foundations of geometric deep learning, focusing on group equivariant and gauge equivariant neural networks. We develop gauge equivariant convolutional neural networks on arbitrary manifolds $$\mathcal {M}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>M</mml:mi> </mml:math> using principal bundles with structure group K and equivariant maps between sections of associated vector bundles. We also discuss group equivariant neural networks for homogeneous spaces $$\mathcal {M}=G/K$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>M</mml:mi> <mml:mo>=</mml:mo> <mml:mi>G</mml:mi> <mml:mo>/</mml:mo> <mml:mi>K</mml:mi> </mml:mrow> </mml:math> , which are instead equivariant with respect to the global symmetry G on $$\mathcal {M}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>M</mml:mi> </mml:math> . Group equivariant layers can be interpreted as intertwiners between induced representations of G , and we show their relation to gauge equivariant convolutional layers. We analyze several applications of this formalism, including semantic segmentation and object detection networks. We also discuss the case of spherical networks in great detail, corresponding to the case $$\mathcal {M}=S^2=\textrm{SO}(3)/\textrm{SO}(2)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>M</mml:mi> <mml:mo>=</mml:mo> <mml:msup> <mml:mi>S</mml:mi> <mml:mn>2</mml:mn> </mml:msup> <mml:mo>=</mml:mo> <mml:mtext>SO</mml:mtext> <mml:mrow> <mml:mo>(</mml:mo> <mml:mn>3</mml:mn> <mml:mo>)</mml:mo> </mml:mrow> <mml:mo>/</mml:mo> <mml:mtext>SO</mml:mtext> <mml:mrow> <mml:mo>(</mml:mo> <mml:mn>2</mml:mn> <mml:mo>)</mml:mo> </mml:mrow> </mml:mrow> </mml:math> . Here we emphasize the use of Fourier analysis involving Wigner matrices, spherical harmonics and Clebsch–Gordan coefficients for $$G=\textrm{SO}(3)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>G</mml:mi> <mml:mo>=</mml:mo> <mml:mtext>SO</mml:mtext> <mml:mo>(</mml:mo> <mml:mn>3</mml:mn> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> , illustrating the power of representation theory for deep learning.