Modeling mechanical relaxation in incommensurate trilayer van der Waals heterostructures

The incommensurate stacking of multilayered two-dimensional materials is a challenging problem from a theoretical perspective and an intriguing avenue for manipulating their physical properties. Here we present a multiscale model to obtain the mechanical relaxation pattern of twisted trilayer van der Waals (vdW) heterostructures with two independent twist angles, a generally incommensurate system without a supercell description. We adopt the configuration space as a natural description of such incommensurate layered materials, based on the local environment of atomic positions, bypassing the need for commensurate approximations. To obtain the relaxation pattern, we perform energy minimization with respect to the relaxation displacement vectors. We use a continuum model in combination with the generalized stacking fault energy to describe the interlayer coupling, obtained from first-principles calculations based on density functional theory. We show that the relaxation patterns of twisted trilayer graphene and ${\mathrm{WSe}}_{2}$ are ``moir\'e of moir\'e,'' as a result of the incommensurate coupling two bilayer moir\'e patterns. We also show that, in contrast to the symmetry-preserving in-plane relaxation in twisted bilayers, trilayer relaxation can break the two fold rotational symmetry about the $xy$ plane when the two twist angles are equal.