Parametric instability in long optical cavities and suppression by dynamic transverse mode frequency modulation

Three-mode parametric instability has been predicted in advanced gravitational wave detectors. Here we present the first observation of this phenomenon in a large scale suspended optical cavity designed to be comparable to those of advanced gravitational wave detectors. Our results show that previous modeling assumptions that transverse optical modes are stable in frequency except for frequency drifts on a thermal deformation time scale is unlikely to be valid for suspended mass optical cavities. We demonstrate that mirror figure errors cause a dependence of transverse mode offset frequency on spot position. Combined with low-frequency residual motion of suspended mirrors, this leads to transverse mode frequency modulation which suppresses the effective parametric gain. We show that this gain suppression mechanism can be enhanced by laser spot dithering or fast thermal modulation. Using Advanced LIGO test-mass data and thermal modeling, we show that gain suppression factors of 10--20 could be achieved for individual modes, sufficient to greatly ameliorate the parametric instability problem.