Noise spectral estimation methods and their impact on gravitational wave measurement of compact binary mergers

Estimating the parameters of gravitational wave signals detected by ground-based detectors requires an understanding of the properties of the detectors' noise. In particular, the most commonly used likelihood function for gravitational wave data analysis assumes that the noise is Gaussian, stationary, and of known frequency-dependent variance. The variance of the colored Gaussian noise is used as a whitening filter on the data before computation of the likelihood function. In practice the noise variance is not known and it evolves over timescales of dozens of seconds to minutes. We study two methods for estimating this whitening filter for ground-based gravitational wave detectors with the goal of performing parameter estimation studies. The first method uses large amounts of data separated from the specific segment we wish to analyze and computes the power spectral density of the noise through the mean-median Welch method. The second method uses the same data segment as the parameter estimation analysis, which potentially includes a gravitational wave signal, and obtains the whitening filter through a fit of the power spectrum of the data in terms of a sum of splines and Lorentzians. We compare these two methods and conclude that the latter is a more effective spectral estimation method as it is quantitatively consistent with the statistics of the data used for gravitational wave parameter estimation while the former is not. We demonstrate the effect of the two methods by finding quantitative differences in the inferences made about the physical properties of simulated gravitational wave sources added to LIGO-Virgo data.