Feature Selection Using Nearest Neighbor Gaussian Processes

We introduce a novel Bayesian approach for feature (variable) selection using Gaussian process regression, which is crucial for enhancing interpretability and model regularization. Our method employs nearest neighbor Gaussian processes as scalable approximations to classical Gaussian processes. Feature selection is performed by conditioning the process mean and covariance function on a random set representing the indices of relevant variables. A priori beliefs regarding this set control the feature selection, while reference priors are assigned to the remaining model parameters, ensuring numerical robustness in the process covariance matrix. For model inference, we propose a Metropolis-within-Gibbs algorithm. The effectiveness of the proposed feature selection approach is demonstrated through evaluation on simulated data, a computer experiment approximation, and two real-world data sets.