Subjective modeling choices and the robustness of impact evaluations in conservation science

Arbitrary modeling choices are inevitable in scientific studies. Yet, few empirical studies in conservation science report the effects these arbitrary choices have on estimated results. I explored the effects of subjective modeling choices in the context of counterfactual impact evaluations. Over 5000 candidate models based on reasonable changes in the choice of statistical matching algorithms (e.g., genetic and nearest distance mahalanobis matching), the parametrization of these algorithms (e.g., number of matches), and the inclusion of specific covariates (e.g., distance to nearest city, slope, or rainfall) were valid for studying the effect of Virunga National Park in Democratic Republic of the Congo on changes in tree cover loss and carbon storage over time. I randomly picked 2000 of the 5000 candidate models to determine how much and which subjective modeling choices affected the results the most. All valid models indicated that tree cover loss decreased and carbon storage increased in Virunga National Park from 2000 to 2019. Nonetheless, the order of magnitude of the estimates varied by a factor of 3 (from -4.78 to -13.12 percentage points decrease in tree cover loss and from 20 to 46 t Ce/ha for carbon storage). My results highlight that modeling choices, notably the choice of the matching algorithm, can have significant effects on point estimates and suggest that more structured robustness checks are a key step toward more credible findings in conservation science.