Uncertainty Quantification for the HPCMP CREATE Sage Surrogate Modeling Software

Within the Department of Defense (DoD), the use of surrogate modelling has rapidly expanded to meet demand for fast exploration of parameter trade spaces. By quickly exploring parameter trade spaces with these accelerated surrogate models, design candidates and operating conditions may be considered before committing costly compute resources to traditional physics simulations. To fulfill this demand, the Sage surrogate modeling software tool has been developed, providing a software framework that can be used to create surrogate models for wide-ranging applications across disciplines such as aerodynamics, RF engineering, blast modeling, trajectory prediction, and beyond. In these applications it is often useful to not only predict responses but also predict the associated uncertainties of these predictions. Accurate uncertainty prediction can guide assessment of design and decision risk, measure model trustworthiness and reliability, and steer adaptive trade space exploration. In this work, we present Sage’s uncertainty quantification (UQ) capabilities and features. We outline the collection of state-of-the-art UQ techniques implemented for Sage’s UQ framework, we explain both the metrics and figures of merit used to validate Sage’s UQ performance, and we discuss how to best apply Sage’s UQ techniques for a variety of use cases. We demonstrate Sage’s UQ performance on a case study applied to predicting aerodynamic coefficients for the Open-Source Fighter Geometry as simulated by the HPCMP CREATE Kestrel aerodynamic analysis tool – showcasing accurate uncertainty predictions and the utility of Sage’s uncertainty plots for an end-to-end application. We find Sage demonstrates strong performance for this task. Finally, we show a comparison study of four unique Sage UQ models, deep evidential regression, deep ensemble, and two dropout models, across four aerodynamic datasets. Our comparison study shows that deep ensembles are generally a strong contender for achieving both accurate regression model performance and accurate UQ predictions together.