Object-Based Verification of a Prototype Warn-on-Forecast System

An object-based verification methodology for the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) has been developed and applied to 32 cases between December 2015 and June 2017. NEWS-e forecast objects of composite reflectivity and 30-minute rotation tracks of updraft helicity are matched to corresponding objects in Multi-Radar Multi-Sensor data on space and time scales typical of a National Weather Service warning. Object matching allows contingency table-based verification statistics to be used to establish baseline performance metrics for NEWS-e thunderstorm and mesocyclone forecasts. NEWS-e critical Success Index (CSI) scores of reflectivity (updraft helicity) forecasts decrease from approximately 0.7 (0.4) to 0.4 (0.2) over 3 hours of forecast time. CSI scores decrease through the forecast period, indicating that errors have not saturated and skill is retained at 3 hours of forecast time. Lower verification scores for rotation track forecasts are primarily a result of a high frequency bias. Comparison of different system configurations used in 2016 and 2017 show an increase in skill for 2017 reflectivity forecasts, attributable mainly to improvements in the forecast initial condition. A small decrease in skill in 2017 rotation track forecasts is likely a result of sample differences between 2016 and 2017. Although large case-to-case variation is present, evidence is found that NEWS-e forecast skill improves with increasing object size and intensity, as well as in mesoscale environments in which an enhanced or higher risk of severe thunderstorms was forecast.