Beyond the Leaderboard: Evaluating the Robustness of Deep Learning Models for Detecting Freezing of Gait

Freezing of gait is a common, debilitating symptom that affects many patients with Parkinson's disease. The lack of a standard, objective method to quantify freezing obstructs research and treatment. Wearable sensors combined with automatic detection algorithms have demonstrated increasingly promising results; nonetheless, video annotation remains the gold standard. After organizing a global machine learning contest to expedite the development of acceleration-based algorithms designed to automatically detect freezing, we tested the transferability of the winning models to a new dataset. Experts reviewed and annotated a test protocol conducted and videotaped in the homes of 12 patients. The models were applied to acceleration data from a lower back sensor worn by the patients. F1-scores, accuracy, recall, specificity, and precision were computed. Intraclass correlations quantified the agreement between model-estimated and annotation-based gold standard outcomes, including the percent time frozen, the number of episodes, and the total freezing duration. While there was a relatively large drop in performance for some of the models, the performance of the third place model showed good transferability to new data. Indeed, the agreement between the third place model and gold standard annotations was similar to or better than that seen when comparing two raters. These results further support the idea that if the goal is to detect freezing duration or percent time frozen, the combination of a single, lower back sensor and the third place model can be used to automatically detect freezing. Still, if the goal is to count episodes or detect freezing subtypes, additional sensors or other modelling approaches are needed.