MANTRA game analytics: effectiveness of educational mobile game on knowledge gain and retention of Female Community Health Volunteers (FCHVs) and women in rural Nepal assessed through game analytics

Mobile technology can deliver public health interventions to reach remote populations such as unique mHealth interventions aimed at low-literacy audiences in low resource settings. This research study assessed a mobile phone-based serious game that teaches geohazard, maternal, and neonatal health messages. This study is part of the Maternal and Neonatal Technologies in Rural Areas (MANTRA) project: Increasing maternal and child health resilience before, during, and after disasters using mobile technology in Nepal. We develop a novel games analytics method assessing the knowledge gain and retention by the participants using the in-app collected session data of ordered player responses to game questions as they play and progress through the game. A full description of the study design can be found in (Mueller et al., BMC Public Health 20:1171, 2020). For each learning objective in each level we compare our observations (the data gathered from the players and coded as described above) and the expected frequencies (the number we would have seen as observed if the null hypothesis is true) with a chi – squared hypothesis test. Finally the test statistic is used to find the p value and compared to p = 0.05. Knowledge gain and retention above 25% was calculated for 7 of 30 learning objectives. Results indicate only knowledge gained, so players answering correctly throughout the game are not the focus. In the maternal health module, a p value of < 0.05 was calculated for five learning objectives in level 1 and no significant learning objectives in level 2. The neonatal module level 1 had four learning objectives with significant results, and level 2 had three. The geohazards module showed the lowest significant results of the three modules, with only two significant learning objectives in level 2. Analyzing the MANTRA mobile health game data showed several successful learning objectives across the three modules of maternal health, neonatal health, and geohazards. Success implies the learning objectives and game help participants gain and retain knowledge, while other learning objectives can be targeted for redesign.