Positive evaluation findings for EduApp4Syria

900 Syrian children in the Azraq refugee camp in Jordan participated in the evaluation of the games Antura and the Letters and Feed the Monster. The evaluation indicates that both games had a small, but positive, impact on literacy levels and psychosocial wellbeing for children playing them.

The evaluation found that:

• Both games led to statistically significant improvements in oral reading fluency, which is a strong predictor of reading comprehension.
  
  
• The effect size for each game, with total playing time of 22 hours (Feed the Monster) and 27 hours (Antura and the Letters) respectively, was equivalent to estimated results from approximately 30 hours of improved classroom instruction in early grade reading. 
  
  
• The games were played with minimum adult intervention, meaning that children were learning on their own.
  
  
• Both games led to improvements in psychosocial wellbeing, as measured with the Strengths and Difficulties Questionnaire (SDQ), whereas there was a decline for the control group.

The evaluation team experienced challenges both related to attrition and dosage, which limits how firm conclusions can be drawn from the data. Some of the involved children dropped out of the study over the eight week period it lasted as they moved within or between camps, or in some cases back to Syria.

Still, the evaluators conclude that the findings are very promising, particularly in light of the widespread smartphone availability among Syrian refugees which makes the approach highly scalable.

Head of jury and technical coordinator Alf Inge Wang is optimistic with regards to the future impact of the two games.

- As a Professor in game based learning I am of course very pleased for evidence that the games help children learn to read. But it is equally important to me that the evaluation finds that the children enjoy playing the games and that parents report that their children are showing a stronger motivation to learn, concludes Wang.

Important milestone

The evaluation has also helped strengthen the two games. Updated versions of the games, which are released now, have been improved based on findings from the evaluation and other tests. A very important milestone in the work on EduApp4Syria, according to project manager Liv Marte Nordhaug.

- The evaluation reports are very thorough and the qualitative and quantitative data have been a great source of knowledge for the game developers and for us. We for instance learnt that too quick progression ended up confusing some of the children, and took this very seriously. Both games now have better paced player progression, better guidance, and have improved the reward systems for players. These are research-based improvements that make it likely that the updated game versions can have an even stronger impact, says Nordhaug.

Since the beta versions of the games were launched one year ago in March 2017, the games have been installed on approximately 80.000 mobile devices, by individual users and by different humanitarian organizations. Feedback on the games and app reviews have been consistently positive.

- Several humanitarian organisations are using the games already. Others have indicated that they want to promote the games if the evaluation findings were positive. We are happy to be able to share the evaluation reports with them.