Think Rapid, Rigorous Evidence in Emergency settings is impossible? Think Again!

We all know that data is critical in helping to identify gaps, target interventions, monitor impact, and make evidence-based decisions. Ideally, to make real-time, data-driven action, the evidence needs to be collected and analyzed quickly. Unfortunately, in crisis settings, one of the major difficulties in education is the lack of accurate, reliable, and timely data. Recent studies conducted across six countries during the COVID-19 pandemic and super typhoon Rai, however, provide insights into how we can overcome these challenges and common research misconceptions. 

Results released last week from across five diverse countries reveal that targeted phone-based tutoring can positively impact learning outcomes during both health and climate emergencies.  These results – from a coalition of NGOs, governments, multilaterals, and researchers in six countries (Botswana, India, Kenya, Nepal, Philippines, and Uganda) - offer some of the most rapid and comprehensive multi-country evidence generated on an education intervention. Moreover, they help fill a critical evidence gap around the effectiveness of mobile phone solutions in emergency settings.

Conducted between 2020-2022, the trials looked at the impact of phone-based tutoring on mathematical learning outcomes. The first trial, based in Botswana in 2020 and conducted by a local NGO called Youth Impact, demonstrated a 31% reduction in innumeracy. Prompted by this success, entities across five countries adapted and trialed the intervention in their respective contexts. On average, across all countries, there was a 65 percent increase in the percent of students who learned division—effectively mastering all basic numeracy concepts delivered in the course of programming. 

Youth Impact has been conducting rapid, rigorous trials since its inception in 2014. In the past year alone, they have completed 15 trials across all their programs and with partners.  While the impressive results from the aforementioned study speak for themselves, what is equally as important is the underlying research implications for entities working in crisis settings, as well as those in many low- and middle-income settings. 

1. Rigorous research can be conducted rapidly and in emergency settings. A typical randomized controlled trial (RCT) can take years to complete, but in this study, all six RCTs were conducted within two years – a massive feat, especially considering many of the partnering organizations were new to conducting such research and were implementing amidst crisis settings. The initial Botswana trial took place during COVID-19 lockdowns and school closures. A typhoon hit the Philippines during their trial and a catastrophic wave of COVID-19 hit India – and the Alokit team - in the midst of their implementation.  Because the premise of the intervention – a remote, phone-based tutoring program – was built to be a resilient education response, the trials were able to continue even when crises occurred. 
    
2. You don’t need to be an academic institution to conduct quality research. A diverse array of NGOs, governments, multilaterals, and researchers conducted these phone-based trials, many of whom had never conducted a RCT before. In the Philippines, the trial was conducted in partnership with the Department of Education, and was the first RCT ever conducted by the department.  Alokit, an organization based in India specializing in school leadership development, completed a full-scale RCT with a small handful of staff members. These trials prove that all types of organizations – not just universities and research institutions - can collect and conduct high-quality research. In fact, local organizations  - especially those with larger-scale operations already running - have the strong advantage of being able to collect data more quickly and easily than outside institutions. 
    
3. Timely and relevant data spurs action and informs ongoing adaptations. Data collected in real-time allows iterations and adaptations which leads to improved impact. For example, during the Botswana study, evidence quickly revealed that despite being cheaper and more scalable, SMS messages alone were not enough to significantly improve student learning. Rather, targeted instruction delivered through interactive phone calls are the primary drivers of learning. As a result, Youth Impact pivoted to this combined SMS+ phone call model for increased learning outcomes. Moreover, the rapid results of the Youth Impact trial in Botswana catalyzed the initiation of the follow-on replication trials. The results were disseminated within weeks of the trial conclusion and, encouraged by the positive impact, partners reached out with interest in the evidence-based solution.   

While there is still much to learn about phone-based learning, the growing evidence behind ConnectEd, the name Youth Impact has now dubbed for this program,  offers an example of how an emergency-born innovation developed for one context can successfully expand in new settings through a rapid, iterative, data-driven, and collaborative process. Moreover, this example shows how rigorous research can be led by local organizations and in demanding situations. Youth Impact is preparing to implement ConnectEd in more emergency contexts beyond COVID-19, including Afghanistan, where they have established a partnership with the Community Driven Development Organization (CDDO) and Uplift Afghanistan. Together, they will launch ConnectEd pilots to reach girls who are unable to access formal education. Undoubtedly, they will collect more data to add to the evidence base of mobile solutions in emergency settings, as well as further cement the power of rapid and rigorous research in crisis-affected countries.  

Slow and steady doesn’t always have to win the race.

For more information about ConnectEd, please visit: https://www.youth-impact.org/connected.

Karen Clune is the Head of Innovation and Development for Youth Impact, an evidence-based health and education NGO based in Botswana and working across Sub-Saharan Africa. Karen brings over 20 years of global development experience. She has led innovative new programs for the U.S. Agency for International Development and Operation Smile, and worked as a community health agent for Peace Corps Jamaica and Senegal. She holds a MPH in Global Health from George Washington University and a B.A. in History from the University of Virginia.