Operationalizing ‘what works’ for education in emergencies research

5 factors that facilitate ethical randomized designs

This post is cross-published on FHI 360's R&E Search for Evidence blog.

Conducting research in conflict and crisis contexts can be fraught with operational and ethical challenges that demand innovative approaches. Can we leverage experimental methods to overcome some of these challenges? In a formal or non-formal education setting – even in some emergency contexts – the lottery system can potentially provide a solution. In an environment where there are significantly more eligible pupils than the program can reach, a randomized lottery system gives pupils a non-discriminatory chance at enrollment and also creates the opportunity to set up a control or comparison group, facilitating an experimental research design that allows us to construct a counterfactual (to understand what would have happened if the education program had never existed). 

This is exactly what we are doing in Northeast Nigeria under their Dubai Cares-funded Evidence for Education in Emergencies (E-Cubed) research program*. In order to understand how accelerated education programs shape adolescents’ trajectories and outcomes, we employed a lottery system to enroll participants into a longitudinal (two-year) mixed-methods study involving repeated structured interviews with all participants, and ethnographic methods with a subset of participants. Half of the pupils enrolled in the study were registered for participation in donor-funded, non-formal learning centers (NFLCs), while the rest received minor incentives to participate in the research control group. In partnership with a donor-funded education program implemented by FHI 360 and Save the Children, our research team is working closely with program staff in the home office and in-country, as well as a robust research team from University of Maiduguri (UNIMAID), to actualize this approach. 

The ethical implications of randomizing access to services in order to conduct experimental research in emergency settings have long been debated, and for good reason. Withholding access to services, even in the service of evidence-building, is difficult to justify in conditions of extreme vulnerability and volatility. However, due to limited resources, many education programs in crisis contexts are unable to reach all eligible children anyway. When carried out properly, using a lottery system in a context with oversubscription, limited resources, and high vulnerability may enable programs to distribute services more ethically than a “first come, first serve” approach. The lottery system can address ethical issues around using often-inadequate vulnerability criteria to determine access to programs, providing a roadmap for operationalizing equitable access while also generating evidence through experimental methodologies. 

One of the truisms of humanitarian work is that every context is different, and therefore generalizations across contexts are of limited utility. While researchers and implementers must always use discretion in selecting appropriate and ethical research methods for a given context, we believe that the use of the lottery system hinges on 5 core enabling factors:  

1. Oversubscription: There must be a high demand for access to the education services being researched, and not enough resources to provide services to all, in order for a control group to be viable. 
    
2. Vulnerability: There must be consistent vulnerability among the target population due to displacement status, socioeconomic status, and other factors, which makes the use of vulnerability criteria to determine access to education services particularly challenging (and therefore the lottery system makes sense as a solution). For example, in the city of Maiduguri in Borno state, education implementers often bemoan the fact that “everyone is eligible” for emergency programming. 
    
3. Relative Stability: The population needs to be steady enough that there won’t be complete attrition from the study over time (depending on the length of the study). This will depend on the volatility of the conflict and makes IDP and refugee populations, rather than those inside a conflict zone, particularly well-suited for this kind of research. 
    
4. Community Buy-In & Collaboration: Building community buy-in to promote local ownership of research activities is critical to sustaining the study. This includes working with and through local partners (research institutes, education service providers, and community-based organizations) to make sure that research methods are understood, locally adapted, and acceptable to all before moving forward. 
    
5. Linkages to the Education Program Being Researched: Making use of the lottery system to structure fair provision of services and integrate research into programs relies on the research team having strong linkages to the program. 

Most members of the Education in Emergencies professional community agree that more evidence is needed on “what works,” but doing research in these contexts is difficult for many reasons. It is time to think creatively. We should stop spending so much of our energy on considerations of whether or not randomized designs are ethical, and, instead, focus more on identifying the conditions under which they are ethical; randomization can be a more ethical solution for enrolling pupils in crisis contexts where everyone is vulnerable and use of vulnerability criteria does not necessarily promote equitable access. 

We believe that this study is one of the first of its kind, filling a critical gap in experimental, longitudinal research on the impact of accelerated education programming in crisis and conflict contexts on pupil’s longer-term outcomes. The lottery system may give us a chance to integrate evidence-building into programming in an ethical way and gain insight, ultimately, into “what works” to better serve children affected by conflict and crisis around the world. 

*About this "Evidence for Education in Emergencies" Project: FHI 360, through its subsidiary focused on serving corporations, foundations, and philanthropic organizations - FHI Partners, is implementing the "Evidence for Education in Emergencies" research project in Northern Nigeria, in partnership with the University of Maiduguri, and generously supported by Dubai Cares.

The views expressed in this blog are the author's own.