Measurement Library Launched for Researchers in Middle East and North Africa

This post originally appeared in USAID ECCN's blog and can be found here.

Children are falling behind in learning worldwide. According to a 2017 UNESCO report more than six in ten children and youths fail to achieve minimum proficiency in reading and math. This is especially pronounced in societies upended by conflict and natural disaster, where young people approach adulthood carrying the weight of trauma and lacking the basic skills necessary to lead healthy, productive lives. Those committed to education in emergencies (EiE) also do not have the high-quality data they need to determine whether their programs are working.

In response, the International Rescue Committee (IRC) and NYU Global TIES for Children (TIES/NYU) launched a strategic partnership which convened a consortium of eight research-practice partnerships active in the Middle East, North Africa and Turkey (MENAT) region*. This consortium developed, adapted and tested a set of measurement tools meant to assess children’s learning and holistic development in crisis and conflict settings. One specific result of this consortium will be the launch of a MENAT Measurement Library, a collection of these tools which will serve as a resource for EiE stakeholders for years to come.

The Measurement Library will be open-source, available to all but targeted to those working with children in crisis in the MENAT region. The Library will aim to set a new and higher standard for the tools the field uses to assess and improve EiE programs.

Researchers who work with students in crisis contexts face multiple challenges, including:

• We lack different, varied methods to assess children’s learning and holistic development, including their social-emotional learning (SEL), in crisis contexts. Unlike outcomes such as literacy and numeracy, where we use performance-based tests to assess students’ knowledge and skills, self-reports constitute the vast majority of tools used to assess children’s SEL in EiE programs. Self-reports are an important instrument in our Measurement Library, but they can provide a biased picture of children’s social-emotional skills, because children may overestimate or underestimate their performance and provide answers they think others will approve of. Going beyond these measures in conflict and crisis-affected settings will give us more a more complete understanding of children’s learning and development in crisis contexts.
• There are few SEL measures with evidence of validity and reliability in crisis and conflict contexts. Very few SEL instruments demonstrate adequate psychometric properties that can be used for rigorous research and impact evaluations. In their review of SEL measures, Halle and Darling-Churchill identified 75 existing measures of SEL, out of which only six showed good evidence of validity and reliability. In conflict and crisis settings, even fewer rigorously tested tools exist to assess children’s SEL competencies.
• SEL tools lack training materials and those available may not be intuitively designed. There is a scarcity of training materials that are easy to understand, and teams often do not have the time and resources to create user-friendly training and capacity-building materials. This limits the ability to build capacity and generate the sort of data that facilitates informed decision-making.

Given these barriers, even the most experienced practitioners make decisions based on intuition, rather than facts. Practitioners need an approach based on evidence. The Measurement Library provides the necessary tools, encompassing a range of assessment methods, from performance-based tasks and scenario-based items to observation tools. The Library will include other assessment methods, such as:

• Children’s Stories: This scenario-based tool describes situations children commonly experience and asks them to imagine what they might feel or do in response. This tool measures children’s hostile attribution bias, emotional regulation/dysregulation, aggression, problem solving, and disengagement.
• Teacher Classroom Observation (TCO): This observation tool assesses teachers’ instructional methods and is useful for evaluation and coaching.

For each measure, the Measurement Library will also include information about the sample (age group, country, language) for which the measure was tested, for what purpose, and evidence of that measure’s validity and reliability. By sharing this information, the Library will provide a more thorough, transparent collection of information.

Decision-making tree: The Measurement Library will provide an informative tool that includes a step-by-step guide that walks the user through the key questions they need to be asking themselves to select and appropriately use the right measure for the right purpose. It will provide resources to guide users in the process of answering those questions, identifying whether there are measures available in the library to meet their needs, and/or guidance on the minimum actions they need to engage in to adjust them for their own purposes.  In the process, the decision-making tree will help the user hone in on the specific ways they need to be thinking about SEL assessment to make informed decisions based on high-quality data.

Training materials: The Measurement Library will also strive to ensure the tools it offers are user-centered and easy to grasp, presented in an engaging and accessible manner. Working toward that goal has entailed enlisting the expertise of a user-experience designer for one of our measures. Numerous participants commented that the TCO instrument was overly-complicated and excessively technical, in particular through the formatting of its features and the language it used. Noting this feedback, the designer was able to overhaul the TCO with an emphasis on presenting it in a more intuitive and accessible fashion.

A better approach to using rigor and evidence to meet the educational and developmental needs of young people in societies torn apart by crisis is possible—but a shift is needed. The Measurement Library will do its part to ensure researchers, practitioners, policymakers and funders can engage in learning and applying what they learn to accomplish this.

*American University of Beirut and Ministry of Public Health (Lebanon); New York University (Sirin), Bahçeşehir University and Hacettepe University; World Learning, Lebanese Ministry of Education and Higher Education Pedagogical and Scholastic Guidance Office (DGE-DOPS), Lebanese Center for Education Research and Development (CERD); Norwegian Refugee Council and University of Tromso; Save the Children; The International Rescue Committee and New York University Global TIES for Children; Queen Mary University of London (QMUL), Institute for Development, Research, Advocacy, and Applied Care (IDRAAC), and Médecins du Monde (MdM); War Child Holland and Kings College London

The views expressed in this blog are the author's own.