Mulago’s take on the LEAP RCT results in Liberia.
Me: I just read the RCT results on LEAP, Liberia’s experiment with outsourced management of government schools.
You: Oh yeah? What do you think?
Me: I think that, with some very doable tweaks, LEAP will have a huge impact going forward.
You: Well of course you’d say that. You funded it.
Me: Fair enough. But….
Me: Look, just give this a fair read and tell me what you think – it’s three pages.
In 2015, Liberia’s education system was in shambles. More than half of the kids were out of school entirely, a quarter of those who made it to the fifth grade couldn’t read a single word, and not a single student passed the college entrance exam. Many teachers were themselves illiterate, and many schools were in ruins. It’s hard to overstate how bad things were.
With little in the way of options, education minister George Werner decided to try something new. He’d visited Bridge Academies, a chain of low-cost private schools in Kenya, and he invited them to Liberia to explore outsourced management of government schools. The idea was to do a 50-school pilot and expand if successful. Mulago put in a million bucks to subsidize the government’s experiment.
Before the launch of the pilot, Minister Werner decided to include an additional seven operators working in 93 schools. This whole enterprise became known as Partnership Schools for Liberia . The program was mostly financed through a pooled philanthropic fund distributed to all the operators. Over the next two years, Mulago put another million bucks into that fund. In year three, the program got a new name: LEAP – the Liberian Educational Access Program. The program has expanded considerably; now, in year four, there are 300 LEAP schools.
The initial pilot of 93 schools was set up as a three-year randomized controlled trial. The researchers, led by Mauricio Romero and Justin Sandefur, did a phenomenal job of gathering the data (often under tough conditions) and crunching the numbers. The result is a treasure trove of information and insights. It’s pretty great – and pretty unusual – to have a study of this depth at this point in the program’s evolution.
The report is mostly concerned with a backward look at happened in the past three years. I’m much more interested in what the research tells us about what the program could and should do to achieve the promise revealed in the study. A good study helps you make good decisions going forward. Mulago funds a lot of early-stage ideas and innovations: the early years are messy, and you have to hang in for a while before a solution’s real potential becomes clear. In their backward-looking report, Romero and Sandefur looked at four domains: learning, access, safety, and cost. Looking forward, here’s what their research tells us about LEAP’s potential and how to realize it.
The cool thing about outsourcing – the fundamental idea, really – is that you can manage toward optimum performance. There was a huge gulf between learning outcomes of the top five and bottom three operators. The dead-obvious thing to do is to get rid of the non-performers. Per the research, that very doable thing actually doubles the program’s effect on learning. Wow!
Other insights about how to tune come from two different ways the researchers look at learning: intent-to-treat (ITT) vs treatment-on-treated (TOT). ITT looks at the program’s effect on all the kids who were assigned to schools – whether they went or not – while TOT looks at the effect on those who actually went. It’s really important to measure both: TOT helps you understand the potential of the intervention, while ITT helps you understand how to tune and manage to make sure kids get it.
In these messy first years of the study, a lot of kids weren’t able attend the schools they were assigned to and this was reflected in a substantial difference between ITT and TOT results. Romero and Sandefur emphasized ITT (as is standard in the academic journals) and pointed out important inequities, mostly having to do with the fact that the biggest operator was working from a completely different contract in the first year. As a result, the Ministry was able to resolve most of those inequities in years two and three. There are still some kids who end up out of school, but it’s mostly due to factors outside the control of operators, and, going forward, it is unlikely that there is any difference between LEAP and non-LEAP schools.
Given the Ministry’s commitment to assuring equal access, the interesting question going forward is “what is the effect on a kid who attends a LEAP school?” To answer that, you look at the TOT results of the top five performers, and it’s here that things get exciting. We’re talking serious gains: the study shows that, in English, the learning gains ranged from 0.4 standard deviations (OK, so standard deviations are this kind of mysterious way that educators report changes in learning. Our education research buddies tell us that they typically find anything over 0.15 pretty exciting.) to 0.9 SD, with an average of 0.6. In math, they ranged from 0.3 to 1.0, with an average of 0.6. Even coming from a low baseline, these are spectacular results: depending on what expert you ask, this translates to the equivalent of one to three extra years of learning for each year of schooling. This is what the program could achieve with doable changes in management, many of which have already been put in place.
This is perhaps the best news in the whole RCT. While there was never any hard target for costs in the program, there was a vague goal of $60 per kid. That came from the difference between the $110 regional average (too low) and the current Liberian spend of $50 (absurdly low). Overall, it seems like a useful number, and what’s great is that most of the operators have already achieved it (one of the top performers actually managed to get down to $37). The average cost dropped by a factor of three between the first and third years of the program and is primed to drop lower in year four. Going forward, it is entirely reasonable for the government to set the operator fee at $60. I suspect that all five of the top operators could meet it.
The most basic thing to say about access is that if you look at those top performers, many thousands of kids got access to a much better education than they could have gotten otherwise. However, it was good that the study took a hard look at overall effects on enrollment. The data show that kids enrolled in programs were an additional 3% more likely to be out of school three years later. Given 1) the chaos of the first years of the program, 2) the fact that the program schools are more rigorous and have longer hours, and 3) much this was mostly an effect generated by one operator, I can live with an initial 3% – especially since the problems that caused it have largely been fixed.
Sexual and physical abuse are common in Liberian schools, and have been for a long time. LEAP schools did no better and no worse. They should have done better. While some of the operators addressed the problem from the outset, we all should have done more. Certainly everyone is taking it seriously now. Operators and government are building real safeguards into operations and policy. The program is making progress and I expect a lot more going forward.
So. Impressive learning gains. Remarkably low cost. Universal access. Real progress toward safety. The RCT reveals both the potential and the changes needed to realize that potential. Many of those changes have been made already, and those still needed are eminently doable. And beyond the RCT itself, there are bunch of things that give me confidence going forward. In no particular order:
If you read Romero and Sandefur’s summary of the results, you’d be forgiven for thinking that the program didn’t accomplish much. They seem to have been in a decidedly sour mood when they wrote it, and the thing is a marvel of negative framing that emphasizes mistakes without acknowledging the effect of corrections. However, with a careful read of the study itself and a basic understanding of how the program unfolded in those messy first years, it’s clear that LEAP represents a remarkable opportunity for Liberia and for those Big Aid funders who are serious about making a difference. I hope they make the most of it.