_________________________________________________________
Me: I just read the RCT results on LEAP, Liberia’s experiment with outsourced management of government schools.
You: Oh yeah? What do you think?
Me: I think that, with some very doable tweaks, LEAP will have a huge impact going forward.
You: Well of course you’d say that. You funded it.
Me: Fair enough. But….
You: …….
Me: Look, just give this a fair read and tell me what you think – it’s three pages.
_________________________________________________________
So.
In 2015, Liberia’s education system was in shambles. More than half of the kids were out of school entirely, a quarter of those who made it to the fifth grade couldn’t read a single word, and not a single student passed the college entrance exam. Many teachers were themselves illiterate, and many schools were in ruins. It’s hard to overstate how bad things were.
With little in the way of options, education minister George Werner decided to try something new. He’d visited Bridge Academies, a chain of low-cost private schools in Kenya, and he invited them to Liberia to explore outsourced management of government schools. The idea was to do a 50-school pilot and expand if successful. Mulago put in a million bucks to subsidize the government’s experiment.
Before the launch of the pilot, Minister Werner decided to include an additional seven operators working in 93 schools. This whole enterprise became known as Partnership Schools for Liberia . The program was mostly financed through a pooled philanthropic fund distributed to all the operators. Over the next two years, Mulago put another million bucks into that fund. In year three, the program got a new name: LEAP – the Liberian Educational Access Program. The program has expanded considerably; now, in year four, there are 300 LEAP schools.
The initial pilot of 93 schools was set up as a three-year randomized controlled trial. The researchers, led by Mauricio Romero and Justin Sandefur, did a phenomenal job of gathering the data (often under tough conditions) and crunching the numbers. The result is a treasure trove of information and insights. It’s pretty great – and pretty unusual – to have a study of this depth at this point in the program’s evolution.
The report is mostly concerned with a backward look at happened in the past three years. I’m much more interested in what the research tells us about what the program could and should do to achieve the promise revealed in the study. A good study helps you make good decisions going forward. Mulago funds a lot of early-stage ideas and innovations: the early years are messy, and you have to hang in for a while before a solution’s real potential becomes clear. In their backward-looking report, Romero and Sandefur looked at four domains: learning, access, safety, and cost. Looking forward, here’s what their research tells us about LEAP’s potential and how to realize it.
Learning
The cool thing about outsourcing – the fundamental idea, really – is that you can manage toward optimum performance. There was a huge gulf between learning outcomes of the top five and bottom three operators. The dead-obvious thing to do is to get rid of the non-performers. Per the research, that very doable thing actually doubles the program’s effect on learning. Wow!
Other insights about how to tune come from two different ways the researchers look at learning: intent-to-treat (ITT) vs treatment-on-treated (TOT). ITT looks at the program’s effect on all the kids who were assigned to schools – whether they went or not – while TOT looks at the effect on those who actually went. It’s really important to measure both: TOT helps you understand the potential of the intervention, while ITT helps you understand how to tune and manage to make sure kids get it.
In these messy first years of the study, a lot of kids weren’t able attend the schools they were assigned to and this was reflected in a substantial difference between ITT and TOT results. Romero and Sandefur emphasized ITT (as is standard in the academic journals) and pointed out important inequities, mostly having to do with the fact that the biggest operator was working from a completely different contract in the first year. As a result, the Ministry was able to resolve most of those inequities in years two and three. There are still some kids who end up out of school, but it’s mostly due to factors outside the control of operators, and, going forward, it is unlikely that there is any difference between LEAP and non-LEAP schools.
Given the Ministry’s commitment to assuring equal access, the interesting question going forward is “what is the effect on a kid who attends a LEAP school?” To answer that, you look at the TOT results of the top five performers, and it’s here that things get exciting. We’re talking serious gains: the study shows that, in English, the learning gains ranged from 0.4 standard deviations (OK, so standard deviations are this kind of mysterious way that educators report changes in learning. Our education research buddies tell us that they typically find anything over 0.15 pretty exciting.) to 0.9 SD, with an average of 0.6. In math, they ranged from 0.3 to 1.0, with an average of 0.6. Even coming from a low baseline, these are spectacular results: depending on what expert you ask, this translates to the equivalent of one to three extra years of learning for each year of schooling. This is what the program could achieve with doable changes in management, many of which have already been put in place.
Cost
This is perhaps the best news in the whole RCT. While there was never any hard target for costs in the program, there was a vague goal of $60 per kid. That came from the difference between the $110 regional average (too low) and the current Liberian spend of $50 (absurdly low). Overall, it seems like a useful number, and what’s great is that most of the operators have already achieved it (one of the top performers actually managed to get down to $37). The average cost dropped by a factor of three between the first and third years of the program and is primed to drop lower in year four. Going forward, it is entirely reasonable for the government to set the operator fee at $60. I suspect that all five of the top operators could meet it.
Access
The most basic thing to say about access is that if you look at those top performers, many thousands of kids got access to a much better education than they could have gotten otherwise. However, it was good that the study took a hard look at overall effects on enrollment. The data show that kids enrolled in programs were an additional 3% more likely to be out of school three years later. Given 1) the chaos of the first years of the program, 2) the fact that the program schools are more rigorous and have longer hours, and 3) much this was mostly an effect generated by one operator, I can live with an initial 3% – especially since the problems that caused it have largely been fixed.
Safety
Sexual and physical abuse are common in Liberian schools, and have been for a long time. LEAP schools did no better and no worse. They should have done better. While some of the operators addressed the problem from the outset, we all should have done more. Certainly everyone is taking it seriously now. Operators and government are building real safeguards into operations and policy. The program is making progress and I expect a lot more going forward.
So. Impressive learning gains. Remarkably low cost. Universal access. Real progress toward safety. The RCT reveals both the potential and the changes needed to realize that potential. Many of those changes have been made already, and those still needed are eminently doable. And beyond the RCT itself, there are bunch of things that give me confidence going forward. In no particular order:
- LEAP was instigated and led by government, in the face of indifference and in some cases outright opposition from its traditional development partners. The development community celebrates "country ownership.” This is what it looks like in practice.
- The program survived a fiercely contested Presidential election that resulted in a different party in power and a 100% turnover in the senior leadership of the Ministry of Education. That it was ultimately adopted by the new team is pretty extraordinary given how controversial it had been in some quarters. But the reason was pretty simple: when the new Ministers embarked on a nationwide tour and visited LEAP schools, they liked what they saw.
- LEAP has expanded to include 75,000 students in 300 schools . The really impressive thing is that much of that successful expansion was in the difficult-to-operate and long-neglected southeast of the country. This is the poorest region of one of the world’s poorest countries. Liberia is one of the most difficult places in the world deliver something like LEAP, and the work of the operators has been nothing short of heroic.
- This all happened in a country where very little has worked. The only other education intervention that people talk about in Liberia – the USAID-funded EGRA Plus program - delivered big literacy gains, but it was over a decade ago, reached fewer than 3,000 students, cost a ridiculous $388 per student, and proved impossible to scale.
If you read Romero and Sandefur’s summary of the results, you’d be forgiven for thinking that the program didn’t accomplish much. They seem to have been in a decidedly sour mood when they wrote it, and the thing is a marvel of negative framing that emphasizes mistakes without acknowledging the effect of corrections. However, with a careful read of the study itself and a basic understanding of how the program unfolded in those messy first years, it’s clear that LEAP represents a remarkable opportunity for Liberia and for those Big Aid funders who are serious about making a difference. I hope they make the most of it.
Comments