An evaluation on an ultra-poverty graduation project in Kenya/Uganda just came out. People are going nuts over the results: "powerful!" "incredible!" They're "lifting people out of poverty!" I read the full report. My own reaction was "WTF!"
It's a well-done RCT (see for yourself). Here are the results: a 6% increase in consumption and a 6% increase in assets. That's from a really low baseline. For perspective, it would be like increasing the US minimum wage by 43 cents an hour.
On the Uganda side of the study, those figures were 3.6% and 2.3%, respectively. That translates to an extra 4 bucks a month of consumption and the accumulation of 15 bucks worth of stuff. Yup. That's it. On what planet does that constitute "lifting people out of poverty?" Are we so in thrall to RCTs that we assume "statistically significant" means "materially consequential?"
Effect sizes matter! A bar graph from the report - net asset increase in Uganda - gives a sense of the magnitude of these changes:
We're told that people buy additional "food, goods, and health care services." How much of that stuff do you buy with $4? We also learn that they purchase "livestock, furniture, savings, and business supplies." Where does one get a $15 cow?
What, exactly, are these families "graduating" out of? The authors claim that we're going to see "$21m in lifetime benefits" How could that be? IN THESE SAME 2 COUNTRIES, the effects of $1000 cash transfers dissipated within four years!
And then you lard up the whole thing with the hefty costs of a DIB, including an 8% return to investors. This is justified because the NGO "exceeded targets" and the NGO "introduced an adaptive management system with customized dashboards."
Well, hurrah for low targets! And anyway, a lot of organizations are putting in great performance management systems, adaptive dashboards and all. It's more like if you don't have one, then why the hell not?
And what are DIBs like this for? Will they drive the bulk of the NGO's future implementation (no.) Is it a one-time process that spectacularly/lastingly boosts performance? (unlikely). Did anything happen here that couldn't be accomplished with smart grant funding? (no).
Is it "incredible" that the NGO managed to replicate their pre-COVID results? Yes! 100%! Celebrate that! Other than that, tho, the sizzle is vastly out of proportion to the steak. It isn't "lifting people out of poverty" - not even close. Bolting on a DIB doesn't help.
I admire this NGO; they do solid work and this IS a cost-effective intervention. But "cost-effective" does not necessarily mean "materially consequential" either, and the sad truth is that it's hard to transform lives on the cheap. And enough "OMG we did a DIB!" Sorry to be the skunk at the picnic, but when the % return to investors exceeds the % by which you increased desperately small incomes, it's probably best to declare your victory quietly.
Comments