Nothing should scale up without solid evidence of impact, and all good organizations get to a point where they need rigorous third-party evaluation to go further. The best known approach is the randomized controlled trial (RCT)—I’ll use “RCT” to mean prospective rigorous trials broadly.
If you’re contemplating doing one, good for you, but know this: Get it wrong and you’re screwed. You may get only one shot at this and "RCT" has come to mean "divine judgement" for lots of funders and policy-makers who don't understand them. You're betting the farm.
Here’s what we tell our fellows:
ASK THE RIGHT QUESTION
Make sure the study answers to the end outcome implied by your mission. Too many expensive RCTs dance around the fundamental impact question without answering it.
DON'T DO IT UNTIL YOU ALREADY KNOW THE ANSWER
Wait until your own internal trials, monitoring, and experimentation have allowed you to both dial-in implementation and get a high degree of confidence in your impact. Science being what it is, results may disappoint. Don't launch until you're pretty damn sure you’ve got it right.
DESIGN IT RIGHT
Make sure the proposed study will produce a solid answer. Get 2nd and 3rd opinions if need be. A bad study is worse than nothing. It hurts everyone. If it's a bad design, it's your fault. Try a “premortem”: imagine a bad result, then brainstorm all the ways you would try to explain it away because of design flaws. Fix them.
KNOW YOUR CUSTOMERS
There are people who need your evidence, and there are people you need to persuade — mostly funders and policy-makers — to help you achieve impact at scale. Talk to the people who matter to make sure your study design will address their concerns and answer their questions.
GET MIDLINE RESULTS
Designed right, an RCT can help you tune the model and your implementation of it. Make sure you build in a midline evaluation, and make sure you get data in time to hone the model and implementation.
AGREE AHEAD OF TIME HOW TO RESOLVE CONFLICTS
Stuff absolutely will come up during the study. Everyone's working hard in good faith, but if you don't have a way to work through inevitable disagreements, the best-designed study can turn into a shit-show.
NEGOTIATE HOW RESEARCHERS WILL REPORT RESULTS
Increasingly, researchers report publicly on results at the midline and before publishing. It matters a lot what they select to highlight and how they frame it. Make sure you have a say in it.
We see way too many organizations try to explain away disappointing results. Don’t be one of them. A good RCT is totally worth the expense and hassle, but a bad one is a disaster. You can’t control everything, but there is a lot you can do to prepare, design, and plan for an accurate outcome that serves everyone.
Comments