This is our 'simple enough to do, rigorous enough to mean something' approach to assessing impact.
We measure impact because it's the only way to know whether our money is doing any good. In fact, we don't invest in organizations that don't measure impact – they're flying blind and we would be too. Those organizations that do measure impact perform better and evolve faster, and discussions around measuring impact almost always lead to new ideas about effectiveness and efficiency.
Everyone's got their own definition of impact and here's ours: Impact is a change in the state of the world brought about by an intervention. It's the final result of behaviors (outcomes) that are generated by activities (outputs) that are driven by resources (inputs).
We're a small shop, so we needed to develop an approach with enough rigor to be believable, but simple enough to be doable. When we work with organizations we use these five steps to determine impact and calculate bang for the donor buck.
1. Figure out what you're trying to accomplish: the real mission
You can't think about impact until you know what you're setting out accomplish. Most mission statements don't help that much. We sit down with leaders of organizations and try to come up with a mission statement that is eight words or less. It should include 1) a target population (or setting), 2) a verb, and 3) an ultimate outcome that implies something to measure – like this:
If we can't get to this kind of concise statement, we don't go any further- either because you don't really know what you're trying to do or because we simply wouldn't be able to know if you're doing it.
2. Pick the right indicator
Try this: figure out the single best indicator that would demonstrate mission accomplished. Ignore the howls of protest: it's a really useful exercise. Here are examples relating to the missions shown above:
Sometimes that best indicator is doable, and that's great. Other times you might need to capture it with a carefully chosen – and minimal – combination of indicators. When there is a behavior with a well-documented connection to impact – like children sleeping under mosquito nets – you can measure that and use it as a proxy for impact. Projects that can't at least identify a behavior to measure are too gauzy for us to consider. Notice that while things like "awareness" or "empowerment" might be critical to the process that drives behaviors, we're interested in measuring the change that results from that behavior.
We don't pretend that this method captures all of the useful impacts and accomplishments of a given organization and their intervention. What it does do for us as philanthropic investors is answer the most critical question: did they fulfill the mission?
3. Get real numbers
You need to 1) show a change and 2) have confidence that it's real. This means that:
There are two parts to figuring this out: the logical side and the technical side. With an adequate knowledge of the setting, you can do a lot by just eyeballing the evaluation plan - looking carefully at the methods to be used to see if they make sense. Most bad schemes have an obvious flaw on close examination: they didn't get good baseline data, they ask the dads when they ought to ask the moms, they're asking in a culturally inappropriate way. The technical part has mostly to do with sample size, and a competent statistician can easily help you figure what is adequate.
Measure lasting impact. In most cases it's also vital to measure impact at reasonable intervals to ensure that impacts are sustained. There aren't any hard and fast rules about intervals; usually the nature of a given intervention suggests a suitable interval for follow-up.
4. Make the case for attribution
If you have real numbers that show impact, you need to make the case that it was your efforts that caused the change. This is the hardest part of measuring impact, because it asks you to be able to say what would have happened without you. When real numbers show there has been a change, a useful thing to ask is "what else could possibly explain the impact we observed?"
There are three levels -- in ascending order of cost and complexity -- of demonstrating attribution:
5. Calculate bang-for-the-buck
Now that know you've got real impact, you need to know what it cost. You can always generate impact by spending a ton of money, but it won't give good value for the philanthropic dollar and it won't be scalable (and it probably won't last). Stick with the key impact you've chosen; don't get sucked in to the current trend of trying to monetize every social impact you can think of.
The easiest – and arguably most valid – way to calculate bang-for-the-buck is to divide the total donor money spent by the total impact. In organizations that do more than one kind of project, it is often possible to split out what they spent for their various impacts. Remember that start-ups are expensive and don't worry so much about their current figures, but do see if their projections for steady-state operations make sense and assume (as we learned the hard way) that they are usually at the very optimistic end of the scale.
In the end though, the key to figuring out real impact is an honest, curious, and constructive skepticism. Everyone benefits from a rigorous look at impact: the doers, the funders, the social sector itself, and most importantly, those who are hoping for a brighter day ahead