Idealistics is shutting down. This blog will continue to be updated at fullcontactphilanthropy.com.
How to select measurable outcomes

At the end of last month I wrote a post advising organizations to setup their evaluation frameworks prior to advertising their greatness to the world, as credible evidence to back up claims of effectiveness is infinitely more persuasive than the alternative. In the post I gave an example of a consultant who helped a former customer of mine create a logic model with poorly defined outcomes, positioning my customer for advertisement rather than measurement. From the post:

One such consultant, who I was later hired to replace to clean up his mess, outlined a data collection strategy that included an outcome of “a healthy, vibrant community free of crime where all people are treated with respect and dignity.”

What an operationalization nightmare. How the heck do you measure that? You don’t. And that’s the point. The logic model was not a functional document used for optimizing results. Instead, it was an advertising document to lull unsavvy donors into believing an organization is effective in the absence of evidence.

Fair criticism, I think (obviously). Well, yesterday Jennifer Banks-Doll asked an even fairer question:

David, I really enjoy your blog and love the example you have given above about the unmeasurable outcome.  So true and so common!  But I feel like you’ve left us hanging…What kind of outcomes would you recommend instead?  Can you give us an example or two?

This is an excellent question, and while I did my best to give an example of the right type of thought process an organization should go through to operationalize outcomes, unfortunately the best answer I can really give is the unsatisfying “it depends”.

While there are those who tend to think through their logic models from left to right, that is, if I do this I expect that, I tend to think the other way around, starting with the goal an organization aims to achieve and moving backwards. There is nothing wrong with beginning the ideation process and goal setting with seemingly unmeasurable ideals. However, what begins as pie-in-the-sky should not be left there.

In my response to Jennifer’s question, I gave the example of an organization trying to create a “safe” neighborhood. Safety is an inherently abstract notion. Part of safety might be actual incidences of crime, but safety might also have to do with perceptions as well. If someone feels unsafe in an area free of crime, is that area “safe”? Well, it depends on how an organization defines its goals.

In the process of creating more exacting definitions of the change you want to see in your target population, you get closer to identifying measurable indicators. The goal is not to create one measure for “safety”, or whatever your outcome of interest is. Instead, what we want to do is come up with a few measures that collectively approximate what our otherwise abstracted outcome is.

One other note on this point. People tend to think that “measurable” necessarily means inherently enumerable. Working again with our safety example, one measure of safety might be crime rates, but another measure might be an opinion survey of residents in an area, asking them if they feel safe. If I had to choose one, my preference would be for the perception survey over the crime data.

Crime statistics, while numeric by nature, are not necessarily pure measures of “safety”. If arrests increase in a neighborhood, this might be due to an increase in criminal activity, but it also might mean there are more police patrolling the area. Indeed, in some of the community development work I have done, I have found (in some cases) that perceptions of safety actually increase with the number of reported crimes.

The point I am trying to illustrate is that organizations should not feel pressure to use public data or seemingly more quantitative metrics if those are not good measures of your intended outcomes. The best indicators are those which most closely approximate measures of the change you want to see in the world. If the best way to get that data is asking people how they feel about a particular issue, then by all means, hit the streets.

  • Jennifer Banks-Doll

    David, thanks for sharing an example, helped me to see if our thought paths were aligning or not.  Looks like they are. ”Safe neighbourhood” is a term used a lot here in Canada as well, great example!  Another common one is “strong” as in the vision of an NP I worked with this year: “Strong Individuals, Strong Families, Strong Communities.”  In the spirit of sharing, I will share a bit about the work I did to help them to renew their strategic plan, and ultimately to help them to know whether they are making progress towards achieving their vision.  It was definitely not easy, and I’m still far from satisfied that the “work” is done, although the contract has been completed. This NP places a high value on consultation and inclusion, so we did an extensive community consultation in preparation for strategic planning.  We opened with a discussion that asked the following: “What is a strong individual?  What is a strong family?  What is a strong community?”  Fascinating stuff!  But figuring out what to do with it is going to be a long-term project, for whom, I’m not sure.  We then discussed what they were doing well, and what needed improvement.  This was more tangible and the themes were directly applicable to the strategic plan. Back to measurement.  The Plan has 5 strategic outcomes, only 1 of which is related to programs and services, and directly connected back to the vision.  That outcome is: “Individuals, Families and Communities are strengthened by programs and services that are of consistently high quality and that reflect the needs of the communities we serve.”  The “strengthening” of individuals, families and communities is defined and measured by using pre/post surveys of clients and staff that track changes in awareness and behaviour by clients, and then rolling the data up into agency-wide averages.  Because it is difficult (likely impossible based on this agency’s capacity) to measure attributable change at a family or community level, the assumption is made that positive change in individuals will translate into positive change at the family and community level.   In summary, I think we took a “pie-in-the-sky”-like vision (which I think is appropriate) and translated it into a measurable outcome, kind of.  I think they can realistically measure the strengthening of individuals.  But I feel that making an assumption that the strengthening of individuals will automatically lead to the strengthening of families and communities is not good enough to be able to convincingly say that you are making progress towards the vision as a whole.  But that’s where we’ve left it, for now.  Hmmmm…