Outputs: These are the activities done by the nonprofit. The meals served by a soup kitchen are outputs.

Outcomes: These are the observed effects of the outputs on the beneficiaries of the nonprofit. The degree to which the meals served by the soup kitchen reduce hunger in the population served by the soup kitchen.

Impact: This is the degree to which the outcomes observed by a nonprofit are attributable to its activities. The impact of the soup kitchen is the degree to which a reduction of hunger in the population they serve is attributable to its efforts. While a soup kitchen might serve a lot of meals and correctly observe that hunger is subsequently less prevalent in the population it serves, the reduction in hunger might simply be attributable to an improving economy, or a new school lunch program or some other activities that are not part of the soup kitchen’s efforts.

Sean’s post made me realize that when I write about outcomes on this blog I am actually referring to impact. By conflating outcomes with impact I hope that I have not been misleading in the past, and I appreciate Sean’s clarification on the terminology. In his piece, Sean went on to argue that outputs are inferior to outcomes as units of measurement, with impact being the gold standard objective. I certainly agree with this final point that impact, the change created as a result of a particular intervention, is the sole, truly meaningful metric.

However, I was more iffy on the assertion that outcomes, as defined by Sean, are clearly superior to outputs. In the comment section on the Tactical Philanthropy blog, I wrote:

Outputs and impact are more pure metrics in that they both isolate an effect of the intervention. The output obviously tells us very little (such as in your example a client receiving food aid) and the impact telling us a lot (the degree to which hunger decreased, nutrition increased, etc., as a result of that aid). While it might seem that outcomes are superior to outputs, there is serious risk that outcomes are misleading, as you get to a bit.

Most damaging, the outcomes metric can mask harms from an output. You allude to a scenario where there are multiple factors at play toward a positive end of an intervention, but an intervention can actually have a negative effect, which might be offset by other environmental factors, not only masking the harm of the intervention but wrapping it in a cloak of success.

Sean responded to my comment by saying:

David,

I would argue that Outcomes are objectively more important (better) than Outputs, but admit freely that they are harder to measure. I guess like any powerful tool, they can become dangerous if used incorrectly (miscalculated).

In hindsight, I believe I overstated my point when I suggested that outcomes are not more telling than outputs. Sean was right to hold his ground. However, I am still troubled by a hierarchical mental framework that holds outcomes over outputs, when outcomes are inherently riddled with extraneous factors outside the control of a particular intervention.

An output does not say anything about a change in a service recipient’s life, therefore making it an indicator of little use for assessing social value of an intervention. However, an output is a pure metric in that the output is clearly the result of the organization administering the intervention. In this way, although an output does not say anything about what happened in an individual’s life, it accurately indicates what a particular agency did.

This may seem like a non-point, but let’s look at the clarity of the output metric versus an outcomes metric. An outcome is a ratio in which the numerator is impacted by any number of variables which are not controlled for. For example, we can look at the change in hunger levels amongst a population that accesses a food pantry. One might find that incidences of hunger have gone down amongst that population over a period of time. In this sense, there is a positive outcome indicator (lower incidences of hunger) which might correlate with a higher level of food distribution by the food pantry.

The problem, of course, is that the lower incidences of hunger could be the result of any number of factors, like lower food prices, better wages, etc. Worse yet, what if the food pantry’s activities actually had a negative impact on food insecurity, or produced some other harms? Outcomes is a metric that risks both masking the negative effect of an intervention and crediting an intervention with factors outside the control of said intervention. This effect works the other way too, a population might do worse over time in a particular indicator due to forces outside the control of that intervention, even though the intervention might have a hidden, positive causal effect. In this way, a focus on outcomes might create incentives for service providers to focus on populations that will do better in certain indicators despite their interventions, thus achieving an outcomes windfall.

My point here is not to continue a debate about whether outcomes or outputs are better metrics. I agree with Sean and the others who blasted my comment, a focus on outcomes is more meaningful than outputs. I think my objection is more rooted in the fear that one might believe that if one has outcomes data, than output data is not relevant. In the absence of impact, both metrics are important. Outputs tell us what a service provider did, outcomes tell us what happened to service recipients over a period of time.

Of course, this debate would be rendered moot if we were better able to assess impact. As many have pointed out though the social sector struggles to reliably collect output or outcomes data, let alone make any serious attempt at impact analysis. While I concede that outcomes are a more important metric than outputs, I believe the path forward is through fostering an appreciation and deep understanding of what specific indicators can, and cannot, tell us about the work we do. Ultimately, we need to collect a number of indicators. More importantly, we need to know what these indicators are actually telling us, and the various ways in which our indicators, especially composite indicators that are impacted by a large number of variables, actually mean.

(Photo by Darren Hester)

Editor’s note:

The current movement to make philanthropy more accountable for lasting social change is critical and must take deep root in our sector. Calls for an impact-oriented social capital marketplace have switched on a bright light in a dark room, urging us to look at all sides of our philanthropic activity – far beyond the honeymoon stage of the initial grant or investment all the way to the “so what” stage, when we may learn what difference any of our actions have made. Of course, philanthropists have used evaluation to examine “so what” questions for a long time, but a new vanguard – consisting of both next-gen social entrepreneurs and social sector veterans – are reconsidering and reframing them.

Our sector is buzzing with efforts to define and implement metrics systems that will support strategic philanthropy and impact investing. Getting better at using metrics is a vital evolution of our social change efforts. And yet, I worry when the conversation about metrics leaves out an essential piece of the puzzle – qualitative methods.

“What?!” you say. “Qualitative methods? I thought this was a serious post about measurement and accountability!” Indeed it is, and I’m grateful that David, who deals more in numbers himself, has invited me to contribute my thoughts on the use of qualitative methodologies. Please read on.

Social Change is Complex

For over 15 years I’ve designed and implemented mixed-method studies of increasingly complex social sector research questions. It used to be that funders wanted to know whether or not a program was producing positive outcomes in participants. Many still want to know this, but in addition, they want to understand more systemic issues. Here are a few examples of questions my team is currently examining:

• How does the collective capacity of many community-based organizations form a nationwide social justice infrastructure?
• To address the achievement gap between white children and children of color, do we need to look for solutions not just inside, but alongside the public education system?
• How can international development programs be sustained by local leaders once aid dollars have stopped flowing?

The field’s growing awareness of interdependencies has led to more meaningful inquiries into the impact of philanthropy and social investing. And the more complex the research questions, the more of a role there is for high quality, systematic qualitative research in addition to measurement of those things we can quantify.

Anecdotes are Not Qualitative Research

High quality, systematic qualitative research is not the same thing as collecting a few anecdotes and photos to put in the sidebar of a report. It is also not the same thing as fielding a written survey asking respondents about attitude or behavior change. I recently listened to a webinar that billed itself as being about “qualitative methods,” and the central discussion was about quantitative techniques for analyzing survey data (for example, running a statistical regression on data collected using 5-point Likert scales). Both definitions of qualitative research would make any self-respecting ethnographer cringe. In my book, both of these approaches – anecdote or self-report survey – reflect a very limited understanding of the range and power of systematic qualitative research.

In future posts, I’ll delve into qualitative methods increasingly used in the field for both data collection and reporting, including visual storytelling, social network analysis, discourse analysis, and visual arrays of massive data sets. I look forward to generating a dialogue with all of you about the productive uses of qualitative methods to advance our commonly held goal of making the world a better place.

(Photo by luz)

The debate over nonprofit executive compensation has little to do with increasing social impact. Rather, such arguments underscore the antiquated view still held by some that charity is about personal sacrifice rather than public benefit. Moreover, the controversy surrounding appropriate levels of executive compensation cannot be satisfactorily resolved until we establish metrics that allow us to meaningfully measure executive output against the only standard that matters, social outcomes.

Until recently, charity has been almost exclusively viewed as a personal sacrafice, a gift to the public realized as an economic loss rather than an investment in the greater good. In this misconcieving of charity, charity workers themselves are also assumed to be in the social sector with an expectation of personal sacrifice, generally viewed as paid volunteers rather than seasoned, well trained professionals.

Charity as sacrifice is a deprecated concept for a reason, it is a model that yields no discernible social impact. In the new era of social investing, and professional social entrepreneurs, working in the social sector is not about self-sacrifice, it is about professionalizing and innovating on the ways we help hurting people.

Debating executive compensation, in and of itself, is a pointless exercise. Compensation in any sector should be commensurate with performance. The problem is that as a sector we lack any real ability to measure social progress. So long as we fail to measure social outcomes, we will lack the sole, bottom-line metric we need to evaluate executive compensation.

Outrage over executive compensation is simply an extension of the same line of illogic used in evaluating charities by their overhead ratios (in fact, executive compensation is an overhead expense). Like overhead ratios, the brouhaha over executive compensation in the social sector underscores our inability to measure social progress, recently illustrated in a post on the Chronicle of Philanthropy by non-profit consultant Rosetta Thurman.

Rosetta, commenting on a recent controversy over the total annual compensation package of the Boys and Girls Club Chief Executive Officer (about $900,000) writes

I absolutely believe that nonprofit CEO’s should earn good pay. But there’s a big difference between good pay and excessive pay. While an organization can use any number of formulas to set compensation, it’s clear that many a reasonable person would deem Ms. Spillett’s salary excessive. I’m certainly not saying that nonprofit CEO’s should take a vow of poverty but that it may be problematic to have leaders in our sector who could be deemed “rich.”

Indeed, if we sent the message to the general public that helping poor people as a career choice was not only morally righteous, but economically rewarding, where would we be? Perhaps we would have attracted the talent we need to establish proper outcomes metrics so we would not feel the need to assess executive compensation packages by the same standard the Supreme Court uses to differentiate art from porn.

Executive compensation should be based on social outcomes. Just as for-profit executives are evaluated on their performance relative to profits, and are compensated accordingly, so too should we in the social sector reward our executives based on their creation of social profit.

Until we establish better social outcomes metrics, and mechanisms for regularly, and rigorously, evaluating organizational output, we cannot have an honest discussion about executive compensation.

(Photo by Robert Couse-Baker)

Those of us who work on poverty alleviation no doubt get into this field out of a commitment to reduce poverty.  As we join organizations we believe in or develop poverty interventions of our own however, our loyalties shift in subtle ways.  Where once the poor themselves were paramount in our ambitions, our ambitions instead become about the success of our solutions.  If our solutions do not reduce poverty, we are perversely incentivized to argue with the evidence before us, rather than admit what we are doing is not working.

In this way, our sector as a whole is not trying to end poverty per se.  Instead, we are trying to be the one’s to end poverty.  If we blind ourselves by ambition and moral outrage, we render ourselves incapable of allowing outcomes metrics and social indicators to guide our poverty interventions.  If we fail to use data, we will fail to end poverty.

(Photo by Piez)

To me, the answer to what makes an organization high achieving is clear.  The social service sector exists to reduce social ills like poverty, homelessness, and food insecurity.  Organizations that have a greater impact on improving the lives of their clients are better than those that have less.  Any evaluative framework that is not centered on measuring changes in client indicators is irrelevant. Despite this obvious point, I am dismayed by how celebrated efforts like the Alliance for Social Investing and Greatnonprofits.org fail to base their evaluative criterion on client outcomes.

There is a lot at stake in getting a rating system right (or wrong).  The potential harm a poor rating system can cause was illustrated last week in a partnership between Greatnonprofits.org and Guidestar.org.  These two rating organizations teamed up to compile a list of the “Top Ten Relief Organizations Working In Haiti.”  The list was compiled based on a handful of donor reviews, and as non-profit consultant Gayle Gifford pointed out, those organizations “that were listed in the Top 10, had ONLY 1 or 2 Reviews. That’s it.”

Greatnonprofits.org and Guidestar.org responded to Ms. Gifford’s criticism by dropping the top ten list all together.  While these rating organizations certainly did the right thing by retracting their list, it is amazing to me that two supposed evaluation leaders in our industry could have compiled such a hasty, pointless agency ranking in the first place. There is so much that is problematic here, least of all the paltry number of reviews the top ten list was based on.

If we are to ever develop a meaningful top ten list of the most effective social programs, we have to embrace the social scientific complexities of evaluating clients’ social outcomes.  This means taking the collection and analysis of client data, in its quantitative and qualitative forms, seriously. Simplistic rating systems that ask donors how they feel about a particular organization may seem seductive, but they could not be more beside the point in determining which organizations are best able to improve the lives of hurting people.  So long as we fail to move towards an evaluative framework that is centered on sound social outcomes practices, the only top ten list we can reliably compile is the “Top Ten Worst Ways to Rank Non-Profits.”

Originally posted on inforumusa.org

(Photo by surfspirit)

About a year ago I wrote about an organization, the Alliance for Effective Social Investing, which aims

To drive more funds to high performing nonprofit organizations by helping donors adopt sound social investing practices.

They plan to do this by creating an evaluation standard by which organizations can be compared to one another.  I recently wrote a post for Inforum where I provided an update on what the Alliance has accomplished (nothing).  Last night I had the displeasure of reading through the group’s most recent paper, Social Investment Risk Assessment Protocol, 11th Version.  The document provides a questionaire and framework for non-profit evaluators.  The idea is that if all evalutors use this assesment tool, then we will have common metrics.  There are two problems with this approach.

1. Inherently not scalable – it is a fantasy to think every organization can get independently evaluated in any meaningful way, with any regularity.  If we can’t do this to every organization, or even a reasonable fraction, there will be no common metrics because the number of evaluated organizations won’t be significant.
2. Subjectivity – the evaluation methodology proposed by the group is based on the subjectivity of the evaluator, rating organizations on a scale of one through five on issues like whether or not an organization holds staff accountable through performance reviews.

On Wall Street, companies are not invested in based on whether or not they have performance reviews.  Companies have performance reviews because it keeps productivity and innovation up.  Higher productivity and innovation means greater profits.  However, presence of performance reviews, in and of itself, is not meaningful.  For companies, they are evaluated in large part on their profits.

So what is the common currency by-which social service and non-profit organizations should be evaluated?  That is the central question, and the one that the Alliance completely fails to address.  The real point should be to evaluate what gets done, not how we do it.  The inadequacy of the Alliance’s approach is on their focus on the how.  In evaluation speak, we refer to this as focusing on outputs, what we do, rather than outcomes, what results we get for the people we serve.

A better common metric are client outcomes such as changes in poverty status, housing status, food insecurity, educational outcomes, etc.  It’s funny how evaluations are incredibly trendy to discuss right now, yet nothing is really being done to move the sector any closer to meaningful evaluation metrics.  So far, this is largely the case in both the domestic and international spaces.  While the Alliance, to date, is a non-factor in seriously providing evaluation frameworks, I’m interested now to see what the Acumen Fund does to move this issue forward with their highly anticipated Pulse evaluation system.