The first step in becoming better consumers of visual media in the social sector is to create a framework for sorting out types of stories. Was this story or this video created for fundraising or social marketing purposes? Is it designed to mobilize a constituency to action? Is it part of a body of systematic qualitative research? At the end of last month, we hosted a conference, inVision 2010, using these three topical strands to organize presenters’ sessions. Within the “visual media as research and evaluation” track, here are some of the key questions that social sector practitioners should keep in mind when presented with visual stories as evidence of change:

• The Anecdote Test: Is this story representative of outcomes frequently and reliably attained by the intervention, or is it an outlier? For example, if it’s the story of one individual whose life is better because of a program (a very common visual storytelling arc), what information, if any, is also available about other people in the program? How sustainable is the change depicted? Is this one positive moment in a longer story of ups and downs? What is the context surrounding this story of change?

• The Special Sauce Test: Does the visual story simply document one or a few examples of positive outcomes, and/or does it also delve into how that change was created? Documenting the existence of positive outcomes at a meaningful scale is usually accomplished more efficiently through counting – either through surveys or interviews – than visual storytelling. However, uncovering the best practices or special sauce of an initiative or program model – or the reasons why it did not create outcomes – is very effectively done by capturing stories directly from participants and having them explain the change in their own words.

• The Theory of Change Test: How was the storyboard for this piece of visual media created? Visual stories don’t come out of nowhere; they are carefully produced. Who authored the storyboard? What decision-making process was employed to select this story for telling? What other stories were available, but not selected, and why? Was the storyboarding process guided by a theory of change for the overall program or initiative? Does this story support or challenge that theory of change? Does it explore the theory of change fully enough? What other stories also need to be told?

There is both art and science involved in good storytelling in the social sector. I have no doubt about the ability of talented media makers to create compelling audiovisual art, nor about our capacity as human beings to be moved by powerful stories. We should welcome and celebrate those capacities in the social sector – emotion and empathy are valid drivers of our work. But I urge us all to get better at our “story science” as we journey toward social impact. Metrics alone are not enough.

(Photo by Carbon Arc)

We tell stories because historically – ancestrally – this is how we learned. Stories were passed from generation to generation and tribe to tribe to teach about how to stay safe, how to find food and shelter, and how to care for one another. Stories emanated from and were received in a blanket of trust. As a result, our brains are wired for narrative.

Pictures – whether we take them with a camera, draw them, or carve them on walls – are actually our oldest system of checks and balances, reflecting the value we place on truth. As much as we’ve relied on storytelling and storytellers, they did always seem to be the ones hanging out around the most peyote, so perhaps some of our other ancestors sought a little independent confirmation of events. We value pictures because we hold that seeing is believing.

But is it? Fast forward to today – the world of Flip cameras, picture mail, digital storytelling, You Tube, and user-generated everything. We have the capability to see more than ever before, and visual media is everywhere around us. Do we still believe it? I would argue that we do… And we don’t.

The power of a compelling image is undeniable. Regardless of our left brain attempts to rationalize or make sense of what we’re seeing, we usually feel the effects of visual media first. And most scholars of human behavior will tell us that our decisions and actions are very much determined by how we feel. Images move us, whether we “believe” them or not.

But we are also skeptical of emotional manipulation, and these days, technological manipulation. And we are INUNDATED with images, in a way that we have never quite been before. We have become skeptics about the value and veracity of images because we have to have a way to sort them. Doubt is a defense mechanism in an ultra-connected world.

In the social sector, we sit squarely on the horns of this dilemma – belief and doubt – when presented with a visual story about a social change effort.

Don’t Throw The Baby Out With The Bathwater

For a long time, much mainstream philanthropy in this country was – many would say still is – motivated by emotional appeals, often including an image of someone less fortunate than the target audience. Donors and nonprofit leaders alike did their work in part for the satisfaction of changing the story implied in that sad image.

Indeed, much individual giving in this country, and even that of organized philanthropy, still follows this basic pattern: Image. Story. Emotion. Action. This causal chain is seared into our DNA. It is a force to be reckoned with.

But the social sector is growing more sophisticated, analytical, and strategic everyday. As much as we live in a world of images, we also live in a world of metrics and measurement, and that is a good thing. We need to know more than we have historically known about the results of our social investments. The stakes are too high, the opportunity cost too precious to tolerate waste or inefficiency. We don’t believe stories alone anymore. We seek more data, more evidence that change is needed, or that change is happening.

But synaptic pathways die hard. In Hope Consulting’s just-released Money for Good study, over 60% of high net worth families – potential impact investors – reported that they do not use any data to guide their charitable giving. Image – story – emotion – is still the most common pathway to philanthropic “action.” Instead of fighting human nature, how can the metrics movement better leverage it to promote strategic philanthropy? If we train ourselves away from acting on our emotions, do we miss critical opportunities to act? Like all great investors, we need to be as good at using our intuition as we are at reading the numbers. The ability to utilize well-told and credible stories is one of our most powerful capacities, and we need to train up. In my next post, I’ll offer a framework for sorting and understanding visual media in the social sector.

(Photo by !borghetti)

At the same time, micro-engagements do provide opportunities for everyone to get involved[...] These micro-philanthropy/service opportunities are also ways for younger people to begin or expand their participation in what will hopefully become a larger and lifetime commitment to service and giving.

David agreed with the potential for engagement via small scale opportunities and clarified that his criticism is directed towards micro interventions that are intended to create direct impact (and presumably fall short).

In reflecting further on the potential benefits and shortcomings of microphilanthropy, I started to think about inputs and outputs and what underlying assumptions might lead people to support or question the value of microphilanthropy. For the purposes of this discussion, I will use the term “microphilanthropy” to include micro-donations (e.g., Kiva or Tipping Bucket), micro-volunteering (e.g., The Extraordinaries), and any other form of micro-engagement (e.g., petitions on Change.org).

At the level of the individual participant, microphilanthropy has two primary outputs. The first is the Action Impact, the actual tangible effect on the target problem, whether in terms of dollars raised or the value of the action towards positive outcomes for the person or people being served. This is the outcome David focuses on. The second output is Change in Philanthropic Demand. This is a measurement of the increase or decrease in a person’s expected future Action Impact, be it via micro- or macro-philanthropy. Presumably, if a previously un-engaged person has a positive experience with microphilanthropy, he or she will have greater demand for other opportunities to engage in philanthropy of all sorts in the future. Likewise, a negative experience with microphilanthropy may make it less likely for that person to increase future philanthropic engagement, decreasing their philanthropic demand. In a worst case scenario, an ineffective microphilanthropy experience might fulfill a previously engaged person’s philanthropic demand while producing less action impact than alternative philanthropic activities, or even diminish their desire to engage in future acts of philanthropy.

Combining the Action Impact and Change in Philanthropic Demand for an individual engaging in microphilanthropy via a given platform, then multiplying that by the number of people reached by that platform (Reach) gives us an admittedly rough estimation of that platform’s outputs.

(Action Impact + Change in Philanthropic Demand) * Reach = Output

In order to generate these outputs, inputs are needed in the form of financial and human resources to build, promote, improve, and maintain platforms for microphilanthropy. I see these costs as largely Fixed Costs to build out scalable platforms, but there are undoubtedly also Variable Costs that will increase in line with the platform’s aforementioned Reach. Finally, don’t forget to multiply the Fixed Costs and Variable Costs by the Opportunity Cost of not having those human and financial resources to direct to other (presumably effective) forms of philanthropy.

(Fixed Costs + Variable Costs * Reach) * Opportunity Cost = Input

Subtracting microphilanthropy’s Input from its Output gives us its Net Impact:

(Action Impact + Change in Philanthropic Demand) * Reach – (Fixed Costs + Variable Costs * Reach) * Opportunity Cost = Net Impact

I fully recognize the limits of attempts to quantify impact in this manner. Attaching actual numbers to any of these variables, especially the outputs, is difficult or impossible. The formula also does not capture the effect on the Philanthropic Demand of people who may not directly participate in microphilanthropy, but whose opinions of the sector are swayed by the reputation (either positive or negative) garnered by microphilanthropy in society at large. Still, I find it a useful exercise to think about the theory of change behind microphilanthropy interventions and examine the assumptions on which they operate.

What do you think? Is thinking about microphilanthropy in terms of a cost-benefit analysis helpful? Is this formula a useful tool for thinking about the risks and opportunities associated with microphilanthropy platforms or does it fail to capture important aspects of these interventions?

(Photo by vaXzine)

To me, the answer to what makes an organization high achieving is clear.  The social service sector exists to reduce social ills like poverty, homelessness, and food insecurity.  Organizations that have a greater impact on improving the lives of their clients are better than those that have less.  Any evaluative framework that is not centered on measuring changes in client indicators is irrelevant. Despite this obvious point, I am dismayed by how celebrated efforts like the Alliance for Social Investing and Greatnonprofits.org fail to base their evaluative criterion on client outcomes.

There is a lot at stake in getting a rating system right (or wrong).  The potential harm a poor rating system can cause was illustrated last week in a partnership between Greatnonprofits.org and Guidestar.org.  These two rating organizations teamed up to compile a list of the “Top Ten Relief Organizations Working In Haiti.”  The list was compiled based on a handful of donor reviews, and as non-profit consultant Gayle Gifford pointed out, those organizations “that were listed in the Top 10, had ONLY 1 or 2 Reviews. That’s it.”

Greatnonprofits.org and Guidestar.org responded to Ms. Gifford’s criticism by dropping the top ten list all together.  While these rating organizations certainly did the right thing by retracting their list, it is amazing to me that two supposed evaluation leaders in our industry could have compiled such a hasty, pointless agency ranking in the first place. There is so much that is problematic here, least of all the paltry number of reviews the top ten list was based on.

If we are to ever develop a meaningful top ten list of the most effective social programs, we have to embrace the social scientific complexities of evaluating clients’ social outcomes.  This means taking the collection and analysis of client data, in its quantitative and qualitative forms, seriously. Simplistic rating systems that ask donors how they feel about a particular organization may seem seductive, but they could not be more beside the point in determining which organizations are best able to improve the lives of hurting people.  So long as we fail to move towards an evaluative framework that is centered on sound social outcomes practices, the only top ten list we can reliably compile is the “Top Ten Worst Ways to Rank Non-Profits.”

Originally posted on inforumusa.org

(Photo by surfspirit)

About a year ago I wrote about an organization, the Alliance for Effective Social Investing, which aims

To drive more funds to high performing nonprofit organizations by helping donors adopt sound social investing practices.

They plan to do this by creating an evaluation standard by which organizations can be compared to one another.  I recently wrote a post for Inforum where I provided an update on what the Alliance has accomplished (nothing).  Last night I had the displeasure of reading through the group’s most recent paper, Social Investment Risk Assessment Protocol, 11th Version.  The document provides a questionaire and framework for non-profit evaluators.  The idea is that if all evalutors use this assesment tool, then we will have common metrics.  There are two problems with this approach.

1. Inherently not scalable – it is a fantasy to think every organization can get independently evaluated in any meaningful way, with any regularity.  If we can’t do this to every organization, or even a reasonable fraction, there will be no common metrics because the number of evaluated organizations won’t be significant.
2. Subjectivity – the evaluation methodology proposed by the group is based on the subjectivity of the evaluator, rating organizations on a scale of one through five on issues like whether or not an organization holds staff accountable through performance reviews.

On Wall Street, companies are not invested in based on whether or not they have performance reviews.  Companies have performance reviews because it keeps productivity and innovation up.  Higher productivity and innovation means greater profits.  However, presence of performance reviews, in and of itself, is not meaningful.  For companies, they are evaluated in large part on their profits.

So what is the common currency by-which social service and non-profit organizations should be evaluated?  That is the central question, and the one that the Alliance completely fails to address.  The real point should be to evaluate what gets done, not how we do it.  The inadequacy of the Alliance’s approach is on their focus on the how.  In evaluation speak, we refer to this as focusing on outputs, what we do, rather than outcomes, what results we get for the people we serve.

A better common metric are client outcomes such as changes in poverty status, housing status, food insecurity, educational outcomes, etc.  It’s funny how evaluations are incredibly trendy to discuss right now, yet nothing is really being done to move the sector any closer to meaningful evaluation metrics.  So far, this is largely the case in both the domestic and international spaces.  While the Alliance, to date, is a non-factor in seriously providing evaluation frameworks, I’m interested now to see what the Acumen Fund does to move this issue forward with their highly anticipated Pulse evaluation system.