Full Contact Philanthropy

Evaluating your organization’s use of metrics

David Henderson January 23, 2012

Evaluating organizational effectiveness is a growing sub-field of the social sector, with a slew of competing measurement frameworks. Something a lot of these frameworks assess is whether organizations make use of data management system. The idea is that an organization that has a data management system in place is more likely to be data savvy and to actively manage to outcomes.

This might be a reasonable proxy for whether an organization actually incorporates evidence in its practice. But from where I stand, data only has value in so far as it helps an organization make higher impact decisions. Therefore, I propose a more robust approach to evaluating an organizations use of metrics.

If an organization’s behavior before implementing a data oriented approach is exactly the same after implementation, then no value has been added. Data should help inform action, not just confirm prior beliefs. It’s hard to imagine any organization (or individual) that does everything so perfectly that there is no room for improvement.

Effective uses of data collection and evaluative analysis should help drive program improvements. If you can’t identify any changes in your organization’s behavior, then whether or not the organization has a data management system and processes in place to collect information, it has not actually benefited from its data efforts.

More important, just because an organization changes its behavior based on metrics does not necessarily mean it benefited from an effective use of information. Indeed, information should help us make better decisions. In some cases, organizations make poor decisions that are backed by data.

A classic example is the Space Shuttle Columbia disaster. NASA used information to back up its decision to go ahead with a shuttle launch in suboptimal weather, that lead to the disintegration of the shuttle’s O-ring and a subsequent explosion that killed all the astronauts on board.

In this case, NASA made a data-backed decision, but it used its data incorrectly and made the wrong inference, resulting in disastrous consequences. Therefore, it’s not only important to collect your data and use it, but to take care to analyze your data properly, and listen to reason of all parties.

Which leads me to my most important indicator of whether an organization uses information well. Information should inform decision making, but it should not necessarily, on its own, dictate what an organization does. While having a data management system in place is great, and using reasoned analysis in the interpretation of data is better, there is no replacing the judgment of experienced practitioners and the feedback of those we serve.

The best organizations include evaluate metrics as a part of their decision frameworks, but they do not supplant their own judgment for a regression.

Data helps answer questions, it does not determine what questions should be answered

David Henderson January 18, 2012

As the furor to incorporate metrics in the social sector grows, organizations are feeling the heat to get more data savvy. In principle, this is a good thing. Information should help inform decision making. But there is a big difference between information informing your agenda and allowing it to set it.

Data should inform your answers to questions, but data sets should never determine what questions you seek to answer. Every organization grapples with a myriad of decision problems, from optimizing resource allocations to increasing the social impact of interventions.

The natural role of information is to help us make more informed decisions. But data does not, on its own, answer any questions. And no data set can (or should) determine the most important issues facing an organization. Those questions should be driven by the organization itself and the people it serves.

Yet time and again I see organizations blankly asserting that they need data. Why?

A lot of organizations don’t have a great answer beyond citing the overall direction of our industry. This is a pretty lousy answer, and more importantly leads to half-baked data collection implementations that do nothing to drive organizational change or improve outcomes.

Each organization I work with, before talking about data management systems, what data points to collect, or internal processes for collecting metrics, I ask them what they do and what problems they face. Simply put, you don’t know what data you need until you know what problems you’re trying to address.

I’m afraid the glorification of trivial info-graphics and blanket mandate that organizations should be “data driven” perpetuates a wrong-headed belief that there is inherent value in data. As someone whose whole lively-hood is based in data collection and analysis, let me be as clear as I can, data only has value when it informs a decision.

As a sector, we’d be wise to focus less on the “data, data, data” mantra, and to instead engage in discussions about the issues organizations face, and where metrics can help inform better decision making. Despite the misleading glee of those who proclaim the data revolution will transform the social sector, data itself is nothing but a distraction unless it answers specific questions an organization faces.

Measuring the social impact of blogging

David Henderson December 30, 2011

Professionally I do two things; I help organizations make high impact data-oriented decisions, and I write. As 2011 draws to a close, I reflect on another year helping a lot of great organizations increase their social impact, and a pile of blog posts that I hope help advance the social sector toward lasting change.

Obviously I believe writing, and the exchange of ideas that comes with it, is important to the growth of our sector and advancement of solutions. If I didn’t believe that, I wouldn’t write anything. But as someone who prefers evidence to anecdotes, facts to feelings, I’m at a loss for much evidence that blogging (at least my blogging) helps move the needle even a little bit.

I feel like I get quite a bit more than I give in terms of writing. And maybe that is okay, so long as I believe writing helps me get better at what I do, and that what I do with my agency has social value.

But my ambition for writing and the promise of free-flows of information in the social sector exceed simply personal gratification and advancement. My hope is that by sharing with one another what works and what doesn’t, that we would improve our own services, turning those little insights into collective action.

While articles about changes in the poverty rate and misleading homeless counts are compelling reads for people like us, if that information exchange doesn’t improve the output of our efforts, then what are we doing? There are certainly times when I worry that the articles we write and share with one another have no value other than to amuse ourselves, like a gossip rag for poverty-geeks.

I hope I am wrong, and in 2012 I plan to actively seek evidence to the contrary. I want to believe that we are evolving into a sector that thrives on sharing best practices and possesses the sophistication to integrate information across the fields of politics, sociology, finance, social work, community development, and a slew of other focus areas that collectively sum to the vastness of the social sector.

Indeed, it is in the vastness of the social sector that I worry the value of our information exchange is lost. As you know, the social sector is complex. Its complexity in part stems from the fact that it is not so much a sector, but rather a sector of sectors (some call it the un-sector). The sector-of-sectors nature I fear lends itself to sharing information in parallel, rather than exchanging information that directly impacts what we do.

I, like you, picked this line of work to improve the lives of hurting people. When we read posts, share information, and write on our blogs, we are diverting our time away from our work. I hope, I think, this is a good use of our time. I think writing matters, and I hope in 2012 I can prove it to myself.

Why predictions are so difficult to make in the social sector

David Henderson December 28, 2011

As 2011 comes to an end and we look forward to 2012, pieces predicting what will happen in the coming year are popular in every industry, including the social sector. While making predictions is no easy task, not even for psychics, making predictions in the social sector is especially difficult.

Most industries tend to have clear industry leaders. These organizations, like Apple, Chevron, and Wallmart are significantly large that they can in many ways move industries on account of their sheer size. In the social sector we have no clear market makers, with the exception of governments. But governments are inherently unpredictable. Democracies are subject to deadlock, compromise, and regular leadership change. Dictatorships, while marked by leadership consistency, lend themselves to rulers with unpredictable and often erratic behavior and management decisions.

Of course, a common metric used for industry predictions is consumer demand. While a company like Amazon can use its vast consumer behavior database and market research to predict the directional winds of what consumers want, addressing needs is a significantly different undertaking. The demand we address tends to be either immediate (homelessness, hunger) or arguably intractable (systematic poverty).

Aggregate social indicators like poverty and unemployment rates are the most popular indicators used for predicting future demand. Yet the reliability of these predictors is questionable considering the regularity of reports about how organizations are blindsided by demand for their services. Indeed, these social predictors are highly imperfect, especially considering our inability to agree on a definition of poverty in the United States and the persistently misleading reports of homeless counts.

While big-data is being put to lucrative use in some sectors, especially technology, the most accurate predictions are based on the aggregation of front-line organizations’ own data. Big-data is itself by definition aggregations of micro-consumer behavior and transactions. The difficulty we face in applying these same tactics in the social sector however is that in order to aggregate meaningful large-scale data-sets, we must first collect and analyze meaningful micro-sets.

To this end, my company is working with organizations helping them setup data collection pipelines and analytic processes so they can better make predictions of both demand and impact. Making increasingly more accurate organizational level predictions is a necessary first step to reliable industry wide predictions.

Despite all the blog posts, books, and questionable models to the contrary, we are a long ways away from being able to predict with any relevance the future of the social sector. Therefore, the only prediction I will make for 2012 is that the future is unclear.

Data is not information

David Henderson 3 comments December 21, 2011

For all the buzz about how data is supposed to change the social sector, there is scant evidence that revolution is truly underway. Certainly there are high-profit efforts to catalogue and aggregate data as social sector organizations are savvy to the importance of documenting their work and outcomes in databases. But moving data from our heads to paper to the cloud does not necessarily create value, much less social change.

A recent article in GOOD asks “is solving nonprofits’ challenges as easy as creating maps?”. One of my company’s areas of focus is mapping social services in communities, helping service seekers connect to programs and helping organizations better understand their geographies. Therefore, as someone who knows a thing or two about mapping in the non-profit sector, I thought I would take a crack at answering this question.

No.

Databases and maps store data. But data is not information. Data becomes information when used to inform decision making. Numbers, regressions, and longitudes and latitudes do not impute meaning. How we use, contextualize, and interpret data determines informational value, not the size of a database nor the contours on a map.

Despite this rather obvious point, our efforts and interests are stuck on this most primitive definition of data. Government and non-profit organizations spend considerable money on database systems to store and collect data without clear strategies for how they will use it. In this way, the social sector suffers the dual cost of both the cash outlay spent on aimless data collection, and the more insidious invisible opportunity cost of what might have been had these organizations successfully converted their data to information.

Indeed, the CTO of Infochimps, a company at the forefront of the big data revolution, smartly argues that storing data is not only a significant pain, but of no real value. Instead what matters is insight.

Data on its own provides no insight, not even big data. Before an organization thinks about collecting data, I first ask what decisions they are trying to inform. The questions an organization is trying to answer drives sound data collection efforts, as these questions provide the necessary context to inform future decision making.

The social sector would do well to focus less on buzz words like open-data, big data, mapping, info-graphics, and any other form of trivial data distractions that masks the real problems our sector exists to address every day.

The data revolution can, and should, be a boon for our sector and those we serve. Databases are cheap and public datasets are copious. Free tools like Google Earth and R afford anyone with the necessary patience to develop real analytic competence the opportunity to do so, and better the lives of those in need. And while there are efforts like Ushahidi that underscore the power of marrying technical know-how with social problems, too much of our public discourse is dominated by proponents of the most trivial data artifacts.

Hurting people could care less about slick info-graphics or mapping their descent into homelessness in “real-time”. Our job is to solve social problems. If our data doesn’t help us help people better, it’s not information, it’s just noise.

Making decisions when targeting multiple outcomes

Michael Gechter 3 comments October 31, 2011

Organizations in the social sector often have broad missions that lead them to target a variety of different outcomes. While targeting multiple outcomes frequently means that our missions accurately reflect the complexity of the social problems we seek to address, doing so can create practical problems for organizational decisionmaking.

Say an organization’s mission is to foster the development of healthy communities in the city where it is based. To this organization, a healthy community is one where people feel safe, have local job opportunities, and can access transportation.

Having split the notion of the “healthy community” into safety, employment, and transportation, our organization has created a decision problem for itself. Should it put all of its programming efforts into safety? All of its efforts into job creation? Or how about splitting its efforts across safety, employment, and transportation?

Since the organization’s mission is to foster healthy communities and it believes that a healthy community is safe, has jobs, and has good access to transportation, it might seem obvious that its programming should target all three areas. But now the organization has three objectives, not just one. Given the option of funding a safety program, an employment program, or a transportation program, how should our organization allocate its scarce resources?

One option is to say “fund the program with the highest predicted impact.” But how do you compare predicted impacts that are expressed in the incompatible terms of safety measurements, employment creation metrics, and proxies for transportation access?

The answer is to make explicit the importance that your agency places on each outcome. Take our organization working on creating healthy communities: their leadership should decide how much weight to put on measures of safety, employment availability, and transportation access when assessing the health of a community. One simple way to come up with weights for each outcome is to say: community health is 40% safety, 50% employment access, and 10% transportation availability. Or 35% safety, 35% employment, and 30% transportation.

Whatever the numbers you come up with, there are big advantages to being explicit. Staff always have to choose a prioritization in order to make any programming decision. Without an explicit prioritization that is shared across the agency, staff members may come up with inconsistent ways of ranking outcomes. A frequent assumption is that all outcomes are equally important (33% safety, 33% employment, and 33% transportation in our example), but that prioritization may be inconsistent with the mission of your organization. Worse still, staff may feel uncomfortable unilaterally coming up with any prioritization scheme and so your agency may become paralyzed by indecision.

Going through the simple exercise of deciding on agencywide priorities for each of the outcomes you target can go a long way towards freeing you from internal gridlock and making sure your decisions reflect your mission.

Possibilities and probabilities of Social Impact Bonds

David Henderson October 24, 2011

Last Friday the White House Office of Social Innovation and Civic Partnership hosted a one day seminar on the pay-for success (also known as social impact bond) approach to social sector financing. I’ve been critical of social impact bonds (SIB) in the past, so was interested to watch the White House’s live stream of the event.

The promise of social impact bonds is to provide a new way of financing some social interventions. Panelists at the White House summit were quick to acknowledge that SIBs are not a cure all approach, and that not all interventions are well suited for this type of funding.

My criticism of SIBs in the past has been that the investment approach assumes evaluations can be performed reliably. Obviously, I think evaluation is a pretty thorny issue that we have a long ways to go on. I was happy to see that proponents of SIBs at this event were aware of this problem, and while more optimistic about which programs are easily measured than I am, realized there were limitations.

The example that has been most used in peddling SIBs, and the only actual implantation of the concept I’m aware of, involves an anti-recidivism program in a UK prison. Prisons provide a nice opportunity for evaluators because you can easily collect longitudinal data (prisoners and parolees are already monitored) and you have a logical control group (neighboring prisons).

Other social interventions don’t provide such fertile ground for experimentation. Yet ascertaining the effectiveness of a program is at the heart of the agreement between an investor and the government, which backs the bond.

Indeed, as was discussed at the seminar, the critical negotiation in a SIB is when the investor and government establish the ground-rules for when the government pays out, and when it doesn’t. The investor has an incentive for weaker evidence of success while the government prefers stronger.

While the SIB discussion centers on thresholds like the number of positive outcomes, the discussion got me thinking about another metric of success, the probability that an intervention actually caused a desired outcome.

For example, an employment program might place a hundred people in jobs in a month, but would those people have found jobs anyway? This is the question a control group helps inform. But the control group does not answer this question definitively. Instead the best evidence one can get is that:

A program exceeds its outcomes threshold
There is a certain probability that the program deserves credit for this success

Therefore, the question is not just about achieving an outcome objective, but how sure we are that the relationship between the intervention and outcome is not due to random chance. Achieving a high significance threshold further limits the possibilities of SIBs, as in order to ensure fairness to both the bond holders and backers, ample data in a controlled environment is a necessity.

The White House seminar, as well as discussions I’ve had since my last post on this topic, softened my stance on social impact bonds. As Tracy Palandjian, CEO of Social Finance smartly said, social impact bonds are about managing “execution risk”. Where there are opportunities for well controlled experiments, like prisons, SIBs might very well help governments manage that risk.

What SIBs won’t do is drive innovation, nor will they replace all existing forms of social funding. But they don’t need to in order to be a success. If SIBs can help governments improve the performance and financing of a subset of their social investment portfolios, that’d be a significant accomplishment anyone would have a hard time questioning.

Focus on big data misses the big picture

David Henderson 1 comment October 20, 2011

We are living in an era of data deluge. Social sector talking heads are preaching the promise of “big data”, yet few arguments have been made about how all this data will facilitate impact.

Data is useful only in so far as it helps inform decision making. And while social sector organizations aim to solve big problems, their solutions are decidedly smaller in scale and scope.

Indeed, an organization developing a job training program might look at macroeconomic conditions to determine the total potential market size for an intervention. But unless that intervention covers a large enough geography to significantly influence over all employment (which is rarely the case) the more useful information is the micro, “small data” that an agency collects itself.

I think about so-called big data, like Census indicators, as providing the context for an intervention. It’s the market data that develops the case for action. But once that decision to act is in place, there’s little left for big contextual data to inform.

The more pressing questions, like is this program working and how do I increase impact, cannot be answered by large public databases. Instead the focus needs to be on developing analytical capacity and program specific data collection feedback loops that capture relevant indicators on an iterative basis.

Yet our current obsession with big data and trivial infographics obscures the real promise of an analytically oriented social sector for soundbites and graph porn.

If we want to tackle the big problems, we need organizations to be able to collect and analyze small data sets relevant to their own work. Our wrongheaded focus on large scale data for the sake of seeming analytical obfuscates the real opportunities data affords.

The reasonable consequences of our unreasonable expectations

David Henderson October 10, 2011

Organizations are often tasked with making bold predictions about future achievement. But as the 10 Year Plan to End Homelessness illustrates, most of the predictions we make in the social sector are based on hope and little else.

Good predictions are based on sound models and historical data. And before you think the concept of modeling is too technical or outside the realm of the social sector, I would argue that every organization has a model. It’s called a theory of change.

Publicly traded companies make revenue predictions that investors analyze to determine whether to buy, hold, or dump stock. Social sector agencies are similarly asked to make predictions about what changes they are going to create in the world and how they are going to do it.

Companies have a lot to lose when they fail to meet projected targets. Analysts rip companies apart and investors lose confidence not only in companies’ predictive abilities, but in management and future viability as well.

But in the social sector we are rarely evaluated based on our predictions. As a result we shoot for the moon, hit the dirt, and call it a success. And while all this unreasonable ambition sounds good, and makes for catchy campaign titles, does it help us advance public welfare?

Our outlandishly high expectations and horribly unfounded predictions undermine donors’ faith in our ability to create real social change. Indeed, it’s seems an unspoken truth of the social sector is that if your theory of change doesn’t draw a straight line between your local intervention and world peace, then you’re just not doing it right.

What’s wrong with setting reasonable objectives? Why not have a theory of change that relates what you do to what you can actually accomplish; instead of relating what you do to some fantastical notion of what you wish might happen?

Instead of spinning increasingly more ludicrous stories, we’d be better served aiming lower and actually hitting our targets. That way we could accurately account for what we do, measure our impact, and iterate on our interventions.

The byproduct of reasoned predictions based on realistic theories of change would be something far better than stories that make donor’s hearts swell. We’d have stories people could actually believe.

Subscribe

Recent Posts

Search