Social Impact Bonds will fail without solving the evaluation problem first
David Henderson     11 comments     April 4, 2011

Social Impact Bonds are the darling of the minute in the social sector. Another day, another push to pretend the social sector is a damsel in distress, only to be saved by the brawn and brilliance of public-sector innovations.

The problems Social Impact Bonds aim to address are real. There are limited financial resources available for social interventions, and we do need to figure out ways to maximize impact per dollar.

Social Impact Bonds attempt to do this by tying returns on investment to outcomes indicators. When an investor purchases a bond in a particular social program, she receives her principal plus an additional monetary return if the agency she invests in yields measurable results. If the agency fails to meet its outcomes threshold, the investor loses her money.

These bonds would be backed by the government, which would theoretically only be obligated to pay for successful social interventions. The intended benefits are

  • The government only pays for what works
  • New money enters the system as social impact bond investors seek profits
  • Non-profits are incentivized to achieve impact

However, it is not clear how different social impact bonds are from the status quo. Funders already require organizations to demonstrate their outcomes. It is unlikely investors are going to be able to do that any better than existing foundation and government grant review committees. Indeed, the real problem of course is ambiguous and non-uniform evaluation criterion, plus the general lack of capacity to monitor impact.

Furthermore, the argument that Social Impact Bonds will bring new money into the sector is an empirical point yet to be seen rather than a logical conclusion. Social investors already get financial benefits in the form of tax write-offs at zero risk. I’m not saying the risk/reward of Social Impact Bonds won’t work, but let’s not be so sure they will, either.

Ultimately, ideas like Social Impact Bonds and non-profit market exchanges that mirror the world of finance dance around, rather than address, the core issues facing the social sector. Investors are a necessary part of a functioning for-profit or non-profit sector, but innovation and execution is king.

We are so enamored of the giving and investing aspects of philanthropy, yet our eyes glaze when the charts and logic models come out. Designing investing models and website that connect us with our favorite causes are so exceptionally beside the point when we lack the cohesions and capacity to measure impact.

It’s no wonder those who focus on the giving side of philanthropy gloss over the implementation aspects. Actually helping people is hard, but giving sure feels great. So too, does making bold claims about how we should only give to effective organizations.

But what does effective mean, and how do we measure it? Those are the questions that actually matter, and without real answers, funding models are not going to make a lick of difference.

(Photo by expta.com)

  • Pingback: 10 Great Social Innovation Reads: March | Social Velocity

  • Al Huntoon

    “Innovation and execution is king,” I couldn” agree more (except maybe I’d say king and queen). If social impact bonds do realize their potential, their use will probably be limited to those areas where outcomes are easy to measure and related to financial benefits. That might be good for those challenges but what about the rest of the nonprofit sector? Can all the hype and over-promising end well?

    • http://www.fullcontactphilanthropy.com David Henderson

      Al, thanks for the correction on the gendered language. Seems we are spinning our wheels looking for easy answers to complicated problems. I don’t think we can get anywhere until we embrace and wade through the complexity.

  • http://twitter.com/lauratomasko Laura Tomasko

    You bring up important points to consider when thinking about the potential promise of social impact bonds. My general sense is that SIBs present an interest model for public-private collaboration, but certainly aren’t appropriate in all cases. While I enjoyed this post, I’m going to push back on some of your statements.

    Looking at the proposed federal pay for success program (in which investors carry the financial risk and reward, as opposed to the proposed Minnesota program in which the nonprofits do), I would challenge your third bullet point that one intended benefit is that “non-profits are incentivized to achieve impact.” For me, the program is less about incentivizing the nonprofits, and more about incentivizing the investors. Yes, many funders do require that organizations demonstrate outcomes, but rarely do funders have to share these demonstrated outcomes beyond the board room. Even further, it’s rare that when funders have to demonstrate outcomes to an outside (let’s say, government) entity that those results are linked to funding outcomes for that project.

    You say that this program doesn’t challenge the status quo, but by having to demonstrate outcomes to a public sector partner, funders have a financial incentive to ensure that 1) the logic model is sound, 2) the proper indicators were selected, 3) the best data collection tools were crafted, and 4) the desired data were collected. Again, this isn’t to say that many funders don’t already emphasize the importance of the evaluation process. My point is that this financing model adds an extra layer of accountability to the process and results, and that extra layer pushes the scope of traditional grant reporting (both public and private).

    I‘m a fan of your recent decision to spend less time arguing why evaluation is important, and more time focusing on how to evaluate. You write, “the real problem of course is ambiguous and non-uniform evaluation criterion, plus the general lack of capacity to monitor impact” and, “what does effective mean, and how do we measure it?” In my mind, SIBs would force nonprofits, investors, and government to struggle with the evaluation implementation challenges that you raise. Yes, at its core, this is a financing model, but it’s a financing model that carries with it a mandate for sound and responsible evaluation at a higher stake than before. Think of all the case studies that can come from this program! Something for future Heinz and Maxwell students to look forward to…

    Lastly, I don’t think the focus needs to be on the “brawn and brilliance of public-sector innovations.” However the social sector is defined, everyone has a stake in its well-being. Bonds are one of the many tools government has at its disposal. Rather than government swooping in and saving the day, government has an opportunity to use one of its tools to drive capital, encourage partnerships, and link outcomes to funds in a way that forces us all to wrestle with these tough and important evaluation implementation challenges you raise.

    Laura Tomasko

    • http://www.fullcontactphilanthropy.com David Henderson

      Laura, thanks for the thoughtful response. You make a persuasive argument that the SIB creates an extra layer of accountability, however I’m still concerned about what type of incentive this layer creates for implementing organizations. Perhaps you are right that the SIB is not about incentivizing non-profits to achieve impact, but there is no denying that if the underlying evaluation criterion, and who the evaluators are, changes, then so too will their behavior.

      You list a number of reasons why “having to demonstrate outcomes to a public sector partner” can lead to better outcomes. While I agree with the points you list out, we also need to consider the perverse incentives that arise as well. Like anything involving statistical interpretation, outcomes evaluation is open to a lot of numeric gymnastics. So just as there might be an emphasis on ensuring logic models are sound, so too will there be efforts to game the evaluation rules, perhaps churning client outcomes, avoiding harder to work with population, etc.

      Ultimately, my core point is not so much about SIBs being great or awful, and more so that an effective SIB is dependent on good evaluation practices, something we flatly lack in the sector.

  • http://www.tacticalphilanthropy.com/ Sean Stannard-Stockton

    The point that evaluation is key to functioning SIBs is completely spot on. But you seem to miss the important point that SIBs are not about getting social investors to put up money, it is about getting private sector profit seeking investors to put up money. That’s why the innovation is important. I do think their use will be limited to areas where there is a direct government cost savings linked to nonprofit outcomes that can be measured accurately. But in areas like education and certain social welfare programs, the evaluation challenge isn’t unsurmountable. The government doesn’t need to know how much a program improves life outcomes or other broad impact measures, they just need to know if they will need to spend less on specific government funded programs.

    The UK SIB that is happening now is a good example. The outcome measurement is pretty simple: Does the nonprofit program result in a lower level of ex-prisoners from a specific prison returning to prison compared to similar facilities where the nonprofit programs are not operating. That’s pretty straightforward.

    • http://www.fullcontactphilanthropy.com David Henderson

      Sean, thanks for the clarification on the targeted investor, although the recent discussion about whether cause marketing trades off with direct giving comes to mind here. Assuming that study bears out it seems a logical extrapolation to SIB’s. However, that’s an empirical point and I try not to debate that which is measurable. Hopefully we’ll have a study on donor/investing trade-offs at some point in the future.

      To the point about the low-hanging fruit, the UK prison example you mention does look like a pretty good experiment, and I’ll admit I had not read the particulars about it before you mentioned it here. So long as SIBs really are targeted at well contained, cleanly measurable situations like this one, I might just change my tune.

      That said, prisons are typically used for experiments because they are controlled environments with good post-service tracking. It’s hard to think of a lot of situations like that, and I’m certainly skeptical that education and social welfare offers a lot of such opportunities. Just look at No Child Left Behind as an example of how poorly crafted outcomes metrics tied to money can lead to misaligned incentives for implementing organizations.

  • RDavies

    One of the real benefits of an SIB of the sort being piloted here in the UK is precisely that it removes some of the problems with outcome evaluation- particularly the thorny attribution issue. Since the intermediary is giving grants to a range of organisations with the aim of producing given agreed outcomes, all that is important is that the actions of those organisations taken together deliver the required results. Both the investors and the intermediary can be agnostic to an extent about exactly which organisation contributed what to a given outcome. Of course, the intermediary will still want some sort of evalutaion framework in place so that the ongoing progress of grantees towards delivering the agreed outcomes can be gauged.

    The main issue I have with SIBs is how sustainable they are as a model, and where they are supposed to get us to. If the point is to be able to attract in mainstream, socially-motivated finance from institutional investors like pension funds then I struggle to see how this is going to work. In order for them to be a viable investment option, they will have to develop a sufficiently strong track record to fit the risk profile of institutional investors (which is pretty low). However, if they have this sort of track record, it will surely be based on having identified interventions that reliably deliver certain outcomes? In which case, after the track record has been established, why would the public sector funder not simply invest in these interventions directly? Basically, by the time you get to the point where a given SIB is an attractive investment opportunity for mainstream investors, the underlying interventions should be sufficiently well-established to be low enough risk for a public sector body to fund them directly and take advantage of the savings without incurring the additional transaction costs associated with structuring a bond.

    If this is true, then the role of SIBs is to allow the public sector to invest in the development of potentially cost-saving interventions without incurring the initial risks. Once the successful interventions in a given area are identified, is there any need for a bond any more? Or would you look to invest in different interventions in that area in order to expand the range of options?

    • George Overholser

      I agree that SIBs simplify matters by simply asking “Did the improvement happen?”, not “Which specific program was the cause of improvement?”.

      There are two additional benefits as well.

      First: Many of the SIB arrangements will be able to piggy-back on administrative data that is now already captured as a by-product of running government. For example, there are now 30 prisons in the UK that use similar methods to track recidivism. This means it is possible, without fielding an expensive custom survey, to answer the question: “Has recidivism in the Peterburough prison broken away from the pack?”

      In my recent travels, I have been impressed by the growth of large databases that share common methodologies across geographies, and can be linked together much more easily than in the past. Progress is also being made in the area of privacy, which is crucial. (See the National Student Clearinghouse, and STRIVE Cincinnati)

      Second: (Sorry to be technical.) Because impact is measured at the “whole prison” level, we sidestep the expensive (and often ethically troubling) need to randomize program participants as a way to prove impact. Thus, the goal is to improve recidivism for the whole prison, not just the individuals who happened to participate in the programs. In the evaluation business this is called “intent to treat” framing — interventions are evaluated as though the programs intend to treat 100% of the population, not just the individuals who may be easiest to treat.

      I also agree with your second point, but am perhaps less worried by it. As I see it, our goal is to influence how government spending is allocated, not to replace it with private spending. To me, it would be wonderful if government spending were to crowd out investment opportunities from the private side. The key, though, would be to maintain enough of an incentive-tied presence to keep the impact metrics in force.

      • RDavies

        It is encouraging to hear you say that the goal of SIBs is to influence how government spending is allocated rather than to replace it with private spending. To my mind that is precisely the role in which they could have a real impact on some deeply entrenched social issues. In particular, by taking the immediate burden of risk funding away from public sector funders, the move from ongoing services which address effects toward targeted interventions which address causes could be made much easier.
        My concern is less with the actual principle of SIBs themselves, than with the way in which their potential seems to be interpreted by some people (particularly those in political circles who have come across the idea). The suggestion from these people (either overt or implicit) is that replacing public funding is precisely what SIBs are intended to do, which is worrying. Perhaps this is just a function of the fact that they haven’t looked at the detail of what these structures actually are and how they work, and once there is a bit more clarity there will be fewer people misinterpreting the role of SIBs? I certainly hope so!

        • George Overholser

          Simple math: when government pays back investors, plus a risk premium, there’s no net reduction in payments for the interventions. There IS (we hope) a net reduction in other areas of government expense, such as closing prisons, but that isn’t a case of philanthropists replacing government outlays.

          Sobering math: If the ENTIRE Gates Foundation corpus were focused exclusively on the US K-12 education system, it would be enough to keep the schools going for just 8 days.

          Encouraging math: There are hundreds of billions spent each year by government on programs that, when evaluated, fail to show discernible impact… and yet there do exist other UNDERFUNDED programs that provide strong evidence of powerful social impact.

          So we don’t have a shortage of money in government social investment. Rather, we have a grand misallocation of money.

          But, then again…. I’m an optimist!