Once it is time to perform an evaluation, a group or individual independent of program staff and management will plan an impact study to determine to what extent a social intervention helps the people it serves. The study runs for a planned duration, at which point the evaluators collect and analyze the results, producing figures that quantify the program’s social impact.

The procedure is just like that of an Internal Revenue Service audit. External agents come in and verify that a program does what it says it does for its clients.

Audits are useful. The threat of IRS audits dissuades some people from tax evasion.

In the social sector, outcomes audits can keep agencies and groups of agencies honest about the impact that their programs are having. A group of MIT economists investigated the impact of a particular microfinance organization and proved false the wild claims that microfinance as a strategy would by itself lead to the “eradication of poverty and hunger, universal primary education, the promotion of gender equality and empowerment of women, reduction in child mortality and improvement in maternal health.”

Here the independence of the auditor helps. It is politically difficult for an agency to make such bold claims about the effectiveness of its programs, then turn around and declare their claims to be false.

But audits fail program staff who are looking to implement interventions that are effective in helping people. Audits are time-consuming: the MIT economists’ study of microfinance began in 2005 and continued data collection until 2008. The first version of the results didn’t appear until 2009.

You can make the case that it takes time for a program’s long-term effects to take hold, but part of the problem is that the audit procedure is generally independent of the organization’s own data collection efforts. The auditors have to start from scratch, which takes more time.

Audits are also self-contained. Once data collection concluded on the microfinance study in 2008, that was it. The agency under investigation requested a follow-up study, but this is the exception, not the norm.

The closed nature of audit studies is a particular problem when findings are inconclusive, as they often are. Further work may help to disentangle conflicting results, but when further audits by program outsiders are the only way to learn more, financial expense or researcher disinterest discourages follow-up.

Long time horizons and the fact that audits are self-contained limit their usefulness to program staff trying to make day-to-day decisions about who to serve, what to do, and what not to do.

What’s missing in the audit framework is an integration of evaluation and service provision. Working on developing data collection software for anti-poverty interventions at Idealistics, I am in the unique position of having access to program data as it is collected. Using program data, we are working on ways to automatically provide program managers and staff with the kind of sophisticated empirical feedback that is generally reserved for external evaluators.

This requires program data to be of high quality, so we assist the agencies we work with in designing their data collection mechanisms with day-to-day evaluation in mind.

My belief is that by bringing evaluation inside of program administration, we will be able to empower program staff to respond to what works and what doesn’t in real time, rather than leaving them helpless, waiting for the next external audit.

(Photo by chadmiller)

Outputs: These are the activities done by the nonprofit. The meals served by a soup kitchen are outputs.

Outcomes: These are the observed effects of the outputs on the beneficiaries of the nonprofit. The degree to which the meals served by the soup kitchen reduce hunger in the population served by the soup kitchen.

Impact: This is the degree to which the outcomes observed by a nonprofit are attributable to its activities. The impact of the soup kitchen is the degree to which a reduction of hunger in the population they serve is attributable to its efforts. While a soup kitchen might serve a lot of meals and correctly observe that hunger is subsequently less prevalent in the population it serves, the reduction in hunger might simply be attributable to an improving economy, or a new school lunch program or some other activities that are not part of the soup kitchen’s efforts.

Sean’s post made me realize that when I write about outcomes on this blog I am actually referring to impact. By conflating outcomes with impact I hope that I have not been misleading in the past, and I appreciate Sean’s clarification on the terminology. In his piece, Sean went on to argue that outputs are inferior to outcomes as units of measurement, with impact being the gold standard objective. I certainly agree with this final point that impact, the change created as a result of a particular intervention, is the sole, truly meaningful metric.

However, I was more iffy on the assertion that outcomes, as defined by Sean, are clearly superior to outputs. In the comment section on the Tactical Philanthropy blog, I wrote:

Outputs and impact are more pure metrics in that they both isolate an effect of the intervention. The output obviously tells us very little (such as in your example a client receiving food aid) and the impact telling us a lot (the degree to which hunger decreased, nutrition increased, etc., as a result of that aid). While it might seem that outcomes are superior to outputs, there is serious risk that outcomes are misleading, as you get to a bit.

Most damaging, the outcomes metric can mask harms from an output. You allude to a scenario where there are multiple factors at play toward a positive end of an intervention, but an intervention can actually have a negative effect, which might be offset by other environmental factors, not only masking the harm of the intervention but wrapping it in a cloak of success.

Sean responded to my comment by saying:

David,

I would argue that Outcomes are objectively more important (better) than Outputs, but admit freely that they are harder to measure. I guess like any powerful tool, they can become dangerous if used incorrectly (miscalculated).

In hindsight, I believe I overstated my point when I suggested that outcomes are not more telling than outputs. Sean was right to hold his ground. However, I am still troubled by a hierarchical mental framework that holds outcomes over outputs, when outcomes are inherently riddled with extraneous factors outside the control of a particular intervention.

An output does not say anything about a change in a service recipient’s life, therefore making it an indicator of little use for assessing social value of an intervention. However, an output is a pure metric in that the output is clearly the result of the organization administering the intervention. In this way, although an output does not say anything about what happened in an individual’s life, it accurately indicates what a particular agency did.

This may seem like a non-point, but let’s look at the clarity of the output metric versus an outcomes metric. An outcome is a ratio in which the numerator is impacted by any number of variables which are not controlled for. For example, we can look at the change in hunger levels amongst a population that accesses a food pantry. One might find that incidences of hunger have gone down amongst that population over a period of time. In this sense, there is a positive outcome indicator (lower incidences of hunger) which might correlate with a higher level of food distribution by the food pantry.

The problem, of course, is that the lower incidences of hunger could be the result of any number of factors, like lower food prices, better wages, etc. Worse yet, what if the food pantry’s activities actually had a negative impact on food insecurity, or produced some other harms? Outcomes is a metric that risks both masking the negative effect of an intervention and crediting an intervention with factors outside the control of said intervention. This effect works the other way too, a population might do worse over time in a particular indicator due to forces outside the control of that intervention, even though the intervention might have a hidden, positive causal effect. In this way, a focus on outcomes might create incentives for service providers to focus on populations that will do better in certain indicators despite their interventions, thus achieving an outcomes windfall.

My point here is not to continue a debate about whether outcomes or outputs are better metrics. I agree with Sean and the others who blasted my comment, a focus on outcomes is more meaningful than outputs. I think my objection is more rooted in the fear that one might believe that if one has outcomes data, than output data is not relevant. In the absence of impact, both metrics are important. Outputs tell us what a service provider did, outcomes tell us what happened to service recipients over a period of time.

Of course, this debate would be rendered moot if we were better able to assess impact. As many have pointed out though the social sector struggles to reliably collect output or outcomes data, let alone make any serious attempt at impact analysis. While I concede that outcomes are a more important metric than outputs, I believe the path forward is through fostering an appreciation and deep understanding of what specific indicators can, and cannot, tell us about the work we do. Ultimately, we need to collect a number of indicators. More importantly, we need to know what these indicators are actually telling us, and the various ways in which our indicators, especially composite indicators that are impacted by a large number of variables, actually mean.

(Photo by Darren Hester)

We tell stories because historically – ancestrally – this is how we learned. Stories were passed from generation to generation and tribe to tribe to teach about how to stay safe, how to find food and shelter, and how to care for one another. Stories emanated from and were received in a blanket of trust. As a result, our brains are wired for narrative.

Pictures – whether we take them with a camera, draw them, or carve them on walls – are actually our oldest system of checks and balances, reflecting the value we place on truth. As much as we’ve relied on storytelling and storytellers, they did always seem to be the ones hanging out around the most peyote, so perhaps some of our other ancestors sought a little independent confirmation of events. We value pictures because we hold that seeing is believing.

But is it? Fast forward to today – the world of Flip cameras, picture mail, digital storytelling, You Tube, and user-generated everything. We have the capability to see more than ever before, and visual media is everywhere around us. Do we still believe it? I would argue that we do… And we don’t.

The power of a compelling image is undeniable. Regardless of our left brain attempts to rationalize or make sense of what we’re seeing, we usually feel the effects of visual media first. And most scholars of human behavior will tell us that our decisions and actions are very much determined by how we feel. Images move us, whether we “believe” them or not.

But we are also skeptical of emotional manipulation, and these days, technological manipulation. And we are INUNDATED with images, in a way that we have never quite been before. We have become skeptics about the value and veracity of images because we have to have a way to sort them. Doubt is a defense mechanism in an ultra-connected world.

In the social sector, we sit squarely on the horns of this dilemma – belief and doubt – when presented with a visual story about a social change effort.

Don’t Throw The Baby Out With The Bathwater

For a long time, much mainstream philanthropy in this country was – many would say still is – motivated by emotional appeals, often including an image of someone less fortunate than the target audience. Donors and nonprofit leaders alike did their work in part for the satisfaction of changing the story implied in that sad image.

Indeed, much individual giving in this country, and even that of organized philanthropy, still follows this basic pattern: Image. Story. Emotion. Action. This causal chain is seared into our DNA. It is a force to be reckoned with.

But the social sector is growing more sophisticated, analytical, and strategic everyday. As much as we live in a world of images, we also live in a world of metrics and measurement, and that is a good thing. We need to know more than we have historically known about the results of our social investments. The stakes are too high, the opportunity cost too precious to tolerate waste or inefficiency. We don’t believe stories alone anymore. We seek more data, more evidence that change is needed, or that change is happening.

But synaptic pathways die hard. In Hope Consulting’s just-released Money for Good study, over 60% of high net worth families – potential impact investors – reported that they do not use any data to guide their charitable giving. Image – story – emotion – is still the most common pathway to philanthropic “action.” Instead of fighting human nature, how can the metrics movement better leverage it to promote strategic philanthropy? If we train ourselves away from acting on our emotions, do we miss critical opportunities to act? Like all great investors, we need to be as good at using our intuition as we are at reading the numbers. The ability to utilize well-told and credible stories is one of our most powerful capacities, and we need to train up. In my next post, I’ll offer a framework for sorting and understanding visual media in the social sector.

(Photo by !borghetti)

Editor’s note:

The current movement to make philanthropy more accountable for lasting social change is critical and must take deep root in our sector. Calls for an impact-oriented social capital marketplace have switched on a bright light in a dark room, urging us to look at all sides of our philanthropic activity – far beyond the honeymoon stage of the initial grant or investment all the way to the “so what” stage, when we may learn what difference any of our actions have made. Of course, philanthropists have used evaluation to examine “so what” questions for a long time, but a new vanguard – consisting of both next-gen social entrepreneurs and social sector veterans – are reconsidering and reframing them.

Our sector is buzzing with efforts to define and implement metrics systems that will support strategic philanthropy and impact investing. Getting better at using metrics is a vital evolution of our social change efforts. And yet, I worry when the conversation about metrics leaves out an essential piece of the puzzle – qualitative methods.

“What?!” you say. “Qualitative methods? I thought this was a serious post about measurement and accountability!” Indeed it is, and I’m grateful that David, who deals more in numbers himself, has invited me to contribute my thoughts on the use of qualitative methodologies. Please read on.

Social Change is Complex

For over 15 years I’ve designed and implemented mixed-method studies of increasingly complex social sector research questions. It used to be that funders wanted to know whether or not a program was producing positive outcomes in participants. Many still want to know this, but in addition, they want to understand more systemic issues. Here are a few examples of questions my team is currently examining:

• How does the collective capacity of many community-based organizations form a nationwide social justice infrastructure?
• To address the achievement gap between white children and children of color, do we need to look for solutions not just inside, but alongside the public education system?
• How can international development programs be sustained by local leaders once aid dollars have stopped flowing?

The field’s growing awareness of interdependencies has led to more meaningful inquiries into the impact of philanthropy and social investing. And the more complex the research questions, the more of a role there is for high quality, systematic qualitative research in addition to measurement of those things we can quantify.

Anecdotes are Not Qualitative Research

High quality, systematic qualitative research is not the same thing as collecting a few anecdotes and photos to put in the sidebar of a report. It is also not the same thing as fielding a written survey asking respondents about attitude or behavior change. I recently listened to a webinar that billed itself as being about “qualitative methods,” and the central discussion was about quantitative techniques for analyzing survey data (for example, running a statistical regression on data collected using 5-point Likert scales). Both definitions of qualitative research would make any self-respecting ethnographer cringe. In my book, both of these approaches – anecdote or self-report survey – reflect a very limited understanding of the range and power of systematic qualitative research.

In future posts, I’ll delve into qualitative methods increasingly used in the field for both data collection and reporting, including visual storytelling, social network analysis, discourse analysis, and visual arrays of massive data sets. I look forward to generating a dialogue with all of you about the productive uses of qualitative methods to advance our commonly held goal of making the world a better place.

(Photo by luz)

I wonder however if the same logic doesn’t extend to the social sector. I work with non-profits big and small, all who claim they offer critical services. They make these claims to solicit donations, but in the aftermath of Too Big to Fail, the idea that certain social sector institutions are absolutely essential might be more unsettling than moving.

In the social sector we defend agencies’ rights to create monopolies, veiled in an avoidance of duplication of services. Social sector agencies argue that there are limited funds to produce social value with. Therefore, in order to maximize social output we should avoid situations where one agency’s service offering overlaps with another. The result is regional monopolies whereby a few organizations are granted full control over a range of social services, thus manufacturing an environment where agencies can claim to be critical backbones of the social sector.

Their claims of being too big, too critical to society, to fail, are legitimated by the dogmatic adherence to avoiding duplication of services. All the praise heaped on these organizations amazingly echoes the same complaints we now have about unwieldy financial institutions. Anti-competitive social sector collusion is the backdrop for the social sector version of Too Big to Fail.

The idea of social sector organizations being too big to fail is even more disturbing considering we don’t have much reason to believe they are effective, even though their entire hegemonic reign is predicated on the assumption of maximum social impact.

Organizations that successfully convince the public they provide essential services might very well be producing more marketing miracles than Hallmark moments. In fact, I have serious doubts about most organizations’ capacities to measure changes in client indicators period, let alone asses the extent to which those changes are the result of program activities.

I recently read a report about program measurement released by the Gates Foundation titled A Guide to Actionable Measurement (non-profit consultant Gayle Gifford has a nice write up about the report on her blog). While the report provides a comprehensive evaluation strategy, I was struck by the simple advice the report offers to not always focus on measuring social impact by relating program outputs to client outcomes.

I often drink the social outcomes measurement Koolaid, but this Gates report got me thinking that  simply understanding whether clients who receive a particular service are better off or not is a logical evaluative starting point. However, the extent of our evaluation woes is so deep that even such a simple, summary data type assessment of client indicators is more elusive than it ought to be, a problem that is compounded by a social sector culture that favors non-profit monopolies.

Through monopolization a handful of organizations, rightly or wrongly, are allowed to claim credit for a community’s status quo. The anti-competitive nature of the social sector creates an atmosphere whereby every organization can claim that reduced funding or closure of their programs will lead to social disarray.

I view the problems of anti-competitiveness and poor evaluation in the social sector to be linked. Anti-competitiveness, and adherence to avoiding duplication of services, supersedes the need for serious evaluation. Evaluation is not only a tool for evaluating client progress, it is a lens through which comparison between interventions, and agencies, can be drawn.

So long as the social sector continues to espouse anti-competitiveness cloaked in the flag of collaboration, we will continue to have a sector that is not necessarily too big to fail, but is certainly too small-minded to succeed.

(Photo by jeffisageek)

The social sector has gotten stuck. We have confused energy and vision with meaningless mission statements and empty slogans featuring words like “eradication”, “sustainability”, and “collaboration”. We promise our donors and investors that we have found the way to end pollution, disease, hunger, and social injustice. We paint these grandiose pictures akin to that of the millennium development goals (MDGs), and then we wonder why our donors are beginning to ask us where their money is going.

People are still poor. The slave trade is still thriving. Children are still hungry. Carbon dioxide emissions remain excessive. All realities that leave us with a question of, “Have we actually done anything at all?”

Yes, we have. We have developed ready-to-use-therapeutic-food to address issues of malnutrition. We have provided millions of dollars in loans to foster entrepreneurship in the developing world. We have used SMS technology to connect people to healthcare and food. We have even set up voluntary carbon markets to hone and incentivize environment-friendly processes. Those of us vested in the work of social development have been innovative and have implemented solid programs with tangible deliverables that have saved lives, developed community leadership, and created safe refuges for countless species.

We are doing good work. But we are growing increasingly lazy in the way that we tell our stories. In fact, I’ll take it one step further – we are lying. People are asking for measurements of success and many of us do not have them – at least, not the ones we promised. The upcoming generation, the famous Millennials, demands to see impact if they are going to donate, and more of the recent social impact investors are expecting organizations to prove they are actually achieving their goals.

It really is not going to matter whether we preach a sermon on eradicating poverty. It is going to matter that through our work we increased a community’s income by 20%. It is going to matter that we perfected a revenue-generating model that allows for the maintenance of the wells that we fund-raise for.  It is going to matter that through our work, 27 women were rescued and protected from the sex trade in the last month due to the donated amount. The marriage of what we do, how we do it, and why we do it, needs to be our focus and message.

And for those of us who are donating, we have some work to do as well.

A professor of mine points out that business plans must “be realistic”. When he reviews business plans that promise the entire world is their market, out comes the red pen and the editing begins. A wise investor knows the age-old saying, “if it seems too good to be true, it probably is.” She won’t invest in a business that promises the world.

This is why we have to seek out organizations that can demonstrate how they are solving important social issues. Investors that do not require evaluations are doing the community a disservice.  If we hand organizations a check, we are saying that we believe in the work they do.  You wouldn’t buy into a product that promised you abs of steel without ever having to leave your couch (at least, I hope you wouldn’t). Why would you believe an organization that promises to eradicate poverty by 2015?

Using buzz words instead of using creative and honest ways to convey the work we are doing is disrespectful to the people and communities we serve. If you do in fact choose to promise to eradicate poverty, I challenge you to do this in front of an audience of children in the slums of Kenya.  If you choose to promise to stop the international sex trade, do it while looking into the eyes of the young rape victims in your own city.  And if you choose to promise a future for all children, I dare you to do so while sitting with a mother who knows that the AIDS that claimed her husband’s life will soon claim hers. They will all ask how. And they deserve a clearly defined answer.

(Photo by Vanguard1219)

Failing the success of sweeping interventions, the sector has recently become obsessed with micro solutions to social problems. The wave of micro activity started with the popularity of microcredit, but has recently devolved into a flurry of any philanthropic word pre-fixed with “micro” such as micro-volunteering, micro-donations, micro-philanthropy, and micro-actions.

My macro point here is that the momentary micro dogma of the social sector distracts us from pursuing real solutions that help people.  What matters, of course, is what works, small, medium, large, or super-sized.  The micro-trend was started by microcredit, the first, and only member of the “micro” solution set that resembles a real intervention rather than a gimmick focused more on size than effectiveness.

Microcredit is micro in so far as it is a small loan to an impoverished person, ostensibly used for wealth creating activities. While microcredit has been heralded in some circles as a powerful poverty-reduction tool, recent evaluative research has raised some important questions. Specifically, David Roodman speculates there might be a microcredit bubble in Bangladesh. Roodman writes

Indeed, multiple borrowing is widespread in Bangladesh now, and it has raised concerns that some Bangladeshis are juggling microcredit loans the way some Americans juggle credit card debt, in a merry-go-round that must one day stop.

As borrowers acquire multiple loans their debts becoming increasing less micro, raising doubts about microcredit’s core promise that the poor need only small loans to lift themselves out of poverty, a falsehood promoted more by Kiva’s marketing than the fundamental tenants of the microcredit movement.

While the idea that a modest investment by Western standards can create sustainable businesses is appealing, evidence of multiple borrowing undermines this hope. Even though most people involved in the day-to-day development of microcredit as a poverty intervention strategy take a sophisticated, nuanced approach to microcredit, its widespread popularity has less to do with actual outcomes and more to do with the suggestion that solving big problems only requires small actions.

And here, I believe, is where the micro thinking begins to unravel.

The current adherence to the micro dogma does not come from a measured understanding of effectiveness of micro approaches. Instead, our fascination with all things micro stems from a hope that simple, small, and intuitive sounding actions can solve tremendously complicated problems. By attempting to reduce the daunting magnitude of poverty to something we can solve through trivial investments, shopping, and meaningless minute-at-a-time volunteer activities, we simply aggregate our micro inabilities to solve social problems into a macro inability to solve social problems.

Over on the Tactical Philanthropy Blog comments section reader Chip McComb sums up the problem with micro giving nicely, in so doing revealing much of what is wrong with micro thinking in general. Chip writes

I fear that as micro giving, and mobile giving becomes more and more prevalent the attitude of those that give, could shift dangerously to think that all giving should be as easy and as pleasing as buying a coke or a big mac, and when it’s not easy or pleasing, it is therefore not worth their time or expense. What a dangerous trap!

I am not arguing that all micro efforts are problematic. There are some great virtues of thinking small, so long as micro means local approaches to social problems, small strategic investments (like microcredit), or other such reasoned uses that resemble actual strategies. My problem with the current wave of micro thinking is that micro has become a euphemism for easy.

Ultimately, what matters is providing solutions that work. In some cases, small interventions might work best, like microcredit, in other cases, perhaps our investments need to be large and patient, rather than micro, like Acumen Fund’s approach to social investing. Whatever the size of the intervention, all our approaches should be well reasoned and rigorously evaluated.

Of course, unlike our modern micro interventions, evaluation is hard, even if it is indispensable in expanding what works and purging what does not.  Recognizing both the importance and complexity of evaluation, perhaps I should pursue the idea of micro-evaluation, a simple evaluative framework that is as easy to use as it is meaningless.

(Photo by Articulate Matter)

that White House officials will travel across the country to find “the most promising nonprofits in America” as the administration decides how to spend a new $50-million fund to help charities expand innovative social projects.

Of course the President’s statement begs the question, what does “most promising” mean? The White House has spoken in vagaries about program effectiveness and scalability, but Betsy Fuch of the Modern Giving blog rightly points out that

With countless private foundations around the country squabbling to define the ideal due-diligence process, how does Washington know exactly how to build one? How will we be certain that those deemed the ‘best’ programs really are the best programs?

The real point here is that we simply do not know how to identify the “best programs” because we don’t know what “best” means. The private sector enjoys a standard quantifiable metric of profit that allows businesses within and across sectors to be compared to one another. Social service evaluation is infinitely more complex.
Harvard business professor Clayton Christensen argues that the Obama administration should use

the convening power of the White House to initiate a focus on impact and metrics. Specifically, the White House should help initiate a process by which categories of social innovations are agreed upon, and metrics can be defined for assessing the impact of innovations in each category on the social problems that they target. Just like independent rating agencies have developed methods for assessing the safety of investments in various securities, methods might emerge that help social investors categorize the type of impact that various social entrepreneurs hope to achieve, and to rate the present and potential effectiveness of their efforts to achieve that impact.

While I agree with Professor Christensen that the White House should use its clout to move the industry toward evaluative metrics, I do not think the government should determine what those metrics are.
As Professor Christensen himself points out, in the private sector we rely on independent rating agencies who compete with one another do develop ever better evaluative metrics. We need a similar approach in social services, whereby multiple well funded, capable entities work on the serious problem of developing competing evaluative frameworks.
Perhaps the Obama administration’s $50 million would be better used as research grants to develop a robust evaluation sub-industry in the social service sector. As it stands, without a clear definition of what “most promising” means, it is impossible to tell which, if any, organizations in the US are worthy of being funded by the Social Innovation Fund.

(Photo by wonderwebby)

Part of the problem is that performing social service evaluations is complex and can be costly. Agencies don’t always have the expertise or funds to perform proper evaluations. The bigger problem, though, is that agencies don’t necessarily have an incentive to conduct evaluations in the first place because of a common and wrongheaded assumption that if a social service agency is doing a nice thing that sounds intuitive, then it must be working. This is a dangerous assumption that is perpetuated in media portrayals of social services.

Take for example a recent article in the LA Times about a pilot program in the LA County jail system to distribute condoms to inmates. The intended purpose of the program was to reduce the spread of HIV amongst inmates. On face, it sounds like a compelling idea. We know that the use of condoms reduces the spread of HIV and other STD’s in the general population.

However, the jail environment is different. There may be a stigma effect in the jail that reduces substantially the number of inmates willing to opt-in to the condom program. Also, since the pilot program only supplied a limited number of condoms, it’s possible that inmates reused the condoms, used them improperly, or even engaged in riskier sexual behavior on account of the presence of condoms.

A prison is a perfect place to do a meaningful evaluation. The jail has medical information on inmates and could easily track which inmates opted-in to the voluntary condom program. Also, since the pilot was not conducted at all LA County jails, the non-participating jails could act as control groups.
While the article mentions a group of health advocates who said

that a successful review of that program could lead to widespread distribution of condoms in prisons throughout the state

the evaluation is at best framed as a necessary formality to confirm what is believed to already be known. Instead of casting a critical eye on the effectiveness of the condom program at reducing HIV transmissions, the author implicitly presumes the effectiveness of the pilot program by failing to challenge at any point the assertion that the condom program reduces HIV transmissions amongst inmates.

My point here is not to say that the pilot was ineffective at reducing the spread of HIV. Rather, my point is that without an evaluation, we simply don’t know. If we want to get real about helping people, we have to know whether we really are helping people.