In a presentation at the Social for Social Innovation at Stanford University (the audio file is available here), Dr. Alana Conner argued for fewer and better evaluations of social service agencies. Her argument hinges on the fact that good evaluations are hard to do. She points out the importance (and difficulty) of finding a control group for an agency’s clients. That is, a group of individuals who are as similar as possible to the clients but who do not receive services from the agency. Comparing the difference in outcomes between the agency’s clients and the control group yields the impact of the agency’s work. Rick Aubry of Rubicon Programs, one of her co-presenters at the lecture, goes on to point out that evaluations require a serious commitment at all levels of the agency, as intakes and follow-ups must be designed to collect data for the evaluation.

Conner’s conclusion, then, is that evaluations should only be done on programs that seem deserving (i.e., programs that clients seem to like or other agencies are interested in copying). While this follows logically from her and her colleagues’ arguments that evaluations are difficult, the thesis seems unsatisfying. Even if a program does not seem deserving of an evaluation, how undeserving is it? Could it be improved with just a few tweaks? Is it an agency’s worst program (should it be cut) or is it somewhere in the middle?

On the other hand, if we do want to do large numbers of evaluations, we need to address her points and make evaluations easier. How can we do this?

Use Statistical Controls

Conner is correct in pointing out that finding and following a control group is a difficult task. The agency must expend resources on tracking people who are not its clients. On an even more fundamental level, where do we find the members of the control group and how can we convince them to accept tracking by an agency that does nothing to help them? Morally, do we really want to explicitly deny services to a group of people who seem to be a perfect fit for the agency’s services. Are there any alternatives?

One alternative lies in intake design. If we’re primarily concerned with extracting an agency’s impact from the impact of other services that clients might be receiving, we can ask clients which types of services they are receiving. With enough data, we can isolate the impact of participating in particular types of other programs from the agency’s own impact.

What this doesn’t do is control for a “rising tide that lifts all boats.” That is, general economic conditions that would positively or negatively impact clients regardless of whether the client gets help from the agency. In this case, again with enough data, statistical controls can be established for the time period in which a client receives services.

Along the same lines is the concern that the impact attributed to other social services would actually be an “interaction effect” - the very logical idea that working with multiple services at the same time actually helps a client more than working with each service individually. If the agency doing the evaluation provides multiple programs, though, it is possible to extract the effects of combining the services of an external agency with each of the evaluating agency’s programs and to separate these effects from the benefits provided by the external agency alone.

Automate the Evaluation Process

Conner’s colleague Aubry is also correct in saying that evaluations are a lot of work. The agency has to design intakes and follow-ups that collect the necessary data and then analyze the data to perform the actual evaluation. With all the work an agency has to do in reporting to its funders, who can afford to do more reporting?

This is where technology gets to interact with social services. Idealistics’s case management software makes data entry a part of the case management process, not another form that needs to be filled out. We design intakes with agencies with an eye towards evaluation and, finally, we lay out reports that the agency can run automatically.

Evaluations Can Be Easy

Most people in social services would agree that evaluating programs is a crucial part of helping people but, with the right intake design and tools, evaluations need not be reserved for only the most promising programs. They can be used as extensively as they should.

Tags: evaluate