1. Strategic Planning Session Part II: Outcome Assessment and Program Evaluation Chapter 16: John Clayton Thomas Presented by : David Rudder, Ph.D.

2. Rules for Outcome Assessment  The organization’s leaders have to be committed to the effort  Outcome assessment should be undertaken on a program-by- program basis  Planning process should entail extensive involvement of staff and perhaps even clients who are connected with the program  The process should be linked to the organization’s information technology; building strong performance assessment systems requires that the system be planned in conjunction with the organization’s technology

3. Defining Program Goals Outcome assessment is a goals-based process in that programs are assessed relative to the goals they are designed to achieve (tend to be abstract, long term)  Outcome goals - are the final intended consequences of a program for its clients or society.  Activity Goals – these specify the actual work of the program (concrete, operational)  Bridging Goals – supposedly connect activities to outcomes  Side Effects - are also consequences of a program for society, but are unintended

4. Defining a Goal (Criteria) 1) Each goal should contain only one idea 2) Each goal should be distinct from every other goal 3) Goals should employ action verbs( e.g. increase, improve, reduce)

5. Types of Measures Outcome assessments can employ several types of measures, and to achieve the benefit of multiple measures will typically use two or more of the types. Examples are: 1) Program Records and Statistics 2) Client Questionnaire Surveys 3) Formal Testing Instruments 4) Trained Observer ratings 5) Qualitative Measures

6. Approaches to Program Evaluation  The objective approach- takes little interest in how a program’s internal mechanics are functioning. It focuses on – is the program sufficiently effective to be continued or should it be terminated?  The Utilization-focused approach – focuses on achieving balance through the use of both qualitative and quantitative data because quantitative data alone are unlikely to capture all that is important about programs whose goals are subjective.  The balanced approach also rejects outcome assessment (did the program work?) as the only purpose of an evaluation.

7. Who does the Evaluation? 1) An internal evaluation performed by the organization’s staff 2) An external evaluation performed by outside consultants 3) An externally directed evaluation with extensive internal staff assistance The best means for conducting an evaluation is through a combination of outside consultants and internal staff. Here outside consultants provide technical expertise plus some independence from internal organizational pressures, while internal staff perform much of the legwork and collaborate with consultants in developing the research design, collecting data, and interpreting findings

8. Summary  Outcome assessment data can reveal whether progress is being made on key agency objectives.  Outcome assessment data alone cannot speak to issues of causality, that is, to whether any observed changes resulted from specific agency program or programs.  Succeeding in these efforts requires a delicate balancing of analytic and scientific expertise with group process skills.