Introduction to Evaluation January 26, 2005

Slide 2 Innovation Network, Inc. Who We Are: Innovation Network National nonprofit organization Committed to evaluation as a tool for empowerment Build evaluation capacity of nonprofits and funders so they can better serve their communities Practice a participatory approach

Slide 3 Innovation Network, Inc. Objectives of This Session Identify how evaluation can be useful in your work Understand both implementation and outcome evaluation Understand how evaluation planning feeds into data collection

Slide 4 Innovation Network, Inc. What is Evaluation? The systematic collection of information about a program that enables stakeholders to better understand the program, improve program effectiveness, and/or make decisions about future programming.

Slide 5 Innovation Network, Inc. What’s in it for you? Understand and improve your program Test your theory of change/program theory Tell your program’s story Be accountable Inform the field Support fundraising efforts

Slide 6 Innovation Network, Inc. Evaluation Principles Evaluation is most effective when it: Is connected to program planning and delivery Involves the participation of stakeholders Supports an organization’s capacity to learn and reflect Respects the community served by the program Enables the collection of the most information with the least effort

Slide 7 Innovation Network, Inc. Implementation and Outcomes Evaluating Implementation/Process : What did you do? How well did you do it? Evaluating Outcomes : What difference did you make through your work? Or, what changes occurred because of your work?

Slide 8 Innovation Network, Inc. Evaluating Outcomes Outcomes : the changes you expect to see as a result of your work Indicators : the specific, measurable characteristics or changes that represent achievement of an outcome. They answer the question: How will I know it?

Slide 9 Innovation Network, Inc. Evaluating Outcomes: Common Types of Change New knowledge Increased skills Changed attitudes, opinions or values Changed motivation or aspirations Modified behavior Changed decisions Changed policies Changed conditions

Slide 10 Innovation Network, Inc. Evaluating Outcomes: Sample Plan OutcomesIndicatorsData Collection Method Data Collection Effort (have, low, med, high) Participants improve job- seeking skills #/% of participants who meet criteria in mock interview Observation of mock interview at end of training session using checklist Low #/% of participants who develop a quality resume Job counselors review resumes based on quality checklist Have Participants improve money management skills #/% of participants who balance their checkbooks Review participants’ check registers & work sheets Low #/% of participants who pay credit card bills in full & on time Review participants’ credit card statements Low

Slide 11 Innovation Network, Inc. Evaluating Implementation Activities and Outputs: The “what” —the work you did, and the tangible products of that work Additional Questions: The “why”— understanding how well you did, and why

Slide 12 Innovation Network, Inc. Evaluating Implementation: What Did You Do? Examine Activities and Outputs Did you conduct activities as planned? Did those activities produce the outputs you envisioned? How to measure? Program documents

Slide 13 Innovation Network, Inc. Evaluating Implementation: How well did you do it? What information will help you understand your program implementation? Think about:: Participation Quality Satisfaction Context How to measure? Program documents, surveys, interviews, comment functions, focus groups, other methods

Slide 14 Innovation Network, Inc. Evaluating Implementation: Sample Plan ActivitiesOutputs & Implementation Questions Data Collection Method Data Collection Effort (have, low, med, high) TRAINING Develop/revise curriculum for training series Meet with potential program clients Coordinate logistics Provide training series to two groups of clients Outputs Curriculum (developed/revised) 2 training series held Completion by 30 of 33 Program records Program records/attendance logs Have Questions Are we reaching the clients we expected? Review of participant intake data Low

Slide 15 Innovation Network, Inc. Data Collection: 3 Steps Choose the method Decide which people or records will be the source of the information Determine the level of effort involved in using that method with that population

Slide 16 Innovation Network, Inc. Data Collection Steps 1 & 2: Choose Method, Identify Source Review documents Observe Talk to people Collect written responses Pictorial/multimedia

Slide 17 Innovation Network, Inc. Data Collection Step 3: Level of Effort Instrument development Cost/practicality of actually collecting data Cost of analyzing and presenting data Also consider: Communication Power Proxy Power

Slide 18 Innovation Network, Inc. Good Data Collection Characteristics: Culturally appropriate (survey, focus group, who asks, manner of asking) Ethical Respectful of participants

Slide 19 Innovation Network, Inc. Continuous Learning Cycle Program Plan Analysis, Reflection & Improvement Evaluation Planning Data Collection

Slide 20 Innovation Network, Inc. Next Steps We can do more! Data Collection and Analysis Jan 9 am Online, instructor-led trainings Online, self-paced modules Individual technical assistance

Slide 21 Innovation Network, Inc. Thanks for Your Participation! Measure results. Make informed decisions. Create lasting change. Innovation Network, Inc K St. NW, 11 th Floor Washington, DC (202) Ehren Reed: