1 Qualitative Evaluation Terms Coding/categorization The process of condensing qualitative data through the identification of common themes. Data Matrix or Display Grouping data within common themes in order to identify patterns in the data. Storytelling Using quotes and detailed description to capture the “story” or essence of a program or event.

2 Analyzing Quantitative Data Can be done by program staff with the help of a spreadsheet (such as Excel). Quantitative data analysis consists of: Complex manipulations of the data designed to understand causal or correlational effects of program interventions. Simple calculations yielding factual information on attendance, usage, changes in performance, and/or changes in knowledge or attitudes (pre/post test). Best done by an evaluation contractor or trained researcher. Or

3 4. Analyze your Data Quantitative Data Analysis–Analysis of numbers. Best presented in the form of pictures – graphs and charts. Qualitative Data Analysis– Analysis of words and pictures. Best presented as “word stories” or “video stories.” 12356

4 2. Gather First Impressions Guiding Questions: What did you talk about with interviewees? What are your initial theories of what is working and what is not? Did anything surprise you? Discuss the quality and breadth of data with data collection team.

5 3. Organize and “Clean” Your Data Examine all the data related to each research question separately. Ideally, there is more than one data source for each question. Throw out surveys or written responses that are less than halfway completed

6 Overview of Data Analysis Process 1. Form a data collection team 2. Gather first impressions of data 3. Organize and “Clean” your data 4. Analyze data5. Prepare findings & recommendations 6. Discuss emerging findings with key stakeholders

7 Tips for Designing & Choosing Methods Consider your resource requirements. Determine the appropriateness of methods for respondents, organizational approach, and setting. Ensure that new data gathered fits well with existing data sources. Determine that methods fit with evaluation questions. Select methods that are easy to use.

8 How, What, Why? Choosing Methods (cont’d) 2. Identify a desired outcome/output and choose a method to measure it. AAttendance: sign-in sheets, intake forms, etc. WWriting skills: writing samples KKnowledge of community: survey of youth knowledge 3. More than one method enhances credibility. Gathering multiple sources of evidence provides opportunities to include different perspectives. SSurveys and interviews: Surveys provide numerical account of youth experience and outcomes. Interviews highlight youth voices and stories.

9 How, What, Why? Choosing Methods 1. Consider evaluation questions. Your choice of tools should be driven by the types of questions you want answered (how, what, how many, why). Why questions are best addressed through interviews and focus groups. How What How Many Why How questions help you understand how something happened or the process of implementing a program. Interviews and focus groups. What questions help you document what program staff have done and what participants have experienced. All methods. How many questions are best addressed through surveys, activity logs, intake data, etc.

10 Implementation Quality & Quantity Outcomes Effectiveness, Magnitude & Satisfaction Setting Assumptions Connecting Logic Model to Evaluation Questions Context Inputs Activities Outputs Short-term Outcomes Intermediate Outcomes Long-term Outcomes Adopted from WKKF Logic Model Development Guide p.36 What aspects of our situation most shaped our ability to do the work we set out to do in our community? What did our program accomplish in the community? What resulted from our work in the community? What have we learned about doing this kind of work in a community like ours?

11 Identify Evaluation Questions Evaluation questions focus on understanding how your program will meet its intended goals. Good questions enable you and others to get the answers you need to tell your story. Pick a program activity that you want to evaluate. Decide whether you want your questions to focus on the processes or outcomes of your work. Use simple, concrete language that focuses on what you need to know.

12 Identify strengths and weaknesses of program activities. Benefits of Evaluation Inform practice to improve program. Build organizational capacity. Inform and refine community change efforts. Enhance personal growth and development among staff and youth participants. Provide evidence of program effectiveness. Improve ability to plan and implement programs/campaigns. Document program progress toward meeting goals. Identify unmet community needs and assess impacts of social change efforts. Provide feedback to staff/participants on their work. Recognize accomplishments and provide suggestions for improvement. Report to community and funders about program effectiveness.