Program Evaluation for Nonprofit Professionals Unit 3: Collecting the Data

You’ve planned your evaluation

It’s time to follow that plan

What is meant by data? Data is simply information. It can come in many forms: Written (documents) Visual (anything observable) Auditory (hearing what people say)

Check your evaluation plan Look at the data collection methods you identified. What do you already have? Output information (training sessions held, clients served, etc.) Customer/client satisfaction information? What will you need to collect?

Common Data Collection Methods 1.Document review 2.Surveys 3.Interviews 4.Focus Groups 5.Observations

Document Review Most commonly, this is a review of records, files or other already existing materials.

Document Review Lowest level of effort. Most commonly used to track outputs for process evaluation. Identify what you are looking for BEFORE beginning your review. Use a guide or a checklist. Example: You may review client intake files to identify how many families with more than 2 children were served in the last year.

Surveys Print or electronic surveys are an easy way to gather information from a large group of people.

Surveys Can be medium to high effort. Are familiar to most people. Can be done on paper or in electronic formats Can gather information for both process and outcomes evaluations. Example: A job training program wants to collect information from graduates on their satisfaction with the program and their job placement results.

Survey Tips Keep questions simple and clear. Have one or more people proof read your survey instrument before you pilot test it. Always pilot test a survey with a small group of respondents to make sure it functions as you expected. Free online survey tools (Survey Monkey, Google Forms) can analyze data for you.

Interviews Data can be collected by interviewing subjects one on one. Interviews can be done in person or over the phone.

Interviews Usually a high level of effort. Can be done in person or over the telephone. Must be conducted one interviewee at a time. More time consuming than surveys. Yield richer information than surveys because of the ability to ask for clarifications and further information. Example: An organization dedicated to assisting homeless men wants to get a better understanding of the current needs of this population.

Interview Tips Always draft an interview protocol – the standard list of questions you will ask each interviewee. Read your questions out loud to a colleague first, and revise. Recording interviews can help you capture data, but always ask permission first!

Focus Groups Focus groups are group interviews.

Focus Groups Generally a high level of effort Usually done in person Allows for interaction between participants Broader reach than individual interviews Used in situations where there is benefit to interaction between participants Example: You want to talk with graduates of a job training program to understand how the program has helped them.

Focus Group Tips Pay close attention to logistics Provide incentives to increase attendance (at least refreshments) Consider providing child care if the participants are parents. Staffed by at least two people: one facilitator and one note- taker Protocol should start general and end specific

Observations Observations are any data collected visually. This includes observations about behavior or environment.

Observations Low level of effort Less intrusive than surveys, interviews and focus groups Example 1: You work for a community development organization that wants to know which community parks get the most use. Example 2: You’ve been running a program to teach conflict resolution to children. You observe their social interactions to assess their use of the strategies.

Observation Tips Always have an observation checklist that outlines what you will look for. This will provide consistency. Test your checklist at least once before putting it into use. Never observe children without parent permission.

Additional Considerations When do you need this information for reporting? Will you collect this information once, or on a regular basis? What tools (surveys, protocols, checklists) will you need to develop? What already exists? What can you borrow from colleagues and partners?

Next Steps 1.Readings – lots more specific information on data collection tools and approaches 2.Take the quiz 3.Bring your questions and ideas to the discussion board.