“Advanced” Data Collection January 27, 2005

Slide 2 Innovation Network, Inc. Who We Are: Innovation Network National nonprofit organization Committed to evaluation as a tool for empowerment Build evaluation capacity of nonprofits and funders so they can better serve their communities Practice a participatory approach

Slide 3 Innovation Network, Inc. Who We Are: Innovation Network

Slide 4 Innovation Network, Inc. Objectives of This Session Review the five major groups of data collection methods Discuss some of the subtleties of web-based data collection

Slide 5 Innovation Network, Inc. Data Collection Program Plan Evaluation Plan Data Collection Strategy

Slide 6 Innovation Network, Inc. Data Collection: 3 Steps Choose the method Decide which people or records will be the source of the information Determine the level of effort involved in using that method with that population

Slide 7 Innovation Network, Inc. Data Collection Steps 1 & 2: Choose Method, Identify Source Review documents Observe Talk to people Collect written responses Pictorial/multimedia

Slide 8 Innovation Network, Inc. Data Collection Step 3: Level of Effort Instrument development Cost/practicality of actually collecting data Cost of analyzing and presenting data Also consider: Communication Power Proxy Power

Slide 9 Innovation Network, Inc. Document Review Examples: Attendance records Application information Feedback forms Web statistics Curricula or content

Slide 10 Innovation Network, Inc. Document Review Efficient and cost-effective Dependent on the existence of documentation Better for process evaluation than outcome evaluation

Slide 11 Innovation Network, Inc. Observation When conducting and evaluation the observer: Does not intervene in any way Does not stand out or draw attention Does not record names or any identifying information It is important that observations be held over time (and at different places/days/times)

Slide 12 Innovation Network, Inc. Observation Inexpensive way to obtain an outside perspective Highly susceptible to bias Subjective; relies on anecdotal data

Slide 13 Innovation Network, Inc. Interviews Structured Follow a predetermined interview protocol Identify broad evaluation questions and specific interview questions in advance Unstructured Begin with one or two open-ended questions Proceed based on the responses received

Slide 14 Innovation Network, Inc. Interviews Provides the story behind someone’s experiences Useful as a follow-up to a survey to further investigate responses Time consuming (both in terms of collection and analysis) Respondents susceptible to “leading” by interviewers

Slide 15 Innovation Network, Inc. Interview Tips Make the respondent feel comfortable Introduce yourself and the evaluation Start with easy, short answer questions – end with more revealing, long answer questions Record the interviews Important for capturing great quotes

Slide 16 Innovation Network, Inc. Focus Groups A small group of people (who have something in common) who are led through a conversation by a moderator and discuss their opinions and provide suggestions on a topic.

Slide 17 Innovation Network, Inc. Focus Groups Efficient and cost-effective way to gather qualitative data Requires a skilled moderator/facilitator Collects consensus rather than individual perspectives

Slide 18 Innovation Network, Inc. Focus Group Tips Ideally 8-10 person groups Maximizes amount of information you can gather without the suppression of voices Don’t get bogged down Rewards Breakfast/Lunch/Dinner Gift certificates

Slide 19 Innovation Network, Inc. Surveys Delivery options: By mail In person Online By telephone

Slide 20 Innovation Network, Inc. Surveys Collect quantitative information Efficient Open-ended questions (qualitative data) can be cumbersome for respondents and data entry

Slide 21 Innovation Network, Inc. Survey Tips Introduce the survey Describe the purpose and what will be done with the results Reinforce that it is confidential and that you request they respond candidly Lead with easy, closed-ended questions Group categories of questions together Consider how long it will take for someone to respond rather than the length of pages

Slide 22 Innovation Network, Inc. Web Statistics # of hits Most popular pages Most popular downloads Common “drop off” points Where are they coming from?

Slide 23 Innovation Network, Inc. Don’t Be Afraid to Ask More! Why are they there? Who are they? What do they do? What are they looking for? Did they find it? What are they going to do with it? Are they willing to help you with future evaluation projects?

Slide 24 Innovation Network, Inc. Usability Testing We can help… x 109

Slide 25 Innovation Network, Inc. Next Steps We can do more! Online, instructor-led trainings Online, self-paced modules Individual technical assistance

Slide 26 Innovation Network, Inc. Next Steps We can do more! Survey Builder Survey Library Data Analysis Tool

Slide 27 Innovation Network, Inc. Thanks for Your Participation! Measure results. Make informed decisions. Create lasting change. Innovation Network, Inc K St. NW, 11 th Floor Washington, DC (202) Ehren Reed: