1. “Advanced” Data Collection January 27, 2005

2. Slide 2 Innovation Network, Inc. Who We Are: Innovation Network National nonprofit organization Committed to evaluation as a tool for empowerment Build evaluation capacity of nonprofits and funders so they can better serve their communities Practice a participatory approach

3. Slide 3 Innovation Network, Inc. Who We Are: Innovation Network www.innonet.org/services/LSC_Train

4. Slide 4 Innovation Network, Inc. Objectives of This Session Review the five major groups of data collection methods Discuss some of the subtleties of web-based data collection

5. Slide 5 Innovation Network, Inc. Data Collection Program Plan Evaluation Plan Data Collection Strategy

6. Slide 6 Innovation Network, Inc. Data Collection: 3 Steps Choose the method Decide which people or records will be the source of the information Determine the level of effort involved in using that method with that population

7. Slide 7 Innovation Network, Inc. Data Collection Steps 1 & 2: Choose Method, Identify Source Review documents Observe Talk to people Collect written responses Pictorial/multimedia

8. Slide 8 Innovation Network, Inc. Data Collection Step 3: Level of Effort Instrument development Cost/practicality of actually collecting data Cost of analyzing and presenting data Also consider: Communication Power Proxy Power

9. Slide 9 Innovation Network, Inc. Document Review Examples: Attendance records Application information Feedback forms Web statistics Curricula or content

10. Slide 10 Innovation Network, Inc. Document Review Efficient and cost-effective Dependent on the existence of documentation Better for process evaluation than outcome evaluation

11. Slide 11 Innovation Network, Inc. Observation When conducting and evaluation the observer: Does not intervene in any way Does not stand out or draw attention Does not record names or any identifying information It is important that observations be held over time (and at different places/days/times)

12. Slide 12 Innovation Network, Inc. Observation Inexpensive way to obtain an outside perspective Highly susceptible to bias Subjective; relies on anecdotal data

13. Slide 13 Innovation Network, Inc. Interviews Structured Follow a predetermined interview protocol Identify broad evaluation questions and specific interview questions in advance Unstructured Begin with one or two open-ended questions Proceed based on the responses received

14. Slide 14 Innovation Network, Inc. Interviews Provides the story behind someone’s experiences Useful as a follow-up to a survey to further investigate responses Time consuming (both in terms of collection and analysis) Respondents susceptible to “leading” by interviewers

15. Slide 15 Innovation Network, Inc. Interview Tips Make the respondent feel comfortable Introduce yourself and the evaluation Start with easy, short answer questions – end with more revealing, long answer questions Record the interviews Important for capturing great quotes

16. Slide 16 Innovation Network, Inc. Focus Groups A small group of people (who have something in common) who are led through a conversation by a moderator and discuss their opinions and provide suggestions on a topic.

17. Slide 17 Innovation Network, Inc. Focus Groups Efficient and cost-effective way to gather qualitative data Requires a skilled moderator/facilitator Collects consensus rather than individual perspectives

18. Slide 18 Innovation Network, Inc. Focus Group Tips Ideally 8-10 person groups Maximizes amount of information you can gather without the suppression of voices Don’t get bogged down Rewards Breakfast/Lunch/Dinner Gift certificates

19. Slide 19 Innovation Network, Inc. Surveys Delivery options: By mail In person Online By telephone

20. Slide 20 Innovation Network, Inc. Surveys Collect quantitative information Efficient Open-ended questions (qualitative data) can be cumbersome for respondents and data entry

21. Slide 21 Innovation Network, Inc. Survey Tips Introduce the survey Describe the purpose and what will be done with the results Reinforce that it is confidential and that you request they respond candidly Lead with easy, closed-ended questions Group categories of questions together Consider how long it will take for someone to respond rather than the length of pages

22. Slide 22 Innovation Network, Inc. Web Statistics # of hits Most popular pages Most popular downloads Common “drop off” points Where are they coming from?

23. Slide 23 Innovation Network, Inc. Don’t Be Afraid to Ask More! Why are they there? Who are they? What do they do? What are they looking for? Did they find it? What are they going to do with it? Are they willing to help you with future evaluation projects?

24. Slide 24 Innovation Network, Inc. Usability Testing We can help… ereed@innonet.org 202.728.0727 x 109

25. Slide 25 Innovation Network, Inc. Next Steps We can do more! Online, instructor-led trainings Online, self-paced modules Individual technical assistance www.innonet.org/services/LSC_Train

26. Slide 26 Innovation Network, Inc. Next Steps We can do more! Survey Builder Survey Library Data Analysis Tool

27. Slide 27 Innovation Network, Inc. Thanks for Your Participation! Measure results. Make informed decisions. Create lasting change. Innovation Network, Inc. 1625 K St. NW, 11 th Floor Washington, DC 20006 (202) 728-0727 www.innonet.org Ehren Reed: ereed@innonet.orgereed@innonet.org www.innonet.org/services/LSC_Train