Collecting Data for Impact Debra Spielmaker Utah State University

Why do you need to collect data? What data do you need to collect?

Stakeholders want to know if you are making a difference (having an impact) and if your difference is practical (impact is worth the cost)

Data for Measuring Impact… Logic Model Source: W.K. Kellogg Foundation, Logic Model Development Guide, p. 3.

Data Collection Methods for Measuring Impacts Interviews Focus Groups Observations Ethnographies, Oral Histories, and Case Studies Content Analysis Documents & Records Tests Surveys

right questions to determine impacts Surveys… Asking the right questions to determine impacts

Survey Development Tips Attributes: Who people are – personal or demographic characteristics Beliefs, Attitudes, Opinions: Perceptions people hold or psychological states* Knowledge: What people know, how well they understand something* Behavior: What people do, have done in the past or plan to do in the future* * These are used for impact statements

Question Construction Open-Ended Use simple wording Avoid jargon or abbreviations Be specific Be clear Avoid assumptions Consider language, reading level, and age Close-Ended Have clear instructions Avoid double-barreled questions Use complete sentences Select clear and logical response categories Define all choices (multiple choice/Likert Scale, rank) Avoid opt-out/neutral/fence-sitting choices Avoid negative words and double negatives Avoid bias in the questions Avoid long questions

Questions to Avoid Questions about what you do not intend to change Questions where you already know the answers Ambiguous, awkward unanswerable questions

Practice Scenarios GMO Seed Kit Use Evaluation Leadership Summit Evaluation Develop as many questions and response options as you can in the time allotted. Share with another group for peer review Share a question with the entire group

Data Analysis Use descriptive statistics N (sample or population number of those surveyed) Mean (Average could be %) Mode Standard Deviation Range If doing tests for statistical significance, seek advice before developing the instrument Caution on self-report survey of perceptions used as an assessment…

Resources for SDL Evaluation Tool Kit – Pell Institute CYFAR Modules – University of Minnesota The "Survey Playbook" by Matthew Champagne Survey Research, Research Design and Data Collection Web Center of Social Research Methods Writing Impact Statements, Multistate Research Fund