1. Collecting Data for Impact
Debra Spielmaker
Utah State University

2. Why do you need to collect data?
What data do you need to collect?

3. Stakeholders want to know if you are making a difference
(having an impact)
and if your difference is practical
(impact is worth the cost)

4. Data for Measuring Impact… Logic Model
Source: W.K. Kellogg Foundation, Logic Model Development Guide, p. 3.

5. Data Collection Methods for Measuring Impacts
Interviews
Focus Groups
Observations
Ethnographies, Oral Histories, and Case Studies
Content Analysis
Documents & Records
Tests
Surveys

6. right questions to determine impacts
Surveys…
Asking the
right questions to determine impacts

7. Survey Development Tips
Attributes: Who people are – personal or demographic characteristics
Beliefs, Attitudes, Opinions: Perceptions people hold or psychological states*
Knowledge: What people know, how well they understand something*
Behavior: What people do, have done in the past or plan to do in the future*
* These are used for impact statements

8. Question Construction
Open-Ended
Use simple wording
Avoid jargon or abbreviations
Be specific
Be clear
Avoid assumptions
Consider language, reading level, and age
Close-Ended
Have clear instructions
Avoid double-barreled questions
Use complete sentences
Select clear and logical response categories
Define all choices (multiple choice/Likert Scale, rank)
Avoid opt-out/neutral/fence-sitting choices
Avoid negative words and double negatives
Avoid bias in the questions
Avoid long questions

9. Questions to Avoid
Questions about what you do not intend to change
Questions where you already know the answers
Ambiguous, awkward unanswerable questions

10. Practice Scenarios GMO Seed Kit Use Evaluation
Leadership Summit Evaluation
Develop as many questions and response options as you can in the time allotted.
Share with another group for peer review
Share a question with the entire group

11. Data Analysis Use descriptive statistics
N (sample or population number of those surveyed)
Mean (Average could be %)
Mode
Standard Deviation
Range
If doing tests for statistical significance, seek advice before developing the instrument
Caution on self-report survey of perceptions used as an assessment…

14. Resources for SDL Evaluation Tool Kit – Pell Institute
CYFAR Modules – University of Minnesota
The "Survey Playbook" by Matthew Champagne
Survey Research, Research Design and Data Collection
Web Center of Social Research Methods
Writing Impact Statements, Multistate Research Fund