1. David Ockert Toyo University
Doing Survey Research How to design, analyze, and write up research using questionnaires
David Ockert
Toyo University

2. “In my experience, academic researchers are usually not statistics experts. Nor should they be—they are experts in their own field.”
- Karen Grace-Martin, founder and president of The Analysis Factor. Karen was a professional statistical consultant at Cornell University for seven years and taught statistics courses at the University of California. She co-wrote an introductory statistics textbook Data Analysis with SPSS.

3. Survey Design Replication? A comparative analysis?
Experiment or intervention?
Original survey instrument?
How many items / questions?
How many numerical options?
(T/F, 3-7 choices?)

4. Experiment Analysis Pre-test Experimental Group & Control Group
Intervention / Experiment
Post-test

5. Gathering Data and Analysis
Classroom only?
Nationwide?
International?
Aggregate by gender? Age? Major?

6. The first research project
A comparative analysis

7. Operationalizing Affective Variables

8. A Comparative Analysis The Survey Instrument
What classroom activities do you enjoy or find motivating?
Circle the number on the right that best matches your opinion.
1 = strongly dislike, 2 = dislike, 3 = neutral, 4 = like, 5 = strongly like
1) Lecture (Listen to the teacher and stay in my seat)
2) Listening exercises (using a CD, tape or DVD)
3) Dialogue / reading practice from the text
4) Writing exercises
5) Translation exercises
6) Grammar drills / practice
7) Small-group / team activities
8) Info-seek / finding information activities
9) Problem-solving activities
10) Activities where I am moving around in the room
11) Tasks that are intellectually challenging
12) Pair-work

9. Reporting the Basic Statistics

10. Checking for Variable Relationships

11. Instrument Reliability
Test re-test?
Split-half method (Cronbach’s alpha)
The alpha reliability estimate of .76 indicates inconsistency in the data.

12. Instrument Validity Face Validity Do the items make sense?
Content Validity
Will the items ‘hang’ together as anticipated?
Principal components analysis (PCA)
For these results, 40% explained variance is satisfactory (for the social sciences).

13. Tests for Factorability

14. Principal Components Analysis

15. The second research project
A pre- post-test control (PPC; Morris, 2008) group design

16. Experiment Surveys & Analysis

17. Survey Descriptive Statistics & Correlations

18. Tests for Factorability

19. The PCA Results

20. Statistical Significance
P < .01 – great
P < .05 – good enough
P < .10 – acceptable for an experimental study (Cohen, 1992)

21. Experimental Group Pre- & Post-test Results

22. Effect Sizes
Cohen (1988, 1992) has provided suggestions about what constitutes a small or large effect for differences in mean scores:
MM r = .20 (small effect): In this case the effect explains 1% of the total variance.
MM r = .50 (medium effect): The effect accounts for 9% of the total variance.
MM r = .80 (large effect): The effect accounts for 25% of the variance. (p. 57)

23. The Effect Sizes

24. Resources Social Sciences Statistics http://www.socscistatistics.com
Psychometrica: The Institute for Psychological Diagnostics
Daniel Soper’s website
GraphPad Software
Karen Grace-Martin’s website The Analysis Factor

25. Why are statistics so daunting??
Example: What is ‘N’? And ‘n’?
Sample? Population? The number of participants?
Or…

26. What about this?!