David Ockert Toyo University Doing Survey Research How to design, analyze, and write up research using questionnaires David Ockert Toyo University

“In my experience, academic researchers are usually not statistics experts. Nor should they be—they are experts in their own field.” - Karen Grace-Martin, founder and president of The Analysis Factor. Karen was a professional statistical consultant at Cornell University for seven years and taught statistics courses at the University of California. She co-wrote an introductory statistics textbook Data Analysis with SPSS.

Survey Design Replication? A comparative analysis? Experiment or intervention? Original survey instrument? How many items / questions? How many numerical options? (T/F, 3-7 choices?)

Experiment Analysis Pre-test Experimental Group & Control Group Intervention / Experiment Post-test

Gathering Data and Analysis Classroom only? Nationwide? International? Aggregate by gender? Age? Major? www.surveymonkey.com

The first research project A comparative analysis

Operationalizing Affective Variables

A Comparative Analysis The Survey Instrument What classroom activities do you enjoy or find motivating?   Circle the number on the right that best matches your opinion. 1 = strongly dislike, 2 = dislike, 3 = neutral, 4 = like, 5 = strongly like 1) Lecture (Listen to the teacher and stay in my seat) 2) Listening exercises (using a CD, tape or DVD) 3) Dialogue / reading practice from the text 4) Writing exercises 5) Translation exercises 6) Grammar drills / practice 7) Small-group / team activities 8) Info-seek / finding information activities 9) Problem-solving activities 10) Activities where I am moving around in the room 11) Tasks that are intellectually challenging 12) Pair-work

Reporting the Basic Statistics

Checking for Variable Relationships

Instrument Reliability Test re-test? Split-half method (Cronbach’s alpha) The alpha reliability estimate of .76 indicates inconsistency in the data.

Instrument Validity Face Validity Do the items make sense? Content Validity Will the items ‘hang’ together as anticipated? Principal components analysis (PCA) For these results, 40% explained variance is satisfactory (for the social sciences).

Tests for Factorability

Principal Components Analysis

The second research project A pre- post-test control (PPC; Morris, 2008) group design

Experiment Surveys & Analysis

Survey Descriptive Statistics & Correlations

Tests for Factorability

The PCA Results

Statistical Significance P < .01 – great P < .05 – good enough P < .10 – acceptable for an experimental study (Cohen, 1992)

Experimental Group Pre- & Post-test Results

Effect Sizes Cohen (1988, 1992) has provided suggestions about what constitutes a small or large effect for differences in mean scores: MM r = .20 (small effect): In this case the effect explains 1% of the total variance. MM r = .50 (medium effect): The effect accounts for 9% of the total variance. MM r = .80 (large effect): The effect accounts for 25% of the variance. (p. 57)  

The Effect Sizes

Resources Social Sciences Statistics http://www.socscistatistics.com Psychometrica: The Institute for Psychological Diagnostics http://www.psychometrica.de Daniel Soper’s website http://www.danielsoper.com GraphPad Software http://www.graphpad.com/quickcalcs Karen Grace-Martin’s website The Analysis Factor http://www.theanalysisfactor.com

Why are statistics so daunting?? Example: What is ‘N’? And ‘n’? Sample? Population? The number of participants? Or…

What about this?!