1. Survey Design and Quantitative Data Analysis
Christopher Pena
Director of Data Management
Mike Furno
Assistant Provost, Institutional Research & Analysis

2. Workshop outline Part I - Research design (15 minutes)
Part II - Survey design (30 minutes)
Part III – Analysis (15 minutes)
Questions and discussion (30 minutes)

3. Defining the research question
All good research begins with a well-defined research question that is clear, concise, and testable. In empirical research, the research question may be used to frame the null hypothesis (H0) and the alternative hypothesis (H1).

4. Types of research methods
Quantitative research
Conceptual: Facts about social phenomena
Methods: Measurement
Analysis: Numerical comparisons, hypothesis testing
Qualitative research
Conceptual: Participant experience
Methods: Interviews, observations, source material
Analysis: Textual, thematic
Mixed methods research employs quantitative and qualitative methods concurrently or in succession.

5. Types of research designs
Associative
Explore relationships between variables
Establish causation: No
Quasi-experimental
Lack random assignment
Establish causation : Maybe
Experimental
Random assignment – gold standard in social sciences
Establish causation : Yes (if conditions are met)

6. Sample vs. population
It is rarely feasible to survey an entire population. An appropriately-selected sample can approximate the general population and yield strong estimates of validity and reliability.
Use a tool like G*Power to determine
the appropriate sample size for your
study.

7. Validity In social sciences, validity depends on the context:
Empirical research
The extent to which evidence supports applying findings to new populations or settings (generalizability).
Measurement
The extent to which evidence supports the application of an instrument for its purported use.

8. Evidence for validity Construct validity Content validity
Criterion-related validity
Convergent/divergent validity
Face validity

9. Theories of validity
The unitary theory of validity is dominant today and suggests that validity may be established by the presence of five or more forms of evidence, each of which is considered to be a form of construct validity.
Other theories:
Trinity – content, criterion-related, and construct validity
Duality – internal and external foci
Argument – explicit rationale

10. Reliability
Reliability estimates the extent to which data are consistent across multiple populations, settings, or replications. Reliability increases with variability in the sample.
Not reliable, not valid
Reliable, valid
Reliable, not valid

11. Types of reliability Internal consistency (Cronbach’s alpha)
Parallel forms
Test-retest
Interrater reliability

12. Rigor of research
Your research question and goals determine the rigor required in your design, data collection, and analysis.
Exploratory
Scholarly
Initial research to understand an issue.
Guide future research.
Share results informally.
Establish causal relationship.
Fill gaps in the literature.
Publish or present results formally.

13. Survey development Determine what you want to measure.
Generate an item pool.
Determine the format for measurement.
Have the initial pool of items reviewed by experts.
Conduct cognitive interviews.
Conduct a pilot administration.
Conduct focus groups.
Revise the survey.
Administer the survey.
Analyze the results.
Share and/or use your findings.

14. Scale development Determine what you want to measure.
Generate an item pool.
Determine the format for measurement.
Have initial pool reviewed by experts.
Consider inclusion of validation items.
Administer items to a development sample.
Evaluate the items
Optimize scale length.
Administer the measure.

15. General considerations
Survey length and number of items
Delivery method
Confidentiality and anonymity
Data management

16. Survey tools Qualtrics REDCap
Availability: Faculty, staff, and students
Training: Online training and resources
Security: Moderate
REDCap
Availability: Faculty and some graduate students
Training: Some online resources
Security: High

17. Closed questions Pros Cons Easier to complete
Easer to encode and analyze data
Fewer errors in the data
Cons
Response options may not be exhaustive
More limited understanding of the context
Easy to add too many questions

18. Open questions Pros Cons Respondents can provide their own responses
More complete picture of context
Cons
More time-consuming to complete
More difficult to encode data
More difficult to analyze data

19. Likert scale
A Likert scale is a commonly-used five-point agreement scale with a neutral midpoint:
Strongly agree
Agree
Neither agree no disagree
Disagree
Strongly disagree
Responses may be ordered and coded in ascending or descending order as needed.

20. Other types of response scales
Agreement
Frequency
Satisfaction
Effectiveness
Efficiency

21. Question design Neutral midpoint Do not know option Other option
Forced responses

22. Best practices Practice item symmetry (parallelism)
Avoid double-barreled questions
Avoid ambiguous wording
Avoid overlapping categories
Provide sufficient options

23. Sources of bias and error
Leading questions
Insufficient options
Ordering of options
Social desirability
Survey fatigue
Sensitive/intrusive questions
Jargon

24. Statistical tests Associations and relationships
Chi-square, correlation, linear regression
Classification and prediction
Logistic regression, discriminant function analysis
Mean differences
T-test, ANOVA, ANCOVA, MANOVA, MANCOVA
Proportions
Test of independent proportions

25. Statistical power
During analysis, it is important to balance the risk of Type I (α) and Type II (β) error. In the social sciences, a p-value of .05 is commonly used to test for statistical significance.
Exploration
Explanation, prediction
Placement
p ≤ .10
p ≤ .05
p ≤ .01
p ≤ .001

26. Effect sizes
Effect sizes quantify the strength of a statistically significant relationship or difference. Many journals in the social sciences now require authors to provide effect sizes.
Cohen’s d
Hedges’ g
Eta-squared (η2)
Correlation coefficient (r)
Coefficient of determination (r2, R2)

27. Descriptive statistics
Frequency
Simple counts, crosstabs
Measures of central tendency
Mean, median, mode
Distribution
Normality, variance, standard deviation

28. Statistical models Classification Explanatory Predictive
Group respondents by characteristics or response patterns
Explanatory
Explain contribution of independent variable(s)
Establish causation
Predictive
Predict dependent variable(s)

29. Factor analysis Classical test theory (CTT) Item response theory (IRT)
Identify underlying factors
Explanatory and confirmatory factor analysis
Item response theory (IRT)
Person ability and item difficulty
Logarithmic model to place persons and items
IRT Model

30. Analytic tools Statistical analysis Factor analysis
SPSS, R, Python, JMP, AMOS
Factor analysis
Winsteps, FACETS, ConQuest
Qualitative analysis
NVivo, Dedoose
Visualization
Tableau, Excel

31. Resources and support Center for Visualization and Statistics
Library research staff
Graduate assistants (RMS, psychology, education, social work)
Institutional Research & Analysis

32. Questions and discussion