1. Conducting Quality Service-Learning Research Andrew Furco Associate Vice President for Public Engagement March 22, 2019 Hong Kong Polytechnic University

2. Building a Quality Study
What are the components found in all high quality studies?

3. Guiding Principles for Quality in Scientific Research
Pose significant questions that can be investigated empirically.
Link research to relevant theory and conceptual frameworks.
Use methods that permit direct investigation of the question.
Provide coherent, explicit chain of reasoning.
Replicate and generalize across studies.
Disclose research to encourage profession scrutiny and critique
Adapted from Shavelson & Towne (2002). Committee on Scientific Principles for Education Research. Washington, DC.

4. Types of Research Studies
Impact
Process/Implementation
Institutionalization
Conceptual Framing
Instrument development
Meta-analyses

5. Conducting Quality Research on Service-Learning
What are some of the unique challenges that service-learning poses in conducting high quality research?

6. The Nature of Service-Learning Research
Treatment is difficult to define
Outcomes are not well-defined
Outcomes are not easily measured
Unit of analysis challenges (i.e, individualized outcomes)
High unpredictability and low capacity for controls
Short duration and bound time frames
Confounding influences

7. Common Problem in Service-Learning Research
Small sample sizes
Correlation ≠ Causation
Self-selection bias (non-random assignment)
Social desirability
Creaming
Lack of controls or comparison groups
Lack of generalizability (external validity)
Not connecting to knowledge base or theory
Lack of clear definitions
Primary self-reported measures
Adapted from Steinberg, Bringle, & Williams, 2009

8. Selecting/Designing Instruments: Issues to Consider
Unit of Analysis (program, department, institution, student population, community, region, etc.)
Time (snapshot, change over time, trends)
Access to data sources
Reliability of data collected
Capacity to analyze data
Instrument validity/psychometrics
Population appropriateness
Cost

9. Reviewing Research
What are some of the inherent assumptions in the study?
Evaluate the research article based on each of the issues listed on the handout.
What are the strengths of the research?
What are the limitations of the research?
What are recommendations for improving the study?
What would you do differently?