Conducting Quality Service-Learning Research Andrew Furco Associate Vice President for Public Engagement March 22, 2019 Hong Kong Polytechnic University

Building a Quality Study What are the components found in all high quality studies?

Guiding Principles for Quality in Scientific Research Pose significant questions that can be investigated empirically. Link research to relevant theory and conceptual frameworks. Use methods that permit direct investigation of the question. Provide coherent, explicit chain of reasoning. Replicate and generalize across studies. Disclose research to encourage profession scrutiny and critique Adapted from Shavelson & Towne (2002). Committee on Scientific Principles for Education Research. Washington, DC.

Types of Research Studies Impact Process/Implementation Institutionalization Conceptual Framing Instrument development Meta-analyses

Conducting Quality Research on Service-Learning What are some of the unique challenges that service-learning poses in conducting high quality research?

The Nature of Service-Learning Research Treatment is difficult to define Outcomes are not well-defined Outcomes are not easily measured Unit of analysis challenges (i.e, individualized outcomes) High unpredictability and low capacity for controls Short duration and bound time frames Confounding influences

Common Problem in Service-Learning Research Small sample sizes Correlation ≠ Causation Self-selection bias (non-random assignment) Social desirability Creaming Lack of controls or comparison groups Lack of generalizability (external validity) Not connecting to knowledge base or theory Lack of clear definitions Primary self-reported measures Adapted from Steinberg, Bringle, & Williams, 2009

Selecting/Designing Instruments: Issues to Consider Unit of Analysis (program, department, institution, student population, community, region, etc.) Time (snapshot, change over time, trends) Access to data sources Reliability of data collected Capacity to analyze data Instrument validity/psychometrics Population appropriateness Cost

Reviewing Research What are some of the inherent assumptions in the study? Evaluate the research article based on each of the issues listed on the handout. What are the strengths of the research? What are the limitations of the research? What are recommendations for improving the study? What would you do differently?