1. Program Evaluation & Research In Service-Learning Service-Learning Mini-Institute Lynn E. Pelco, Ph.D. Division of Community Engagement

2. Objectives Understand the importance of assessing student learning and evaluating outcomes, including community benefit, Understand the difference between program evaluation and research, Know where to find tools to aid you with evaluation and research, Be aware of outlets for scholarship related to service-learning

3. What do we really mean when we say we’ll “assess” something?

4.  Assessment vs. Measurement Assessment is the process of documenting knowledge, skills, attitudes and beliefs. Measurement is the assignment of numbers to objects or events.  Evaluation/Program Evaluation A systematic method for collecting, analyzing, & using information to answer questions about the effectiveness & efficiency of a service-learning course/program.  Research/Scholarship An investigation or experimentation aimed at the revision of accepted theories in the light of new facts, or practical application of such new or revised theories.  Qualitative, Quantitative, & Mixed Methods Research Qualitative research is a method of deep inquiry into the reasons underlying human behavior that typically does not utilize statistical/computational techniques. Quantitative research is an empirical investigation of social phenomena via statistical, mathematical or computational techniques. Mixed methods methodology integrates qualitative and quantitative techniques.  Scholarship of Teaching & Learning SoTL is scholarly inquiry into student learning which advances the practice of teaching by making research findings public

5. In the context of service-learning… Why do we do evaluation and/or research? Who are our stakeholders? What do we want to know? How do we measure? When do we measure? What do we do with the information?

6. Developing Research Projects Around Theories Scholarship of Teaching & Learning Theories:  Allport’s Contact Hypothesis, Lewin’s Change Theory, Discipline-based theories:  What theories exist in the center of the intellectual dialogue within your discipline?  What are the names of the top journals in your discipline? Over the past year, what theories appeared on the pages of these journals? Newly published 2-volume book: Clayton, Bringle & Hatcher (2012) Research on service learning: Conceptual frameworks and assessments.

7. What might we evaluate? Cost and Efficiency Outcomes and Impact Process and Implementation Program Design and Theory Needs

8. Getting Started: Developing a Logic Model 1.Goals (determines type of assessment) 2.Assumptions (theoretical, conceptual) 3.Inputs (resources – people, places, things) 4.Activities (things that will occur) 5.Outputs* (things that will be produced) 6.Outcomes* (measureable changes – can be short-term and/or long-term) *How will you measure outputs and outcomes?

9. Examples of Measurement Methods MethodCostLaborAdvantageDisadvantage Commercial TestsHighLowBenchmarkingNot program-specific In-House TestsLowHighDirect and program- specific No benchmarks SimulationsHigh Clear behavioral measure Development time Focus GroupsLowHighDeeper examinationSubjective ObservationsLowHighBehavioral specificityRubric complexity SurveysMed Targeted questionsIndirect measure Archival DataLow Many sources of dataRetrospective PortfoliosLowHighMultiple work samplesComplex scoring Course- Embedded LowHighCourse-specific; individual feedback Restrictive sampling; feedback time

10. Logic models (a.k.a., logical framework, theory of change, or program matrix) are graphical depictions of the logical relationships between the resources, activities, outputs and outcomes of a program. Logic Models Inputs Activities Outputs Outcomes Statement of Goals Each resource involved Each major activity that will take place Each “item” to be produced and how it will be measured Each potential change and how it will be measured

11. Disseminating Results Internal stakeholders –Your own students –Prospective students –VCU faculty and administration External stakeholders –Your community partners –Greater Richmond community –The academic community

12. Presenting and Publishing Results Many avenues for presenting and publishing outcomes associated with service-learning Remember that publication of research involving human subjects requires approval from VCU’s Institutional Review Board (IRB) For more information and resources, visit –http://www.community.vcu.edu/research/disseminating-your- research/publishing-outlets/http://www.community.vcu.edu/research/disseminating-your- research/publishing-outlets/ –http://www.servicelearning.org/what-you-do/conduct-research

13. In your groups: 1.Pick a specific service-learning course or experience that one of you is considering. 2.Decide what aspect you want to assess/evaluate. 3.Draft a model to help you plan your strategy. Build a Logic Model InputsActivitiesOutputs Outcomes Statement of Goals Each resource involved Each major activity that will take place Each “item” to be produced and how it will be measured Each potential change and how it will be measured

14. Questions?