1. Assessing K-12 Service-Learning Impacts and Quality Shelley H. Billig, Ph.D. RMC Research Corporation

2. Overview Evaluation Toolkit Logic Models and Theory of Change Measuring Impact –Academic impact –Civic impact –Personal/social impacts –Others Measuring Quality

3. Evaluation Toolkit How-To Guide For Traditional Evaluation Overview Steps for Planning and Implementation Worksheets Detailed Guidance

4. Session Experiences “Dip in” for various aspects of the evaluation and assessment process.

5. Logic Models Shows how and why the program will work; Connects program activities to program outcomes through outputs: –e.g., activity (input)  Increased student engagement in schools (output)  Increased student academic achievement (outcome).

6. TX K-12 Learn and Serve Logic Model

7. Try It! Start with your activities – what will you do? Specify your goals – what will you accomplish? Identify those factors that will influence results, both positively and negatively. Include demographics, context, implementation issues

8. Logic Model Use Once the logic model has been created with stakeholders, review it. Use the logic model to design or revamp the evaluation. Use the constructs identified in the logic model to guide the development of measures (instrumentation).

9. Logic Model Resources University of Wisconsin-Extension Program Development and Evaluation, www.uwex.edu/ces/pdande www.uwex.edu/ces/pdande W.K. Kellogg Foundation, www.wkkf.org www.wkkf.org –Evaluation Handbook, 1998 –Logic Model Development Guide, 2001

10. Assessing Academic Impact –Where is service-learning likely to have the greatest academic impact? Knowledge areas (general and specific) Types of skills (general and specific) Moderators –What measures are available?

11. Academic Skills Areas of likely impact –Higher order thinking skills such as problem solving, analysis, transfer of learning –Specific knowledge related to service activities, e.g., measurement, writing, synthesizing materials, research What are the available measures? –Rubrics –Test scores –Writing scores –Problem solving essays –Measures of academic engagement –Absenteeism –Surveys –More!

12. Assessing Civic Impacts Where is service-learning likely to have the greatest civic impact? –Knowledge? –Skills? –Dispositions? –Attachment to community? Moderators What measures are available?

13. Civic Impacts Areas of likely impact –Knowledge of community needs –Skills for planning and implementing projects –Taking responsibility –Ethic of service Available measures –Surveys –Focus groups/interviews –Hours of service –Actual changes

14. Assessing Social/Personal Impacts –Where is service-learning likely to have the greatest impact on social/personal areas? Knowledge areas (general and specific) Types of skills (general and specific) Moderators –What measures are available?

15. Social/Personal Outcomes Areas of Likely Impact –Development of character traits such as responsibility, trust, and caring –Respect for diversity –More bonds with adults –More protective factors such as resilience –Greater self-efficacy Measures –Surveys –Focus groups/interviews –Rubrics –Self-assessments

16. Sample Measures of Outcome Areas Examine the samples What is the best use of each? By whom and for what purpose? What do you like and dislike about each?

17. Measuring Moderators Moderators are characteristics that influence outcomes. Examples: Instructor experience; program design; program quality. Need to identify and measure those factors like to serve as moderators.

18. What moderators are likely to be in play in your settings? Consider demographics. Consider program design. Consider quality. Consider instructor characteristics like experience, levels of training, and expertise. Consider partner characteristics.

19. Characteristics of Good Assessments Valid Reliable Objective Useful Items measure only one construct at a time (not double barrelled…) Related to the logic model or assessment questions

20. Open Discussion Questions and Answers

21. Resources National Service-Learning Clearinghouse www.servicelearning.org www.servicelearning.org Billig, S. H., & Waterman, A. S., eds. (2003). Studying service-learning: Innovations in education research methodology. Mahwah, NJ: Lawrence Erlbaum Associates. Available at www.erlbaum.com www.erlbaum.com Bringle, R. G., Phillips, M. A., & Hudson, M. (2004). The measure of service learning: Research scales to assess student experiences. Washington, DC: American Psychological Association. Available at www.apa.org/books www.apa.org/books