How To Track PSE Outcomes: Designing Intentional Evaluations to Improve Program Quality Ross Whiting, Ph.D. Inclusive Education Project Manager Academy for Adult Learning March 23, 2017

Pennsylvania’s University Center for Excellence in Developmental Disabilities Education, Research and Service

Outline Outputs vs. Outcomes Example: the Academy for Adult Learning Why Track PSE Outcomes Types of Evaluation Practical: Students, Mentors/Tutors, Faculty Theoretically Rich: Students Examples Developing program-specific evaluations.

Outputs vs. Outcomes Outputs – what you produce 81 graduates. Classes in 13 of 17 Temple Colleges and Schools. 1,280 hours of mentor and tutor support. Outcomes – the impact of your efforts 93% of grads say the AAL prepared them for work. Self-Determination (Wehmeyer & Kelchner, 1995) scale percentile score rose from 47% to 66%. 90% of mentors and tutors say their experience benefitted them personally.

Two-Year AAL Four-Year AAL Bi-weekly Seminar Leadership, Diversity, and Inclusion Certificate; Enrichment Auditing, PASCEP classes For-credit, auditing, and PASCEP classes; Certificates possible One semester employment experience Five semester employment experience Separate orientation and training Inclusive orientation and training Student-only capstone project Joint mentor-mentee capstone project IOD Supported Certificate Temple University sanctioned credential and certificates High-quality mentor and tutor supports High-quality mentor and tutor supports; ongoing trainings Faculty/Professional Advisory Council Faculty/Professional and Participant Advisory Councils Pre-post evaluation Pre-post and ongoing digital evaluation

Why track PSE Outcomes? Learn Improve Performance Meet program goals Useful, actionable information Improve Performance Identify process problems Demonstrate Superior Outcomes (Or seek technical assistance) Program working? Share it! Need support? Reach out!

Learn – Improve – Demonstrate: Types of Evaluation Used to Track Outcomes Practical Used to evaluate growth in areas affected by your policies Students: skill development, independence, confidence Theoretically Rich Based on research; different use and audience Self-Determination (Wehmeyer and Kelchner) Self-Efficacy (Bandura, 1977) Motivation and Learning (Pintrich, 1991)

Learn – Improve – Demonstrate: Practical Student Evaluation Learn – Pre-eval Supports Accommodations Social life Learning preferences Learn - Post-eval: Current activities Social and Academic Experiences Accommodations Growth in skills and abilities (prepared to acquire job and skills for work, etc.)

Learn – Improve – Demonstrate: Practical Student Evaluation 93% of students said skills gained in AAL prepared them for work; 50% of students currently working; 55% actively applying to other jobs. Students would like more employment experience during time in the AAL. Improve: Expanding employment experiences Demonstrate: Outcomes to be reported

Learn – Improve – Demonstrate: Practical Student Evaluation 100% enjoyed experiences and felt respected 7% felt lonely 14% struggled Improve: Coordinated and active digital monitoring in classrooms Mentors and tutors: supports, accommodations, inclusion Demonstrate: Technical Assistance received

Learn – Improve – Demonstrate: Practical Mentor and Tutor Evaluation Learn – Pre-eval Demographics Experiences Expected role and time commitment Expected benefits Learn - Post-eval: Demographics Experiences Role and time commitment Evaluation of bi-weekly meetings Support from others and staff; support requests Benefits and challenges Contributions to student learning

Learn – Improve – Demonstrate: Practical Mentor and Tutor Evaluation 83% found bi-weekly mentor/tutor meetings useful (17% no) 89% received appropriate support from Temple Staff (11% no) 84% of mentors and tutors felt that they support each other (16% no) Hardest part of role: not knowing how to work with a person with ID Improve: Monthly joint enrichments Demonstrate: Outcomes to come!

Learn – Improve – Demonstrate: Practical Faculty Evaluation 100% said AAL students: Add value to their classes Made them more aware of their own teaching style Demonstrated clear growth and development Had appropriate supports in class Would teach AAL students again, and would recommend teaching AAL students to other faculty. Improve: High quality faculty connections (inclusion supplement) Demonstrate: DREAM symposium 2017

Learn – Improve – Demonstrate: Theoretically Rich Student Evaluation ARC Self-Determination Scale (Wehmeyer & Kelchner, 1995) Academic Self-Efficacy Subscale (Zajacova, Lynch, and Espenshade, 2005) Motivated Strategies for Learning Questionnaire (MSLQ), (Pintrich, 1991).

Self-Determination Study Ross D. Whiting, Ph.D., Urban Education, Temple University Moira Kirby, Ph.D. Candidate, Special Education, Temple University Self-determination: Acting as the primary causal agent in one’s life (Wehmeyer, Kelchner, and Richards, 1996) Self-determination linked to improved quality of life (Lachapelle, et al., 2005) Two-year, mixed-methods study following 12 students in the AAL. ARC Self-Determination Scale (Wehmeyer & Kelchner, 1995) Autonomy, Self-Regulation, Psychological Empowerment, Self-Realization Observations of weekly seminar, document analysis, and informal interviews.

Arc Self-Determination Scale Percentile score growth – as a class. (n=12) Self-Determination, Category or Component Parts ARC Scale Percentile Mean Beginning Year 1 ARC Percentile Mean End Year 2 Self-Determination 47.27 63.67 Autonomy 51.18 62.34 Self-Regulation 54.45 60.75 Psychological Empowerment 48.00 71.33 Self-Realization 65.55 71.58

Arc Self-Determination Scale Student ARC SD Scale Percentile Beginning Year 1 ARC SD Scale Percentile End Year 2 Percentile Change Michael 69 37 -32% Felicia 45 -24% Connor 65 53 -12% Katrina 40 29 -11% Courtney 67 57 -10% Quinton 90 95 +5% Rodger 21 +32% Shanice 70 +33% Alexander 26 73 +37% Latoya 39 79 +40% Grace 44 88 +44% Drew 22 85 +63% Individual Growth Most grew; some did not. Observational data useful.

Qualitative Data Analysis Open coding – Atlas.TI People Topics Acting on Preferences, Independence, Support etc. Instruction Prepared Lecture, group work, activity, etc. Interrater Reliability: 89% (Moira Kirby, PhD Candidate, Special Education)

Top three themes among students with self-determination growth: Acting on preferences Discuss or demonstrate acting on preferences and beliefs Independence Discuss or demonstrate making choices or autonomy Support (independence) Receives appropriate support Requests support

Learn – Improve – Demonstrate Student Self-Determination Most students have improved SD; some did not. Students who grew: Acted on preferences, Were independent, and Had appropriate supports Improve: Four-year program: more choice, training for mentors and tutors, digital student evaluations Demonstrate: State of the Art; manuscripts in preparation

Learn – Improve – Demonstrate: Developing Your Own Tools: Learn Identify areas of importance; evaluate (Google forms, survey monkey, etc.) Students, practical: Independence, choice, career/academic skills, advocacy, leadership, social life and participation, supports. Students, theoretically rich: self-determination, self-efficacy, motivation, attitudes. Mentors/tutors: skills, attitudes, social life, etc. Faculty: teaching skills, attitudes, supports, etc

Learn – Improve – Demonstrate: Developing Your Own Tools: Improve Be critical of your policies and practices. Example: Supporting mentors/tutors Majority positive; improvement needed Example: Self-determination Growth as a class; not as individuals Identifying what caused growth is key

Learn – Improve – Demonstrate: Developing Your Own Tools: Demonstrate Positive outcomes – demonstrate One-sheets or information booklets Conference presentations Peer-reviewed or non-peer reviewed journals Room for growth: seek technical assistance Find partners; draw from their knowledge Reach out to faculty and experts Be comfortable with criticism.

Bibliography Bandura, A. (1977). Self-efficacy: toward a unifying theory of behavioral change. Psychological review, 84(2), 191. Lachapelle, Y., Wehmeyer, M. L., Haelewyck, M. C., Courbois, Y., Keith, K. D., Schalock, R., & Walsh, P. N. (2005). The relationship between quality of life and self‐determination: an international study. Journal of Intellectual Disability Research, 49(10), 740-744. Pintrich, P. R. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). Wehmeyer, M.L., Kelchner, K., Richards, S. (1996). Essential characteristics of self determined behavior of individuals with mental retardation. American Journal on Mental Retardation, 100(6), 632-642. Wehmeyer, M. L. and Kelchner, K. (1995) The Arc’s self-determination scale. The Arc of the United States. Zajacova, A., Lynch, S. M., & Espenshade, T. J. (2005). Self-efficacy, stress, and academic success in college. Research in higher education, 46(6), 677-706.

Institute on Disabilities at Temple University 1755 N 13th Street Student Center, Room 411S Philadelphia, PA 19122 Tel: 215-204-1356 Fax: 215-204-6336 Email: IOD@temple.edu Web: www.disabilities.temple.edu Ross Whiting, Ph.D. Tel: 215-204-7660 Email: ross.whiting@temple.edu