1. How To Track PSE Outcomes: Designing Intentional Evaluations to Improve Program Quality
Ross Whiting, Ph.D.
Inclusive Education Project Manager
Academy for Adult Learning
March 23, 2017

2. Pennsylvania’s University Center for Excellence in Developmental Disabilities Education, Research and Service

3. Outline Outputs vs. Outcomes Example: the Academy for Adult Learning
Why Track PSE Outcomes
Types of Evaluation
Practical: Students, Mentors/Tutors, Faculty
Theoretically Rich: Students
Examples
Developing program-specific evaluations.

4. Outputs vs. Outcomes Outputs – what you produce
81 graduates.
Classes in 13 of 17 Temple Colleges and Schools.
1,280 hours of mentor and tutor support.
Outcomes – the impact of your efforts
93% of grads say the AAL prepared them for work.
Self-Determination (Wehmeyer & Kelchner, 1995) scale percentile score rose from 47% to 66%.
90% of mentors and tutors say their experience benefitted them personally.

5. Two-Year AAL Four-Year AAL
Bi-weekly Seminar
Leadership, Diversity, and Inclusion Certificate; Enrichment
Auditing, PASCEP classes
For-credit, auditing, and PASCEP classes; Certificates possible
One semester employment experience
Five semester employment experience
Separate orientation and training
Inclusive orientation and training
Student-only capstone project
Joint mentor-mentee capstone project
IOD Supported Certificate
Temple University sanctioned credential and certificates
High-quality mentor and tutor supports
High-quality mentor and tutor supports; ongoing trainings
Faculty/Professional Advisory Council
Faculty/Professional and Participant Advisory Councils
Pre-post evaluation
Pre-post and ongoing digital evaluation

6. Why track PSE Outcomes? Learn Improve Performance
Meet program goals
Useful, actionable information
Improve Performance
Identify process problems
Demonstrate Superior Outcomes (Or seek technical assistance)
Program working? Share it!
Need support? Reach out!

7. Learn – Improve – Demonstrate: Types of Evaluation Used to Track Outcomes
Practical
Used to evaluate growth in areas affected by your policies
Students: skill development, independence, confidence
Theoretically Rich
Based on research; different use and audience
Self-Determination (Wehmeyer and Kelchner)
Self-Efficacy (Bandura, 1977)
Motivation and Learning (Pintrich, 1991)

8. Learn – Improve – Demonstrate: Practical Student Evaluation
Learn – Pre-eval
Supports
Accommodations
Social life
Learning preferences
Learn - Post-eval:
Current activities
Social and Academic Experiences
Accommodations
Growth in skills and abilities (prepared to acquire job and skills for work, etc.)

9. Learn – Improve – Demonstrate: Practical Student Evaluation
93% of students said skills gained in AAL prepared them for work;
50% of students currently working; 55% actively applying to other jobs.
Students would like more employment experience during time in the AAL.
Improve:
Expanding employment experiences
Demonstrate: Outcomes to be reported

10. Learn – Improve – Demonstrate: Practical Student Evaluation
100% enjoyed experiences and felt respected
7% felt lonely
14% struggled
Improve:
Coordinated and active digital monitoring in classrooms
Mentors and tutors: supports, accommodations, inclusion
Demonstrate: Technical Assistance received

11. Learn – Improve – Demonstrate: Practical Mentor and Tutor Evaluation
Learn – Pre-eval
Demographics
Experiences
Expected role and time commitment
Expected benefits
Learn - Post-eval:
Demographics
Experiences
Role and time commitment
Evaluation of bi-weekly meetings
Support from others and staff; support requests
Benefits and challenges
Contributions to student learning

12. Learn – Improve – Demonstrate: Practical Mentor and Tutor Evaluation
83% found bi-weekly mentor/tutor meetings useful (17% no)
89% received appropriate support from Temple Staff (11% no)
84% of mentors and tutors felt that they support each other (16% no)
Hardest part of role: not knowing how to work with a person with ID
Improve:
Monthly joint enrichments
Demonstrate: Outcomes to come!

13. Learn – Improve – Demonstrate: Practical Faculty Evaluation
100% said AAL students:
Add value to their classes
Made them more aware of their own teaching style
Demonstrated clear growth and development
Had appropriate supports in class
Would teach AAL students again, and would recommend teaching AAL students to other faculty.
Improve: High quality faculty connections (inclusion supplement)
Demonstrate: DREAM symposium 2017

14. Learn – Improve – Demonstrate: Theoretically Rich Student Evaluation
ARC Self-Determination Scale (Wehmeyer & Kelchner, 1995)
Academic Self-Efficacy Subscale (Zajacova, Lynch, and Espenshade, 2005)
Motivated Strategies for Learning Questionnaire (MSLQ), (Pintrich, 1991).

15. Self-Determination Study
Ross D. Whiting, Ph.D., Urban Education, Temple University
Moira Kirby, Ph.D. Candidate, Special Education, Temple University
Self-determination: Acting as the primary causal agent in one’s life (Wehmeyer, Kelchner, and Richards, 1996)
Self-determination linked to improved quality of life (Lachapelle, et al., 2005)
Two-year, mixed-methods study following 12 students in the AAL.
ARC Self-Determination Scale (Wehmeyer & Kelchner, 1995)
Autonomy, Self-Regulation, Psychological Empowerment, Self-Realization
Observations of weekly seminar, document analysis, and informal interviews.

16. Arc Self-Determination Scale
Percentile score growth – as a class. (n=12)
Self-Determination, Category or Component Parts
ARC Scale Percentile Mean Beginning Year 1
ARC Percentile Mean End Year 2
Self-Determination
47.27
63.67
Autonomy
51.18
62.34
Self-Regulation
54.45
60.75
Psychological Empowerment
48.00
71.33
Self-Realization
65.55
71.58

17. Arc Self-Determination Scale
Student
ARC SD Scale Percentile
Beginning Year 1
ARC SD Scale Percentile End
Year 2
Percentile Change
Michael 69 37
-32%
Felicia 45
-24%
Connor 65 53
-12%
Katrina 40 29
-11%
Courtney 67 57
-10%
Quinton 90 95
+5%
Rodger 21
+32%
Shanice 70
+33%
Alexander 26 73
+37%
Latoya 39 79
+40%
Grace 44 88
+44%
Drew 22 85
+63%
Individual Growth
Most grew; some did not.
Observational data useful.

18. Qualitative Data Analysis
Open coding – Atlas.TI
People
Topics
Acting on Preferences, Independence, Support etc.
Instruction
Prepared Lecture, group work, activity, etc.
Interrater Reliability: 89% (Moira Kirby, PhD Candidate, Special Education)

19. Top three themes among students with self-determination growth:
Acting on preferences
Discuss or demonstrate acting on preferences and beliefs
Independence
Discuss or demonstrate making choices or autonomy
Support (independence)
Receives appropriate support
Requests support

20. Learn – Improve – Demonstrate Student Self-Determination
Most students have improved SD; some did not.
Students who grew:
Acted on preferences,
Were independent, and
Had appropriate supports
Improve:
Four-year program: more choice, training for mentors and tutors, digital student evaluations
Demonstrate:
State of the Art; manuscripts in preparation

21. Learn – Improve – Demonstrate: Developing Your Own Tools: Learn
Identify areas of importance; evaluate (Google forms, survey monkey, etc.)
Students, practical:
Independence, choice, career/academic skills, advocacy, leadership, social life and participation, supports.
Students, theoretically rich:
self-determination, self-efficacy, motivation, attitudes.
Mentors/tutors: skills, attitudes, social life, etc.
Faculty: teaching skills, attitudes, supports, etc

22. Learn – Improve – Demonstrate: Developing Your Own Tools: Improve
Be critical of your policies and practices.
Example: Supporting mentors/tutors
Majority positive; improvement needed
Example: Self-determination
Growth as a class; not as individuals
Identifying what caused growth is key

23. Learn – Improve – Demonstrate: Developing Your Own Tools: Demonstrate
Positive outcomes – demonstrate
One-sheets or information booklets
Conference presentations
Peer-reviewed or non-peer reviewed journals
Room for growth: seek technical assistance
Find partners; draw from their knowledge
Reach out to faculty and experts
Be comfortable with criticism.

24. Bibliography
Bandura, A. (1977). Self-efficacy: toward a unifying theory of behavioral change. Psychological review, 84(2), 191. Lachapelle, Y., Wehmeyer, M. L., Haelewyck, M. C., Courbois, Y., Keith, K. D., Schalock, R., & Walsh, P. N. (2005). The relationship between quality of life and self‐determination: an international study. Journal of Intellectual Disability Research, 49(10), Pintrich, P. R. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). Wehmeyer, M.L., Kelchner, K., Richards, S. (1996). Essential characteristics of self determined behavior of individuals with mental retardation. American Journal on Mental Retardation, 100(6), Wehmeyer, M. L. and Kelchner, K. (1995) The Arc’s self-determination scale. The Arc of the United States. Zajacova, A., Lynch, S. M., & Espenshade, T. J. (2005). Self-efficacy, stress, and academic success in college. Research in higher education, 46(6),

25. Institute on Disabilities at Temple University 1755 N 13th Street Student Center, Room 411S Philadelphia, PA Tel: Fax: Web: Ross Whiting, Ph.D. Tel: