1. Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter

2. Introduction Jan Norton Email: snowfanjan@gmail.com Phone: (319) 499-1372 Master’s in Educational Research& Psychology 25+ years as director of multi-discipline tutoring centers with CRLA certifications Former reviewer for CRLA tutor training certification; current NADE certification reviewer for tutoring, SI, and developmental education programs NCLCA Lifetime Learning Center Leadership certification

3. Webinar Contents Terminology The Black Box Timetables Data/Measurement Data/Measurement Cautions Program Outcomes & Examples Student Outcomes & Examples Closing

4. Terminology Visions Missions Values Goals Objectives Outcomes Program vs. Student Outcomes Learning Outcomes

5. The Black Box Something Happens

6. The Black Box Something Happens What is going on in there? How do we measure it?

7. Timetables Scope of assessment ◦ ≥ 6 years ◦ 2 years ◦ 1 year ◦ 1 semester ◦ 1 course ◦ 1 assignment ◦ 1 tutoring session

8. Timetables Planning for Assessment ◦ Continual processes ◦ Multiple times per semester ◦ Once or twice per semester ◦ Two or three times per year ◦ Once per year ◦ Every two years ◦ Every 4 years

9. Data/Measurement Why Measure What to measure ◦ Before (Benchmarks) ◦ During ◦ After (Outcomes)

10. Data/Measurement How to measure ◦ Program-generated data ◦ Course-specific data ◦ Campus data ◦ Campus surveys ◦ Standardized tools ◦ Locally developed tools

11. Data/Measurement More about locally developed tools ◦ Surveys ◦ Likert-type scales ◦ Explicated scales ◦ Behavioral rubrics ◦ Self assessment by student ◦ Tutor assessment ◦ Observer assessment

12. Data/Measurement Cautions Direct vs. Indirect Measures Data access and consistency Realistic limitations Cooperation The Baseline Challenge Time & money (and staff and comfort and ……)

13. Chat Time for Questions & Contributions Finished the foundation: what we can and can’t measure, when & how to measure, and what data we will gain Moving on to examples of program-level outcomes

14. Program Outcomes Possible Areas of Assessment Focus ◦ Establishing baselines ◦ Usage ◦ Satisfaction or quality ◦ Level/variety of services ◦ Staff development ◦ Reaching certification or meeting standards

15. Program Outcomes Examples Outcome Statement ◦ As a result of the new marketing strategy, the number of students attending group tutoring sessions will increase by 10% in Spring 2013. Timetable ◦ One semester Starting Measure (Benchmark) ◦ Average of past attendance Ending Measure ◦ Workshop attendance records

16. Program Outcomes Examples Outcome Statement ◦ During 2012-13, the number of tutors who earn CRLA Level I certification will increase by 5%. Timetable ◦ One academic year Starting Measure (Benchmark) ◦ Number of tutors achieving Level I in previous year(s) Ending Measure ◦ Number of tutors achieving Level I in 2012-13

17. Program Outcomes Examples Outcome Statement ◦ Outside reviewers using the NADE self-study process will score the tutoring program at 85% or above on all sections. Timetable ◦ One year Starting Measure (Benchmark) ◦ Internal review & self-study Ending Measure ◦ External review

18. Program Outcomes Examples Outcome Statement ◦ As a result of tutor training sessions, tutors will demonstrate an improved understanding of Bloom’s Taxonomy. Timetable ◦ One semester or year, depending on training schedule Starting Measure (Benchmark) ◦ Pre-test Ending Measure ◦ Post-test

19. Chat Time for Questions & Contributions Finished program-level outcomes Moving on to examples of student outcomes

20. Student Outcomes: Measures Qualitative ◦ Interviews & focus groups ◦ Opinion surveys ◦ Observations or self reports Quantitative ◦ Assignment & course grades ◦ Instrument scores ◦ Observation rubrics

21. Student Outcomes: Types Affective ◦ Build confidence ◦ Improve attitude ◦ Reduce anxiety Cognitive ◦ Mastery of material ◦ Conscious strategies use

22. Student Outcomes Examples Outcome Statement ◦ After being tutored for at least one hour, clients will be more confident in their ability to accomplish the academic task they focused on in tutoring. Timetable ◦ One tutoring session Starting Measure (Benchmark) ◦ Client self-report Ending Measure ◦ Client self report

23. Student Outcomes Examples Outcome Statement ◦ Students who work with the study skills tutors will demonstrate an increased ability to use memorization strategies. Timetable ◦ One tutoring session Starting Measure (Benchmark) ◦ Tutor or observer checklist Ending Measure ◦ Tutor or observer checklist

24. Student Outcomes Examples Outcome Statement ◦ 75% of the students who work with a reading tutor for at least two sessions will score at least 75% on the main ideas reading test. Timetable ◦ Within three weeks Starting Measure (Benchmark) ◦ Reading quiz at start of tutoring Ending Measure ◦ Chapter reading test in class

25. Student Outcomes Examples Outcome Statement ◦ Students who attend at least 10 group tutoring sessions for Algebra Review 090 will earn an average grade of B or above in the course. Timetable ◦ One semester, length of course Starting Measure (Benchmark) ◦ Placement score into course Ending Measure ◦ Course grade

26. Student Outcomes Examples Outcome Statement ◦ After attending mandatory tutoring for all 5 of their developmental education courses, students will be retained until the start of their junior year at a rate at least 25% higher than developmental students for whom tutoring was not required. Timetable ◦ Two years Starting Measure (Benchmark) ◦ Developmental course placement Ending Measure ◦ Enrollment level

27. Student Outcomes Examples Outcome Statement ◦ Students attending SI for World Religions will “[apply] previously understood information, concepts, and experiences to a new situation or setting.” (CAS) Timetable ◦ Within one semester Starting Measure (Benchmark) ◦ Pre-survey Ending Measure ◦ Post-survey

28. Closing Check with your institutional research or assessment office Review early drafts of outcomes statements with your supervisor Pre-test any locally created measurement tool Be prepared for outcomes that are not met or wildly exceeded

29. Closing Remember the black box: we do not own or control the box, and our ability to see into it is very limited Questions & Contributions?

30. Thank You for Participating ! Jan Norton snowfanjan@gmail.com (319) 499-1372