1. Tutor Program Evaluation: How Effective is my Program?
Geoff Thames – Senior Coordinator, Learning Support Services

2. The SALT Center
Comprehensive academic support program for students with learning and attention challenges
3 Main Components
Weekly meetings with Strategic Learning Specialists
Tutoring
Tech Coaching

3. Tutoring Services SALT Tutoring Math & Science Lab Writer’s Lab
1 hour appointments
Standing Appointments
Math & Science Lab
Drop-in tutoring
1 hour appointments
Writer’s Lab
Drop-in tutoring
1 hour appointments
Explain 3 different subcenters
Students can make their own appts through TutorTrac
Students log their own visits when they attend tutoring sessions

4. Find someone to work with (small groups are good!)
Why Evaluate?
Find someone to work with (small groups are good!)
Briefly describe your program
Discuss what you would like to find out about your program
Consider these questions in your discussion:
What are the risks of evaluation?
What are some potential benefits of evaluation?

5. Where to Start? Have clean data! Develop a process Time commitment
Determine ‘target’ factors – what is realistic given staffing/time constraints?
Think longitudinally – keep the process consistent, if possible
This will allow for analysis of cohorts
Potential source of limitations, if inconsistent
Time commitment
When to clean?
Who will clean?
‘Visits Export’ and ‘Students By – Appointments/Students by??’ reports

6. So, you are sitting on a mountain of data….
Consider specific student outcomes
Course grades
GPA
Exam scores
What else?
What center usage factors might influence these outcomes?
Visits with tutors
Hours spent in tutoring
Certification level of tutors
Supplemental instruction
Specialist visits

7. r = .74!!!
Test Score
Tutoring Usage

8. What could influence outcomes in math?
Motivation?
Prior academic achievement?
Demographic information?
Visiting with university support staff?
These data might require special access or permission to collect
Possible Institutional Review Board (IRB) approval
Omission of certain variables might bias the results of your analysis! (Stock & Watson, 2007):
1. is there another variable that might correlate with a student’s likelihood to seek tutoring?
Could this variable directly influence the outcome of a student’s likelihood of passing the math course?

9. Determine the effect of ‘X’ on ‘Y’
Research Question:
What is the effect of math tutoring on the incidence of passing a math course?
Outcome, or Dependent Variable is binary: The student either passes or does not pass the course.
The input, or Independent Variable is continuous: The amount of hours that a student spent meeting with math tutors.
Think in terms of
The effect of a unit increase in ‘X’ on the outcome ‘Y’
Linear models derived from line function: Y = mX + B

10. Logistics of Program Evaluation
Can’t randomly assign students to use or not use tutors
Counterfactual approach
Consideration of the outcome, had tutoring services not been available
Control Variables (Stock & Watson, 2007)
Correlation with the likelihood of a student to seek out math tutoring
Potential to influence the outcome of passing the math course
Avoid ‘Overfitting’ the model (Tabachnick & Fidel, 2013)
Ratio of cases to variables should be considered
Dependent on the sample size

11. Spend some time with your dataset:
Data Screening
Spend some time with your dataset:
Get to know’ the cases – there might be some issues
Look for outliers, or extreme values
Look for patterns that don’t make sense
Possible data entry errors?
How was the coding system set up?
Let’s try screening some data
Work with your group/partner – Find some issues in the dataset
Explain your rationale if you believe that you find errors

12. We found that math tutoring helps!
Workshop Activity
We found that math tutoring helps!
What do we do with this finding?
We also found that the more a student visits with academic support staff, the less likely they are to pass math…
How do we explain this?
Consider if we found that math tutoring didn’t help…
What do we do?

13. Reporting Results Go back to “Effect of unit increase in X on Y”
Discuss Model Fit
Basically, how good is the model?
Example:
A binary logistic regression analysis was conducted to determine the effect of math tutor usage on students’ likelihood of passing a math course. The Nagelkerke pseudo R2 statistic revealed that 56.5 percent of the variability was accounted for in this regression analysis. According to the Hosmer and Lemeshow test, the regression model did not deviate significantly from a perfect model. When controlling for all other variables, one additional hour of math tutor usage was significantly associated with a 19 percent increase in the likelihood of a student passing a math course. Thus, students who use additional math tutoring are more likely to pass the math course.

14. 1. Determine research question 2. Data Cleaning
Recap
1. Determine research question
2. Data Cleaning
3. Target additional data (prior academic achievement, etc.…)
Get permission!
4. Data Screening
5. Data Analysis (Select model based upon research question and constraints of data collected)
6. Reporting

15. References
Stock, J., & Watson, M. (2007). Introduction to econometrics. Pearson Education, Inc.
Tabachnick, B., & Fidel, L., (2013). Using multivariate statistics. Sixth edition. Pearson.

16. Geoff Thames: gthames@email.arizona.edu
Thank you!
Contact info:
Geoff Thames: