Tutor Program Evaluation: How Effective is my Program? Geoff Thames – Senior Coordinator, Learning Support Services

The SALT Center Comprehensive academic support program for students with learning and attention challenges 3 Main Components Weekly meetings with Strategic Learning Specialists Tutoring Tech Coaching

Tutoring Services SALT Tutoring Math & Science Lab Writer’s Lab 1 hour appointments Standing Appointments Math & Science Lab Drop-in tutoring 1 hour appointments Writer’s Lab Drop-in tutoring 1 hour appointments Explain 3 different subcenters Students can make their own appts through TutorTrac Students log their own visits when they attend tutoring sessions

Find someone to work with (small groups are good!) Why Evaluate? Find someone to work with (small groups are good!) Briefly describe your program Discuss what you would like to find out about your program Consider these questions in your discussion: What are the risks of evaluation? What are some potential benefits of evaluation?

Where to Start? Have clean data! Develop a process Time commitment Determine ‘target’ factors – what is realistic given staffing/time constraints? Think longitudinally – keep the process consistent, if possible This will allow for analysis of cohorts Potential source of limitations, if inconsistent Time commitment When to clean? Who will clean? ‘Visits Export’ and ‘Students By – Appointments/Students by??’ reports

So, you are sitting on a mountain of data…. Consider specific student outcomes Course grades GPA Exam scores What else? What center usage factors might influence these outcomes? Visits with tutors Hours spent in tutoring Certification level of tutors Supplemental instruction Specialist visits

r = .74!!! Test Score Tutoring Usage

What could influence outcomes in math? Motivation? Prior academic achievement? Demographic information? Visiting with university support staff? These data might require special access or permission to collect Possible Institutional Review Board (IRB) approval Omission of certain variables might bias the results of your analysis! (Stock & Watson, 2007): 1. is there another variable that might correlate with a student’s likelihood to seek tutoring? Could this variable directly influence the outcome of a student’s likelihood of passing the math course?

Determine the effect of ‘X’ on ‘Y’ Research Question: What is the effect of math tutoring on the incidence of passing a math course? Outcome, or Dependent Variable is binary: The student either passes or does not pass the course. The input, or Independent Variable is continuous: The amount of hours that a student spent meeting with math tutors. Think in terms of The effect of a unit increase in ‘X’ on the outcome ‘Y’ Linear models derived from line function: Y = mX + B

Logistics of Program Evaluation Can’t randomly assign students to use or not use tutors Counterfactual approach Consideration of the outcome, had tutoring services not been available Control Variables (Stock & Watson, 2007) Correlation with the likelihood of a student to seek out math tutoring Potential to influence the outcome of passing the math course Avoid ‘Overfitting’ the model (Tabachnick & Fidel, 2013) Ratio of cases to variables should be considered Dependent on the sample size

Spend some time with your dataset: Data Screening Spend some time with your dataset: Get to know’ the cases – there might be some issues Look for outliers, or extreme values Look for patterns that don’t make sense Possible data entry errors? How was the coding system set up? Let’s try screening some data Work with your group/partner – Find some issues in the dataset Explain your rationale if you believe that you find errors

We found that math tutoring helps! Workshop Activity We found that math tutoring helps! What do we do with this finding? We also found that the more a student visits with academic support staff, the less likely they are to pass math… How do we explain this? Consider if we found that math tutoring didn’t help… What do we do?

Reporting Results Go back to “Effect of unit increase in X on Y” Discuss Model Fit Basically, how good is the model? Example: A binary logistic regression analysis was conducted to determine the effect of math tutor usage on students’ likelihood of passing a math course. The Nagelkerke pseudo R2 statistic revealed that 56.5 percent of the variability was accounted for in this regression analysis. According to the Hosmer and Lemeshow test, the regression model did not deviate significantly from a perfect model. When controlling for all other variables, one additional hour of math tutor usage was significantly associated with a 19 percent increase in the likelihood of a student passing a math course. Thus, students who use additional math tutoring are more likely to pass the math course.

1. Determine research question 2. Data Cleaning Recap 1. Determine research question 2. Data Cleaning 3. Target additional data (prior academic achievement, etc.…) Get permission! 4. Data Screening 5. Data Analysis (Select model based upon research question and constraints of data collected) 6. Reporting

References Stock, J., & Watson, M. (2007). Introduction to econometrics. Pearson Education, Inc. Tabachnick, B., & Fidel, L., (2013). Using multivariate statistics. Sixth edition. Pearson.

Geoff Thames: gthames@email.arizona.edu Thank you! Contact info: Geoff Thames: gthames@email.arizona.edu