NSF LSAMP National Meeting Evaluation Presentation Kenneth Rath SageFox Consulting Group

About SageFox Evaluators for two current LSAMP projects Garden State LSAMP Urban Massachusetts LSAMP Former evaluators for two previous LSAMP projects New England LSAMP (Phase 1) California State University LSAMP (SFSU segment) 15 years of experience evaluating STEM pathways programs for URM students

What We Look For: Evidence That… An activity has occurred The outcomes of that activity are positive Opportunities for continuous improvement The activity can be scaled and sustained The last one is one of the key foci of LSAMP

The Question: The Problem: How effective is a given intervention at helping reach LSAMP goals? The Problem: How does our understanding of the intervention match what the students are experiencing?

Problem 1: Knowing What Students are Doing We want to be able to match participation in activities with outcomes, as reported on surveys Attendance data are not collected for many activities

GS-LSAMP: Activities Across 8 Campuses Recruitment by students On-campus research Off-campus research Research conferences LSAMP meetings LSAMP club Other STEM clubs Networking event Transfer events GRE workshop Career workshop Interviewing workshop Academic advising Study circles Study tips workshop Faculty presentations of research Presentations by alumni Presentations by industry representatives Student presentations of research Peer mentoring High school outreach Tutoring PLTL Supplemental instruction Academic support workshops Summer bridge program Trips to STEM-related sites Community service activities

GS-LSAMP: Legacy Survey Which of the following academic support activities did you participate in this year?  Academic support workshops  Academic advising seminars  Received one-on-one tutoring  Worked as a one-on-one tutor  Assistance with test prep  On-line tutoring support Which of the following community activities did you participate in this year?  Peer networking opportunities  Access to LSAMP room/lounge  Worked with a student mentor  Worked as a student mentor  GS-LSAMP Scholars IT Network  Personal or mentee relationship with the LSAMP coordinator or other LSAMP-associated faculty member  Participated in a community with other LSAMP students on your campus  Participated in a community with other LSAMP students on other campuses

Issues Survey only sort of matches activity list Activities in same category could be very different at different campuses Students may not be able to match their activity with the category given Activities may or may not be associated with LSAMP Students say they participate in activities that coordinators are not aware of happening on their campuses

Next Steps Discussions with all site coordinators to identify specific LSAMP-funded or promoted activities on their campuses Customized surveys for each campus Ability to make generalized statements about very few interventions

Problem 2: Knowing What It Means When Students are Doing Something Some core activities are supported at many campuses in the alliance Although there may be a model for how these activities are supposed to work, fidelity to that model can vary

UM-LSAMP: Supplemental Instruction All eight campuses have utilized SI Based on U. Missouri-Kansas City Model: Focus on academic success and student retention Target historically difficult courses Peer-assisted study sessions led by those who were successful in the course Students work collaboratively on class and supplemental material Focus on achieving excellence/mastery rather than remediation

SI Implementation: Variance from Model Identity of facilitator (not always a peer) Relationship of facilitator to course instructor Training of facilitators Covering multiple courses Possible online/virtual components Student attendance Receipt of feedback on performance Relationship of activity to other activities on campus

SI Implementation: Difficulties Site coordinator may not know structure of SI implementation SI designer may have a better idea, but things change in implementation SI implementation may change at an institution over time or from one class to another Getting data from students on what happened is challenging Variations in response rate Timing of survey vs. when students attend Matching SI attendance to course performance may be hampered by poor (or no) attendance data

SI Implementation: Making Sense of It All Effects of one implementation may not be predictive of another implementation or say anything about the value of the intervention overall Knowing the overall impact of an intervention and how it is to be implemented is crucial for scaling

So What Do We Do? Try (sometimes successfully) to convince leaders and representatives of importance of understanding what is happening related to outcomes Focus first on the intervention at the campus level Try to understand impacts at the alliance level, knowing that the degree of variation and limited funding makes that very difficult Try to make some sense of the variation in outcomes based on how it was implemented to find the best model(s) for the circumstances