1. NSF LSAMP National Meeting Evaluation Presentation
Kenneth Rath
SageFox Consulting Group

2. About SageFox Evaluators for two current LSAMP projects
Garden State LSAMP
Urban Massachusetts LSAMP
Former evaluators for two previous LSAMP projects
New England LSAMP (Phase 1)
California State University LSAMP (SFSU segment)
15 years of experience evaluating STEM pathways programs for URM students

3. What We Look For: Evidence That…
An activity has occurred
The outcomes of that activity are positive
Opportunities for continuous improvement
The activity can be scaled and sustained
The last one is one of the key foci of LSAMP

4. The Question: The Problem:
How effective is a given intervention at helping reach LSAMP goals?
The Problem:
How does our understanding of the intervention match what the students are experiencing?

5. Problem 1: Knowing What Students are Doing
We want to be able to match participation in activities with outcomes, as reported on surveys
Attendance data are not collected for many activities

6. GS-LSAMP: Activities Across 8 Campuses
Recruitment by students
On-campus research
Off-campus research
Research conferences
LSAMP meetings
LSAMP club
Other STEM clubs
Networking event
Transfer events
GRE workshop
Career workshop
Interviewing workshop
Academic advising
Study circles
Study tips workshop
Faculty presentations of research
Presentations by alumni
Presentations by industry representatives
Student presentations of research
Peer mentoring
High school outreach
Tutoring
PLTL
Supplemental instruction
Academic support workshops
Summer bridge program
Trips to STEM-related sites
Community service activities

7. GS-LSAMP: Legacy Survey
Which of the following academic support activities did you participate in this year?
 Academic support workshops
 Academic advising seminars
 Received one-on-one tutoring
 Worked as a one-on-one tutor
 Assistance with test prep
 On-line tutoring support
Which of the following community activities did you participate in this year?
 Peer networking opportunities
 Access to LSAMP room/lounge
 Worked with a student mentor
 Worked as a student mentor
 GS-LSAMP Scholars IT Network
 Personal or mentee relationship with the LSAMP coordinator or other LSAMP-associated faculty member
 Participated in a community with other LSAMP students on your campus
 Participated in a community with other LSAMP students on other campuses

8. Issues Survey only sort of matches activity list
Activities in same category could be very different at different campuses
Students may not be able to match their activity with the category given
Activities may or may not be associated with LSAMP
Students say they participate in activities that coordinators are not aware of happening on their campuses

9. Next Steps
Discussions with all site coordinators to identify specific LSAMP-funded or promoted activities on their campuses
Customized surveys for each campus
Ability to make generalized statements about very few interventions

10. Problem 2: Knowing What It Means When Students are Doing Something
Some core activities are supported at many campuses in the alliance
Although there may be a model for how these activities are supposed to work, fidelity to that model can vary

11. UM-LSAMP: Supplemental Instruction
All eight campuses have utilized SI
Based on U. Missouri-Kansas City Model:
Focus on academic success and student retention
Target historically difficult courses
Peer-assisted study sessions led by those who were successful in the course
Students work collaboratively on class and supplemental material
Focus on achieving excellence/mastery rather than remediation

12. SI Implementation: Variance from Model
Identity of facilitator (not always a peer)
Relationship of facilitator to course instructor
Training of facilitators
Covering multiple courses
Possible online/virtual components
Student attendance
Receipt of feedback on performance
Relationship of activity to other activities on campus

13. SI Implementation: Difficulties
Site coordinator may not know structure of SI implementation
SI designer may have a better idea, but things change in implementation
SI implementation may change at an institution over time or from one class to another
Getting data from students on what happened is challenging
Variations in response rate
Timing of survey vs. when students attend
Matching SI attendance to course performance may be hampered by poor (or no) attendance data

14. SI Implementation: Making Sense of It All
Effects of one implementation may not be predictive of another implementation or say anything about the value of the intervention overall
Knowing the overall impact of an intervention and how it is to be implemented is crucial for scaling

15. So What Do We Do?
Try (sometimes successfully) to convince leaders and representatives of importance of understanding what is happening related to outcomes
Focus first on the intervention at the campus level
Try to understand impacts at the alliance level, knowing that the degree of variation and limited funding makes that very difficult
Try to make some sense of the variation in outcomes based on how it was implemented to find the best model(s) for the circumstances