1. What Really Works? An Impact Case Study on Learning and Acting At Scale
Brandon Moore, Tina Donahoo

2. Session Summary 01 About Mt. San Jacinto College 02
The Problem with Data and Assessment 03
Rigorous Findings on the Learning Resource Center 04
Plan for Using Results 05
Civitas Learning Findings

3. One Campus – San Jacinto
One Center – Menifee (larger than San Jacinto)
Two Off-Site Locations – San Gorgonio and Temecula
Recently acquired 350,000 square foot building in Temecula
26,000 Headcount and 12,000 FTES
HSI Designation – 50% Latina/Latino
Demographic differences between campuses

4. What we’ve learned and where we’re going…
Before
Today
Trying to focus efforts and resources toward interventions that actually increase persistence
People like data when it supports current beliefs and push back when it challenges what they believe or feel is true
The result: bean counting … AKA usage metrics
Can assess interventions, like the Learning Resource Center, with ease
Minimize selection bias to understand actual impact of program
The result: data-informed decisions that get us closer to closing achievement gaps
Brandon: Share story of what they did before and after
Then turn it over to Tina to talk about Impact

5. IMPACT!
Statistically rigorous tool used to measure the impact on persistence of a program, initiative, or service
Prediction-based, Propensity Score Matching
Matching a student who participated in an initiative with a similar student who did not
Two-dimensional matching ensures matched pair of students are similar in both persistence likelihoods & propensity to participate in the initiative and controls for selection bias.
What makes impact different - the rigorous statistical method that we use to measure efficacy removing the need for an RCT
Measure on two different variables to identify truly comparable groups of students for a more accurate apples-to-apples comparison of outcome

6. Learning Resource Center…What We Found
Learning Resource Center Impact Analysis
4.3 percentage point(pp) lift in persistence for students who visit the Learning Resource Center
Subpopulation Findings
Greatest impact on critical student groups:
6.8 pp for lowest prediction probability group
5.8 pp for first term students
4.92 pp for African American students
4.8 pp for Hispanic or Latino students
Student Engagement Findings
6.8 percentage point lift in persistence for bottom quartile students, who made up less than 17% of the total number of visitors
Application of Findings
Next Step:
Identifying data- informed action to close achievement gaps based on sub-population findings
Use Impact more!

7. What We’re Doing Now
Take a more granular look at the Learning Resource Center
Assess the impact of various services provided by the LRC
Understand what’s working and for whom
Before we just tell students to just show up at the LRC we need to have an idea of what services to direct them to
Prioritize other persistence initiatives for measurement

8. Measurement Intelligence Unification WHY OUR CIVITAS EXISTS
Quantify impact of student supports
Drive continuous learning and improvement across all initiative types
Open and connected intelligence
Powering differentiated and personalized student supports
Flexible starting points
Bringing together people, technology, and supporting services
Improving outcomes across student lifecycle
Believe so strongly
Our priority as a business is to build the strongest and most impactful System of Intelligence for Higher Education and graduate more students every year.
The keys to doing so are dependent on a combination of Intelligence, Measurement & Unification
Measurement –Measurement must be the ability to not only see what is working and what is not, but for whom. We have developed tools and methods that drive optimization. Continuous learning is critical for assessments, accreditations and grant evaluation, and fundraising.
Intelligence –This cannot be data that lives in closed systems and operates on averages, creating noise that’s reacting to symptoms rather than root causes. It must be connected intelligence that illuminates the differences between students and populations and evolves along with life and logistics.
Unification -- This is about effectively bringing people and tools together -- to deliver coordinated care to support our students. Making sure that every dollar you invest in your team is more effective.
Our applications make it intuitive and simple for institutions and people to make effective, timely & personalized decisions which lead to improved student outcomes.
Our community is on a mission to help a million more students each year earn degrees by 2025.

9. 60% 40% …but 20% …but 16% THE OPPORTUNITY TO IMPROVE STUDENT SUCCESS
— ANALYSIS BASED ON 1,000+ INITIATIVES RUN BY 55 INSTITUTIONS —
60%
40%
Learning & Sharing
of initiatives
show a statistically significant positive impact…
of initiatives
show little to no statistically significant impact…
…but 20%
of the students participating in these “successful” initiatives WERE NOT positively impacted.
…but 16%
of the students participating in these “unsuccessful” initiatives WERE positively impacted.
Decisioning
Key points:
Brandon’s findings were highly positive and that’s excellent
The news isn’t the same across the board
In the more than 1,000 initiatives run through Impact, we’ve seen that 40% of initiatives show little to no impact on persistence
But the key is knowing for whom things are and aren’t working:
For the 60% that help, one out of five students aren’t benefiting from the initiative
For the 40% that aren’t, 16% of students ARE benefitting
This particular finding is highly generalized and the story will change from institution to institution- the key is knowing what actionable insights you can pull from your unique data at your unique institution which serves your unique students

10. AT EXCLUSIVELY 2-YEAR INSTITUTIONS, THE STORY CHANGES ...
61%
23%
16%
Learning & Sharing
of initiatives
show a statistically significant positive impact…
of initiatives
show little to no statistically significant impact…
of initiatives
show a negative impact…
Decisioning
TRANSITION: Just to bring the point of individual insights home, here are the findings across exclusively 2 year institutions- they look different than the general population

11. 8.18%
average lift
Advising 1
TOP 3
MOST EFFECTIVE INITIATIVES ACROSS 2-YEAR INSTITUTIONS
4.59%
average lift
Tutoring 2
3.80%
average lift
First Year Seminar 3

12. civitaslearning.com/clarity
Want to know more?
Come chat with one of us!
civitaslearning.com/clarity