What Really Works? An Impact Case Study on Learning and Acting At Scale Brandon Moore, Tina Donahoo

Session Summary 01 About Mt. San Jacinto College 02 The Problem with Data and Assessment 03 Rigorous Findings on the Learning Resource Center 04 Plan for Using Results 05 Civitas Learning Findings

One Campus – San Jacinto One Center – Menifee (larger than San Jacinto) Two Off-Site Locations – San Gorgonio and Temecula Recently acquired 350,000 square foot building in Temecula 26,000 Headcount and 12,000 FTES HSI Designation – 50% Latina/Latino Demographic differences between campuses

What we’ve learned and where we’re going… Before Today Trying to focus efforts and resources toward interventions that actually increase persistence People like data when it supports current beliefs and push back when it challenges what they believe or feel is true The result: bean counting … AKA usage metrics Can assess interventions, like the Learning Resource Center, with ease Minimize selection bias to understand actual impact of program The result: data-informed decisions that get us closer to closing achievement gaps Brandon: Share story of what they did before and after Then turn it over to Tina to talk about Impact

IMPACT! Statistically rigorous tool used to measure the impact on persistence of a program, initiative, or service Prediction-based, Propensity Score Matching Matching a student who participated in an initiative with a similar student who did not Two-dimensional matching ensures matched pair of students are similar in both persistence likelihoods & propensity to participate in the initiative and controls for selection bias. What makes impact different - the rigorous statistical method that we use to measure efficacy removing the need for an RCT Measure on two different variables to identify truly comparable groups of students for a more accurate apples-to-apples comparison of outcome

Learning Resource Center…What We Found Learning Resource Center Impact Analysis 4.3 percentage point(pp) lift in persistence for students who visit the Learning Resource Center Subpopulation Findings Greatest impact on critical student groups: 6.8 pp for lowest prediction probability group 5.8 pp for first term students 4.92 pp for African American students 4.8 pp for Hispanic or Latino students Student Engagement Findings 6.8 percentage point lift in persistence for bottom quartile students, who made up less than 17% of the total number of visitors Application of Findings Next Step: Identifying data- informed action to close achievement gaps based on sub-population findings Use Impact more!

What We’re Doing Now Take a more granular look at the Learning Resource Center Assess the impact of various services provided by the LRC Understand what’s working and for whom Before we just tell students to just show up at the LRC we need to have an idea of what services to direct them to Prioritize other persistence initiatives for measurement

Measurement Intelligence Unification WHY OUR CIVITAS EXISTS Quantify impact of student supports Drive continuous learning and improvement across all initiative types Open and connected intelligence Powering differentiated and personalized student supports Flexible starting points Bringing together people, technology, and supporting services Improving outcomes across student lifecycle Believe so strongly Our priority as a business is to build the strongest and most impactful System of Intelligence for Higher Education and graduate more students every year. The keys to doing so are dependent on a combination of Intelligence, Measurement & Unification Measurement –Measurement must be the ability to not only see what is working and what is not, but for whom. We have developed tools and methods that drive optimization. Continuous learning is critical for assessments, accreditations and grant evaluation, and fundraising. Intelligence –This cannot be data that lives in closed systems and operates on averages, creating noise that’s reacting to symptoms rather than root causes. It must be connected intelligence that illuminates the differences between students and populations and evolves along with life and logistics. Unification -- This is about effectively bringing people and tools together -- to deliver coordinated care to support our students. Making sure that every dollar you invest in your team is more effective. Our applications make it intuitive and simple for institutions and people to make effective, timely & personalized decisions which lead to improved student outcomes. Our community is on a mission to help a million more students each year earn degrees by 2025.

60% 40% …but 20% …but 16% THE OPPORTUNITY TO IMPROVE STUDENT SUCCESS — ANALYSIS BASED ON 1,000+ INITIATIVES RUN BY 55 INSTITUTIONS — 60% 40% Learning & Sharing of initiatives show a statistically significant positive impact… of initiatives show little to no statistically significant impact… …but 20% of the students participating in these “successful” initiatives WERE NOT positively impacted. …but 16% of the students participating in these “unsuccessful” initiatives WERE positively impacted. Decisioning Key points: Brandon’s findings were highly positive and that’s excellent The news isn’t the same across the board In the more than 1,000 initiatives run through Impact, we’ve seen that 40% of initiatives show little to no impact on persistence But the key is knowing for whom things are and aren’t working: For the 60% that help, one out of five students aren’t benefiting from the initiative For the 40% that aren’t, 16% of students ARE benefitting This particular finding is highly generalized and the story will change from institution to institution- the key is knowing what actionable insights you can pull from your unique data at your unique institution which serves your unique students

AT EXCLUSIVELY 2-YEAR INSTITUTIONS, THE STORY CHANGES ... 61% 23% 16% Learning & Sharing of initiatives show a statistically significant positive impact… of initiatives show little to no statistically significant impact… of initiatives show a negative impact… Decisioning TRANSITION: Just to bring the point of individual insights home, here are the findings across exclusively 2 year institutions- they look different than the general population

8.18% average lift Advising 1 TOP 3 MOST EFFECTIVE INITIATIVES ACROSS 2-YEAR INSTITUTIONS 4.59% average lift Tutoring 2 3.80% average lift First Year Seminar 3

civitaslearning.com/clarity Want to know more? Come chat with one of us! civitaslearning.com/clarity