Targeting Support and Intervention with Predictive Analytics to Improve Retention and Success Sherry Woosley, Ph.D. Director of Analytics & Research EBI

Minnesota Symposium on Analytics in Support of Advising Plan History: MAP Project: MAP-Works – Overview – Predictive Analytics – Using the Information – Outcomes Lessons Learned

Minnesota Symposium on Analytics in Support of Advising Making Achievement Possible (MAP) History

Minnesota Symposium on Analytics in Support of Advising In 1988, Ball State had a number of concerns… – Unrealistic expectations of first-year students – Retention rates – Questions about the efficacy of our intervention efforts – Mid-term was too late Created Making Achievement Possible (MAP), a student retention initiative History

Minnesota Symposium on Analytics in Support of Advising Collaborative Effort: MAP Committee

Minnesota Symposium on Analytics in Support of Advising Long History of Assessing MAP Evaluations about survey content and the reports Comparisons between MAP responses and similar surveys Assessment notes exploring MAP responses and outcomes Studies of staff usage

Minnesota Symposium on Analytics in Support of Advising When Ball State started… We didn’t predict risk. We gave feedback to students based on professional judgment. We gave all the data to staff. We believed that –patterns of data (rather than individual data points) were important. –staff could recognize the patterns.

Minnesota Symposium on Analytics in Support of Advising Issues Our assumption that staff could/would recognize patterns wasn’t exactly correct. –Some patterns are not obvious. –Folks had trouble dealing with so much data. –“When I have 300 students in my hall, I don’t have time to look at 300 reports to find those who need the most immediate assistance.” Adjusting well to BSU A Fit in well at BSU A Pleased with BSU A Satisfied with courses A Making friends at BSU A Attending all classes A Satisfied with social life A Involved in campus activities D Satisfied with academic life A Managing time well U Adjusting to study demands U Comfort asking instructors for help A Comfort with different ethnic (2) A Have someone to talk to (4) A Feel safe on campus A Most students have same values U *Feeling more tense U *Would rather be home D *Feel anxious about decisions U *Feel diff from other students D *Being on own is not easy D *Much harder time than others D PERSONAL REFLECTIONS SA -strongly agree A -agree U -undecided D -disagree SD -strongly disagree *Since these items were asked in the negative D & SD responses are desirable.

Minnesota Symposium on Analytics in Support of Advising Issues We also faced a technology/resource issue: –Although our IR office could do statistics and basic reporting, we didn’t have the technology skills, resources (and time!), or expertise to move this forward. –Our computing services office also didn’t have the resources or interest in taking this on. Solution: Collaborative partnership with EBI.

Minnesota Symposium on Analytics in Support of Advising EBI is an assessment and benchmarking leader in higher education. They have have survey more than 9 million people worked with colleges and universities worldwide have partnered with a variety of higher education organizations. Educational Benchmarking Most importantly, they shared our vision!

MAP-Works

Components of a successful retention strategy: Early, systematic, and comprehensive identification of student issues Motivate, inform and coordinate faculty/staff to intervene effectively Provide reporting that benchmarks, and quantifies efforts and outcomes Focuses students on college success behaviors

Student Profile Data ERP or other Interface Student Profile Info Campus Resources Survey Data Alerts/Notes Social Norming Expectations Campus Resources The MAP-Works Process Advisors Residence Hall Staff Seminar Instructors Coaches Dashboard Talking Points Share Notes/Alerts Coordinate Actions

Minnesota Symposium on Analytics in Support of Advising Predictive Analytics

Minnesota Symposium on Analytics in Support of Advising First Steps in Predictive Modeling In 2005 and 2006, we –Rewrote the survey (using all the information we had gathered through the years). –Ran a pilot administration of the survey. –Confirmed the reliability and validity of the survey. –Began predictive modeling.

Minnesota Symposium on Analytics in Support of Advising Survey Instrument(s) Theories (Astin, Bandura, Bean, Braxton, Chickering, Kuh, Pascarella, Tinto, Upcraft, Weidman, etc.) Validity testing with students Theme coding of open-ended items On-going statistical analysis

Minnesota Symposium on Analytics in Support of Advising Process Statistical Methods Descriptives Correlations Regression Models (Logistic & Linear) Discriminant Analysis Classification Trees Measurement and Path Models Neural Networks And more… Statistical Methods Descriptives Correlations Regression Models (Logistic & Linear) Discriminant Analysis Classification Trees Measurement and Path Models Neural Networks And more… Variables Demographics Admissions Survey Factors (points in time) Non-response to a survey Mid-Term Grades Outcomes Retention Academic Performance Variables Demographics Admissions Survey Factors (points in time) Non-response to a survey Mid-Term Grades Outcomes Retention Academic Performance

It’s fairly obvious that “Travis Gatlin” is at risk… It’s less obvious that “Jessica Anderson”, is equally at risk. MAP-Works Risk Indicator is not a sum of all the reds

Minnesota Symposium on Analytics in Support of Advising Lessons about Modeling Modeling is complex, particularly when you’re trying to predict –Individual behavior –Two different outcomes (persistence and academic performance) –Across a variety of campuses –Using data from different points in time We also must be aware of the workloads we create.

Minnesota Symposium on Analytics in Support of Advising Lessons about Modeling Focusing on the accuracy of any retention model is important … but not as important as using the data.

Minnesota Symposium on Analytics in Support of Advising Using the Information

Minnesota Symposium on Analytics in Support of Advising Students Individualized reports with peer comparisons, feedback, and resources Faculty/Staff Talking Points / Dashboards Tracking Screen Searching Group level results Decision makers All of the above plus Executive Summaries Group Level & Crosstab capabilities Usage reports Outcomes reporting Benchmarking reports Reporting

Minnesota Symposium on Analytics in Support of Advising PROVIDING INDIVIDUALIZED FEEDBACK TO STUDENTS

Minnesota Symposium on Analytics in Support of Advising Realign Expectations

Minnesota Symposium on Analytics in Support of Advising Link Behaviors and Outcomes

Minnesota Symposium on Analytics in Support of Advising Connect to Campus Resources

Minnesota Symposium on Analytics in Support of Advising ENGAGING OUR FACULTY / STAFF AROUND INDIVIDUAL STUDENTS

Minnesota Symposium on Analytics in Support of Advising Find My At-Risk Students

Minnesota Symposium on Analytics in Support of Advising Easily Understand Risk

Minnesota Symposium on Analytics in Support of Advising See the Holistic Picture of a Student

Minnesota Symposium on Analytics in Support of Advising ENGAGING OUR FACULTY / STAFF AROUND GROUPS

Minnesota Symposium on Analytics in Support of Advising Target Interventions to Particular Students

Minnesota Symposium on Analytics in Support of Advising Understand Group Differences Overall response rate is interesting but not all that useful… Here are response rates by floor. This was just response rates – imagine survey items by floor or other groups – especially if the data is timely...

Minnesota Symposium on Analytics in Support of Advising Plan Programming or Activities Thanks Nate Cole and Chris Mullen from University of Northern Colorado for this specific idea. They had an in-service event to have housing staff practice developing programs based on data.

Minnesota Symposium on Analytics in Support of Advising Executive Summaries Outcomes Benchmarking Usage Reports Other Reporting Communication tools Alerts Referrals Activity records Additional Coordination Features I forgot to show you…

Minnesota Symposium on Analytics in Support of Advising Lessons about Reporting To get people to pay attention to the information, the reporting must –Be visual –Be intuitive –Be interactive Our results must connect to the existing roles and tasks of faculty/staff. What’s the point if no one uses the information???

Minnesota Symposium on Analytics in Support of Advising What evidence do we have that this is approach works? Outcomes

Minnesota Symposium on Analytics in Support of Advising Ball State Improvements in –Retention: up 5% since Fall 2005 –Academic performance: about 10% fewer students on academic probation at the end of the first semester –Efficiency: we get data and use it faster –Knowledge of student issues: homesickness, social integration, etc. –Efforts related to specific sub-populations: commuters, etc.

Minnesota Symposium on Analytics in Support of Advising

There is no silver bullet (not even analytics). Analytics helps if we are thoughtful and get – The right information – To the right people – At the right time – In the right formats. Final Thoughts We cannot forget that this is really about informing and motivating behavior.

Sherry Woosley, Ph.D. Director of Analytics & Research EBI Thank you.