1. Developing metrics and predictive algorithms for your institution– Marist Story JISC LEARNING ANALYTICS NETWORK EVENT SANDEEP JAYAPRAKASH

2. Sandeep Jayaprakash LEAD DATA SCIENTIST MARIST COLLEGE & APEREO Twitter - @sandeep_jay1

4. Presentation Overview Open Academic Analytics Initiative (OAAI) ◦Objectives ◦Data extraction & preparation ◦Predictive Models and Results ◦Impact on Student Success ◦Deliver insights to end users

5. PRACTICAL EXAMPLE: EARLY ALERT SYSTEM Predictive Analytics in Action: Open Academic Learning Analytics Initiative

6. Open Academic Analytics Initiative EDUCAUSE Next Generation Learning Challenges (NGLC) in United States Funded by Bill and Melinda Gates Foundations $250,000 over a 15 month period Goal: Leverage Big Data and analytics to create an open-source academic early alert system and research “scaling factors”

7. Input Data Considerations A predictive model is usually as good as its training data Good: ◦Volume - Have lots of data (multiple semesters) ◦Variety - Diverse data ◦Veracity – Data should support the value system Not so good: ◦Data Quality Issues ◦Unbalanced classes (at Marist, 6% of students at risk. Good for the student body, bad for training predictive models )

8. Learning Analytics preparedness Learning Analytics is inter-disciplinary and coordination among different groups High level buy in from management for a smooth execution Data comes from a wide range of systems Ethics and policy should go hand in hand

9. Open Academic Analytics Initiative (OAAI) Jisc Data Specification Github link

10. Predictive Modeling process

11. Feature Extraction - Data Quality Issues Variability in instructor’s assessment criteria Variability in workload criteria across modules Variability in period used for prediction (early detection) Variability in grading criteria across modules (partial grades with variable contribution)

12. Data Quality Issues Variability in VLE tools deployment by instructors Variability in usage of tools by students Result - Missing Values and holes in your data Modules Tools

13. How Do we address them ? Handling Variability - Use ratios and class averages ◦Activity - Percent of usage over Avg percent of usage per course ◦Grades - Effective Weighted Score / Avg Effective Weighted Score Handling missing values ◦Follow an 80 / 20 rule for selection of metrics ◦Perform data imputations to further enrich the quality ◦Build cohort based models to leverage more predictors Sampling - Balance the datasets

14. Predictors of Student Risk  Some predictors were discarded if not enough data was available.  VLE predictors were measured relative to course averages.

15. Machine Learning Classifiers Logistic Regression C4.5/C5.0 Boosted Decision Tree Support Vector machines

16. Predictive Performance of Marist Model

17. Research Design Models were developed based on Marist data ◦ 85 % accuracy in capturing at-risk students Deployed OAAI system to 2200 students across four institutions ◦Two Community Colleges (FE institutions) ◦Two Historically Black Colleges and Universities (BaME institutions) Design > One instructor teaching 3 sections of the same module ◦One section was control, other 2 were treatment groups Each instructor received an AAR three times during the semester ◦Intervals were 25%, 50% and 75% into the semester

18. Institutional Profiles

19. Predictive Model Portability Findings Conclusion 1.Predictive models are more “portable” than anticipated. 2.It is possible to create generic models and the “process can be ported” for use at specific types of institutions. 3.It opens a possibility to explore library of open predictive models and techniques that could be shared across institutions to Learn @ Scale

20. Intervention Research Findings - Final Course Grades Analysis showed a statistically significant positive impact on final course grades ◦No difference between treatment groups Saw larger impact in spring than fall Similar trend among low income students

21. Intervention Research Findings - Content Mastery Student in intervention groups were statistically more likely to “master the content” than those in controls. ◦Content Mastery = Grade of C or better Similar for low income students.

22. Intervention Research Findings - Withdrawals Students in intervention groups withdrew more frequently than controls Possibly due to students avoiding withdrawal penalties. Consistent with findings from Purdue University

23. "Not only did this project directly assist my students by guiding students to resources to help them succeed, but as an instructor, it changed my pedagogy; I became more vigilant about reaching out to individual students and providing them with outlets to master necessary skills. P.S. I have to say that this semester, I received the highest volume of unsolicited positive feedback from students, who reported that they felt I provided them exceptional individual attention! Instructor Feedback

24. JAYAPRAKASH, S. M., MOODY, E. W., LAURÍA, E. J., REGAN, J. R., & BARON, J. D. (2014). EARLY ALERT OF ACADEMICALLY AT-RISK STUDENTS: AN OPEN SOURCE ANALYTICS INITIATIVE. JOURNAL OF LEARNING ANALYTICS, 1(1), 6-47.EARLY ALERT OF ACADEMICALLY AT-RISK STUDENTS: AN OPEN SOURCE ANALYTICS INITIATIVE More Research Findings…

25. Dashboards – Deliver insights

26. Learning Activity Radar Chart Short Demo video link

27. Radar chart - Low Risk pattern example

28. Early Alert Insights - Risk Quadrant Student Engagement Student Performance High Performance High Engagement Low Performance High Engagement Low Performance Low Engagement High Performance Low Engagement

29. Future Research ◦Expand our feature set ◦Try to further minimize percentage of false alarms we raise ◦Scalability enhancements leveraging Hadoop/Spark ◦Dynamic modeling capabilities ◦More UX research on building intuitive dashboards

30. Want the latest updates? Join the mailing list! analytics@apereo.org (subscribe by sending a message to analytics+subscribe@apereo.org) Apereo Learning Analytics Initiative Wiki: https://confluence.sakaiproject.org/x/rIB_BQ GitHub: https://github.com/Apereo-Learning-Analytics-Initiative Sandeep Jayaprakash: Sandeep.Jayaprakash1@marist.edu