Developing metrics and predictive algorithms for your institution– Marist Story JISC LEARNING ANALYTICS NETWORK EVENT SANDEEP JAYAPRAKASH.

Slides:



Advertisements
Similar presentations
MCMS Mining Course Management Systems
Advertisements

Taking and Managing Risks with Technology This 30 min presentation will address the conference theme: How can we use what we have learned? Mark Stone:
Forecasting the Demand Those who do not remember the past are condemned to repeat it George Santayana ( ) a Spanish philosopher, essayist, poet.
June 10-15, 2012 Growing Community; Growing Possibilities Josh Baron, Marist College Sandeep Jayaprakash, Marist College Nate Angell, rSmart.
Outcomes of the ACODE 60 workshop on Learning Analytics Associate Professor Rob Phillips Murdoch University, Perth.
1. Abstract 2 Introduction Related Work Conclusion References.
G EORGE W ASHINGTON U NIVERSITY C ENTER FOR E QUITY AND E XCELLENCE IN E DUCATION Education Without Borders: Bridges to Literacy October 19,
Techniques for Improving Student Learning Outcomes Lynn M. Forsythe Ida M. Jones Deborah J. Kemp Craig School of Business California State University,
Enterprise systems infrastructure and architecture DT211 4
Advisors Are Selling, But Are Faculty Buying? Assessing Faculty Buy-in of an Early Academic Alert System Abstract The advising community recognizes the.
Challenge, Potential, and Success: A University-wide Approach to Classroom Response Systems Ed Evans Purdue University Director of Learning Spaces
Race to the Top Program Update January 30, State Funding 2.
Want to be first in your CLASSE? Investigating Student Engagement in Your Courses Want to be first in your CLASSE? Investigating Student Engagement in.
Learning Analytics and Assessment Analytics and Feed Forward.
Cabrillo College Emerging Scholars Institute Draft Proposal Update Spring 2008.
Chapter 9 – Business Intelligence
Let Them In: Increasing Access, Completion, and Equity in College English.
The Intel Advantage in Education Robert Shults Intel Corporation.
Using CMS Data as a Force for Good? Applying Academic Analytics to Teaching and Learning Leah P. Macfadyen Science Centre for Learning and Teaching, UBC,
Chapter 9 – Classification and Regression Trees
ISLN Network Meeting KEDC SUPERINTENDENT UPDATE. Why we are here--Purpose of ISLN network New academic standards  Deconstruct and disseminate Content.
Learning Analytics Initiative and working in an International community (Apereo Foundation) A.M.Berg, Innovation WG, Community Officer LAI.
Mapworks 101: Understanding Individual Student Transitions Chelsey Knoespel/Joel Hochstein/Kurt Earnest ASC/DOR.
Upgrading from WebCT to Blackboard Learn Upgrade Update April 2011 Technology Facilitated Learning Access and Distributed Learning University of Ulster.
Predicting Earthquakes By Lois Desplat. Why Predict Earthquakes?  To minimize the loss of life and property.  Unfortunately, current techniques do not.
Today Ensemble Methods. Recap of the course. Classifier Fusion
Upgrading from WebCT to Blackboard Learn VLE Upgrade Update April 2011 Technology Facilitated Learning Access and Distributed Learning.
Copyright © 2012, SAS Institute Inc. All rights reserved. ANALYTICS IN BIG DATA ERA ANALYTICS TECHNOLOGY AND ARCHITECTURE TO MANAGE VELOCITY AND VARIETY,
MEAP / MME New Cut Scores Gill Elementary February 2012.
Data Mining BY JEMINI ISLAM. Data Mining Outline: What is data mining? Why use data mining? How does data mining work The process of data mining Tools.
Data Analysis Superintendents Trust. Increase test scores and graduation rates through targeted efforts and investments that lead to student success Proactively.
GETTING YOUR COLLEGE HIRES JOB READY Jillian Payne, Director Analytic Development Program.
Using Data Analytics for School Library Assessment and Improvement Dr. Lesley Farmer California State University Long Beach.
Automatic Discovery and Processing of EEG Cohorts from Clinical Records Mission: Enable comparative research by automatically uncovering clinical knowledge.
The ABLE project: How do we put what we have learnt from learning analytics into practice Tinne De Laet, Head of Tutorial Services, Engineering Science,
Special Challenges With Large Data Mining Projects CAS PREDICTIVE MODELING SEMINAR Beth Fitzgerald ISO October 2006.
Learning Analytics – The Apereo Approach ESUP DAYS & APEREO EUROPE 2016 TUESDAY, FEBRUARY 2, 2016.
Learning UEL: An update on developments Gary Tindell Information Improvement Manager Feb 3rd 2016.
Updated Section 31a Information LITERACY, CAREER/COLLEGE READINESS, MTSS.
Categories Library Analytics Questions Improve individual student performance - user stories from (e..g staff, employers, parents, learners) that generate.
External Review Exit Report Campbell County Schools November 15-18, 2015.
The Neural Engineering Data Consortium Mission: To focus the research community on a progression of research questions and to generate massive data sets.
Making the Learning Analytics Processor (LAP) better, stronger, faster, and more Moodle friendly Lou Harrison Director of Educational Technology Services.
Current R& KE Issues David Sweeney
A Generic Approach to Big Data Alarms Prioritization
OAAI: Deploying an Open Ecosystem for Learning Analytics
Deck Customization Checklist
Developing an early warning system combined with dynamic LMS data
Presented by Khawar Shakeel
(includes online “demo” video)
Learning Analytics: Impact and potential at the Université de Lorraine
Links in the Chain: turning learning analytics data into actions
Student Equity Planning June 9, 2015
Learning Gain: Evaluation, Evidence and Enhancement
NBA Draft Prediction BIT 5534 May 2nd 2018
Sr. Vice President, Student Success
Student Data Analytics
Redesign of OPRE 202: Statistical Data Analysis
Gerd Kortemeyer, William F. Punch
Dr. Morgan C. Wang Department of Statistics
Data Analytics at CNU Dmitriy Shaltayev
Soft Error Detection for Iterative Applications Using Offline Training
Using Data for Improvement
Using Advanced Analytics to Boost Student Success
Analytics: Its More than Just Modeling
Opening Up Learning Analytics: Addressing a Strategic Imperative
Leverage Consensus Partition for Domain-Specific Entity Coreference
Dr. Paul Dosal, Vice President - Student Affairs & Student Success
Split-Block Class Schedule at Yorktown High School
Leveraging Technology to Increase Learning Through Student-Feedback Tools       Leveraging Technology to Increase Learning Through Student-Feedback.
Presentation transcript:

Developing metrics and predictive algorithms for your institution– Marist Story JISC LEARNING ANALYTICS NETWORK EVENT SANDEEP JAYAPRAKASH

Sandeep Jayaprakash LEAD DATA SCIENTIST MARIST COLLEGE & APEREO Twitter

Presentation Overview Open Academic Analytics Initiative (OAAI) ◦Objectives ◦Data extraction & preparation ◦Predictive Models and Results ◦Impact on Student Success ◦Deliver insights to end users

PRACTICAL EXAMPLE: EARLY ALERT SYSTEM Predictive Analytics in Action: Open Academic Learning Analytics Initiative

Open Academic Analytics Initiative EDUCAUSE Next Generation Learning Challenges (NGLC) in United States Funded by Bill and Melinda Gates Foundations $250,000 over a 15 month period Goal: Leverage Big Data and analytics to create an open-source academic early alert system and research “scaling factors”

Input Data Considerations A predictive model is usually as good as its training data Good: ◦Volume - Have lots of data (multiple semesters) ◦Variety - Diverse data ◦Veracity – Data should support the value system Not so good: ◦Data Quality Issues ◦Unbalanced classes (at Marist, 6% of students at risk. Good for the student body, bad for training predictive models )

Learning Analytics preparedness Learning Analytics is inter-disciplinary and coordination among different groups High level buy in from management for a smooth execution Data comes from a wide range of systems Ethics and policy should go hand in hand

Open Academic Analytics Initiative (OAAI) Jisc Data Specification Github link

Predictive Modeling process

Feature Extraction - Data Quality Issues Variability in instructor’s assessment criteria Variability in workload criteria across modules Variability in period used for prediction (early detection) Variability in grading criteria across modules (partial grades with variable contribution)

Data Quality Issues Variability in VLE tools deployment by instructors Variability in usage of tools by students Result - Missing Values and holes in your data Modules Tools

How Do we address them ? Handling Variability - Use ratios and class averages ◦Activity - Percent of usage over Avg percent of usage per course ◦Grades - Effective Weighted Score / Avg Effective Weighted Score Handling missing values ◦Follow an 80 / 20 rule for selection of metrics ◦Perform data imputations to further enrich the quality ◦Build cohort based models to leverage more predictors Sampling - Balance the datasets

Predictors of Student Risk  Some predictors were discarded if not enough data was available.  VLE predictors were measured relative to course averages.

Machine Learning Classifiers Logistic Regression C4.5/C5.0 Boosted Decision Tree Support Vector machines

Predictive Performance of Marist Model

Research Design Models were developed based on Marist data ◦ 85 % accuracy in capturing at-risk students Deployed OAAI system to 2200 students across four institutions ◦Two Community Colleges (FE institutions) ◦Two Historically Black Colleges and Universities (BaME institutions) Design > One instructor teaching 3 sections of the same module ◦One section was control, other 2 were treatment groups Each instructor received an AAR three times during the semester ◦Intervals were 25%, 50% and 75% into the semester

Institutional Profiles

Predictive Model Portability Findings Conclusion 1.Predictive models are more “portable” than anticipated. 2.It is possible to create generic models and the “process can be ported” for use at specific types of institutions. 3.It opens a possibility to explore library of open predictive models and techniques that could be shared across institutions to Scale

Intervention Research Findings - Final Course Grades Analysis showed a statistically significant positive impact on final course grades ◦No difference between treatment groups Saw larger impact in spring than fall Similar trend among low income students

Intervention Research Findings - Content Mastery Student in intervention groups were statistically more likely to “master the content” than those in controls. ◦Content Mastery = Grade of C or better Similar for low income students.

Intervention Research Findings - Withdrawals Students in intervention groups withdrew more frequently than controls Possibly due to students avoiding withdrawal penalties. Consistent with findings from Purdue University

"Not only did this project directly assist my students by guiding students to resources to help them succeed, but as an instructor, it changed my pedagogy; I became more vigilant about reaching out to individual students and providing them with outlets to master necessary skills. P.S. I have to say that this semester, I received the highest volume of unsolicited positive feedback from students, who reported that they felt I provided them exceptional individual attention! Instructor Feedback

JAYAPRAKASH, S. M., MOODY, E. W., LAURÍA, E. J., REGAN, J. R., & BARON, J. D. (2014). EARLY ALERT OF ACADEMICALLY AT-RISK STUDENTS: AN OPEN SOURCE ANALYTICS INITIATIVE. JOURNAL OF LEARNING ANALYTICS, 1(1), 6-47.EARLY ALERT OF ACADEMICALLY AT-RISK STUDENTS: AN OPEN SOURCE ANALYTICS INITIATIVE More Research Findings…

Dashboards – Deliver insights

Learning Activity Radar Chart Short Demo video link

Radar chart - Low Risk pattern example

Early Alert Insights - Risk Quadrant Student Engagement Student Performance High Performance High Engagement Low Performance High Engagement Low Performance Low Engagement High Performance Low Engagement

Future Research ◦Expand our feature set ◦Try to further minimize percentage of false alarms we raise ◦Scalability enhancements leveraging Hadoop/Spark ◦Dynamic modeling capabilities ◦More UX research on building intuitive dashboards

Want the latest updates? Join the mailing list! (subscribe by sending a message to Apereo Learning Analytics Initiative Wiki: GitHub: Sandeep Jayaprakash: