Making Course Status Visible: The UCF Online Student Performance Dashboard Thomas Cavanagh, Associate Vice President Center for Distributed Learning UCF

UCF Orlando, FL Metropolitan, suburban university 64,000+ students One of the largest universities in U.S. Carnegie classification: RU/VH Research University: Very High Research Activity 216 degree programs across 11 colleges 11 Campuses throughout Central Florida

Academic Year 2016-2017 42.2% total university SCH online/blended 32.5% fully online SCH 81% of all students took at least one online/blended course 83.7% of all undergraduates 62.8% of all graduate students 72% of all students took at least one fully online course 74.9% of all undergraduates 52.6% of all graduate students

Course Term Year Degree STUDENT SUCCESS Courtesy of Joel Hartman

Predictive Analytics Toolkit UCF-generated analyses Civitas Learning EAB SSC-Campus UCF-developed Student Dashboard Courtesy of Joel Hartman

EAB SSC-Campus

Civitas Illume App

LMS Data Increases Predictive Power Comprehensive Data and Modeling Canvas is a rich data set: >90% of students in Illume have Canvas data 78 of 427 predictors in Illume come from Canvas data Strong Model Prediction: 86% Area Under the Curve LMS Data is Predictive of Persistence at UCF 1 of the top 10 predictors for students overall is their activity on Canvas 2 of top 10 predictors for students taking at least one online course is their Canvas activity Courtesy of Joel Hartman

UCF Student Performance Dashboard https://lti.online.ucf.edu/spd-standalone/

Fall Survey Results (n=96) The majority of respondents were juniors (54%) or seniors (24%), expecting As (41%) or Bs (49%) 41% of students never used the dashboard, but 25% used it daily. Of the very few students (n=28) who offered open ended responses as to why they did not use the dashboard, the overwhelming majority indicated a lack of awareness regarding the dashboard rather than dislike of the dashboard. The dashboard was reported as being easy or very easy to use by 79% of respondents (n=52), with the ‘assignments due soon display’ and ‘current score display’ being the most utilized and most helpful features. The ‘goals display’ was rated as the least helpful feature by 20% of the respondents.

Fall Survey Results (n=96) Most (72%) of the respondents felt the dashboard was somewhat or very accurate, although 15% were unsure. Students expressed concerns over grades being displayed as artificially low during the beginning of the semester as a lack of grades input made it appear as if they were failing. Students liked the accessibility and visualization of the dashboard, especially for keeping up to date with assignments and course performance. 57% of students who used the dashboard believed it motivated them to change their behavior either a little or a great deal in regards to the course.

Questions Thomas Cavanagh, Ph.D. cavanagh@ucf.edu @tbcavanagh