www.umbc.edu/blackboard/reports (includes online “demo” video) Leveraging CMS Usage Data to Raise Awareness of Underperforming Students John Fritz www.umbc.edu/blackboard/reports (includes online “demo” video) Problem Methods Results Conclusions Recently, more institutions have pursued "academic analytics” to shed light on and improve student success. Typically associated with business and marketing—Amazon uses data mining of other people's past purchases to suggest books you might be interested in buying next—academic analytics can be used to profile and even predict students who may be at risk, by comparing their pre-college demographics and performance with past students. However, it is still not clear how best to present the lessons institutions learn from their analytics data to current students who may need to know them-in the form of an intervention. How do we lead them to the water they should drink without raising concerns about privacy or Big Brother watching them? Why shouldn’t they think they are the exceptions to the rules our data models suggest? As John Campbell, a leader in academic analytics has said, what is an institution's "ethical obligation of knowing”? UMBC’s Blackboard Activity Reports Similar to other institutions, UMBC has determined that a relationship may exist between student success as defined by grades and activity in the campus' online course management system (CMS). Specifically, over two academic years (2007-09), UMBC’s “Most Active Blackboard Courses” reports show students earning a D or F in 81 courses used the CMS about 35 percent less than students earning a grade of C or higher. SU2009 (9 courses | 42 percent less) SP2009 (11 Courses | 47 percent less) FA2008 (13 courses | 40 percent less) SU2008 (7 courses | 33 percent less) SP2008 (26 courses | 32 percent less) FA2007 (15 courses | 36 percent less) Key Questions While the sample of courses needs to be expanded and studied further, these student CMS activity data patterns raise some intriguing questions: Does this usage pattern hold true throughout the semester? If so, how might students’ awareness, motivation and performance change if they could know this information sooner? To answer these questions we have created the following “self service” tools students can use to compare their own activity against an anonymous summary of their peers: Check My Activity (CMA) allows students to see how they use the CMS compared to other students in the same course. Grade Distribution Report (GDR) shows students their grade and CMS activity compared to other students in the same course – provided the instructor enters a grade in the grade book. The purpose of these tools is to provide early and frequent system feedback directly to students, so they are the first to know if and how their CMS activity might be an indicator of their engagement in a course. During Fall 2008 and Spring 2009, we studied how students used the CMA and GDR tools in SCI100, a 200-student lab science course for non-majors that is equally enrolled across all class standings (e.g., freshmen to seniors). In addition to confirming a national study showing that students value checking their grades more than any other CMS function, we found the following: 28 percent of SCI100 students who used the CMA tool were “surprised” by their own data. 54 percent said they would be “more inclined” to use the CMA before future assignments are due if they had access to a GDR showing Bb activity for past assignments. Student Use of the CMA & GDR Tools While SCI100 student feedback has been encouraging, adoption by the larger student body has been slow since the tools were announced in Spring 2008, and reported in the student newspaper in Spring 2009. To help, a series of Fall 2009 Blackboard system-wide announcements encouraged students to use the CMA & GDR tools (below). The SCI100 studies and FA2009 Bb System Announcements suggest that students who are not proactively introduced to the CMA and GDR tools may not understand how or why they would use them. Also, since these tools are not part of the delivered Blackboard software, they are not easy to find. They are available as “self service” reports on our main Blackboard Reports site at www.umbc.edu/blackboard/reports. Future Plans Make the CMA & GDR tools easier to find and use by developing a My Activity “Building Block” or direct links from our myUMBC portal (used by 90 percent of undergraduates to access Blackboard) Find out why students return to the CMA & GDR tools and what, if anything, their use can mean for changing student awareness, behavior and academic performance. Encourage other institutions to download, install and test UMBC’s CMA & GDR code. Also, we will collaborate with others who are doing similar work (e.g., Project ASTRO).