Kim Arnold, Purdue John Fritz, UMBC October 12, 2010 Using CMS Activity Data of Good Students to Raise Awareness of Underperforming Peers Kim Arnold, Purdue John Fritz, UMBC October 12, 2010

Overview Introductions Presentations Break UMBC’s “Check My Activity” (CMA) Purdue’s “Signals” Break Sneak Peak: ELI’s “Evidence of Impact” Survey Discussion & Brainstorm Wrap Up

A VISION OF THE FUTURE . . . Winter 2011 Read short “scencario” from beginning of Chapter 9, “Academic Analytics in the Future of Higher Education,” of Phil Goldstein’s 2005 ECAR Publication on Academic Analytics. A VISION OF THE FUTURE . . .

ACADEMIC ANALYTICS: GOLDSTEIN (2005) Stage 1: Extraction and reporting Stage 2: Analysis and monitoring Stage 3: “What-if” decision support (Scenarios) Stage 4: Predictive modeling and simulation Stage 5: Automatic triggers and alerts Stage 1: Extraction and reporting of transaction-level data Stage 2: Analysis and monitoring of operational performance Stage 3: What-if decision support (such as scenario building) Stage 4: Predictive modeling and simulation Stage 5: Automatic triggers of business processes (such as alerts)

GOLDSTEIN’S FUTURE OF ANALYTICS

New Tools & Approaches Typically associated with business and marketing—Amazon uses data mining of other people’s past purchases to suggest books you might be interested in buying next—academic analytics can be used to profile and even predict students who may be at risk, by analyzing demographic and performance data of former students. But Campbell and others stop short of defining the optimal way to present the lessons institutions learn from their data to current students who may need to know them—in the form of an intervention. Citing concerns some students may have about privacy or Big Brother watching them, he even speculates what an institution’s "ethical obligation of knowing" really can or should be.

EXAMPLES OF OTHER CMS “DATA MINING” PROJECTS 5/30/08, Chronicle of Higher Education Argosy University Purdue University Slippery Rock University of Pennsylvania South Texas College SUNY Buffalo Tiffin University University of Alabama University of Central Florida University System of Georgia Blackboard Greenhouse Grant - Project ASTRO OSCELOT.org, Advanced System Tracking & Reporting tool Hofstra University

“COLLEGES MINE DATA TO PREDICT DROPOUTS” “At the University System of Georgia, researchers monitored how frequently students viewed discussion posts and content pages on course Web sites for three different courses to find connections between online engagement and academic success. In the graph below, students who were "successful" received an A, B, or C in the class, and students who were "unsuccessful" received a D, F, or an incomplete.” - 5/30/08 Chronicle of Higher Ed. Key (to UMBC) is that CMS usage ALONE is established as an indicator of student success. To date, many academic analytics projects have focused on predictive data models that may have more to do with what students did or where they came from BEFORE stepping foot on campus.

Educause Center for Applied Research

Online Gradebook: A Symbiotic Relationship?

UMBC “CHECK MY ACTIVITY” Demo UMBC “CHECK MY ACTIVITY”

About UMBC Founded in 1966 “Research extensive university” Carnegie classification Fall 2009 Stats 12,870 Students 9,947 undergrad, 2,923grad 730 Faculty 480 FT, 250 PT Selected Brags #1 “Up and Coming National University” U.S. News America’s Best Colleges, 2010 1st in undergrad chemistry degrees awarded to African Americans One of 50 Best Colleges for Women 7-time National College Chess Champions John

About Blackboard @ UMBC Blackboard Learn 9.1 As of Spring 2010 (began using in SP200) 95% of all students 75% of all instructors 65% of all courses (1,645 FA2010) 356 Communities Includes all student, faculty and staff senates Support Staff: 2 FTE (Admin & Support) 1 Server Admin John

PROBLEM HOW DO YOU ANSWER? “So, is Blackboard making a difference?” Former UMBC Provost Art Johnson in 2002 PROBLEM HOW DO YOU ANSWER? John

Bb System Reporting John

Bb Course Reporting Here’s what I want? But if I want it, maybe others do, too!

Questions Functional Technical What is the relationship between Blackboard use and teaching and learning? What tools can we give users to shed light on (and improve) their own performance within the system? Technical How do we query the system without breaking it? How do we scale and maintain the process? Bb – Core product? Community – Building Blocks? Other? John

www.umbc.edu/blackboard/reports GO TO LIVE DEMO HERE – TIME PERMITTING

Check My Activity Demo http://www.screencast.com/t/jmZzozpPRZiG

Inside Blackboard

Inside myUMBC Campus Portal

Inside myUMBC Campus Portal

Bb Activity by Grade Distribution

Bb Activity by Grade Distribution

Bb Activity of D & F students Based on voluntary participation by instructors in 131 courses, students earning a D or F tend to use Bb 39 percent less than students earning higher grades. SP2010 21 courses | 37 percent less FA2009 29 courses | 37 percent less SP2009 11 courses | 47 percent less FA2008 13 courses | 40 percent less SU2008 7 courses | 33 percent less SP2008 26 courses | 32 percent less FA2007 15 courses | 36 percent less Does the pattern hold true during the semester? What if students knew this information sooner? NEW: iStrategy analysis of 1,461 SP2010 courses shows D&F students used Bb 47% less than students earning C or higher. We did not have access to “official” grades in SIS, so we studied “unofficial” grades in the CMS.

FA2008 SCI100 Findings: Utility How would you describe the CMA’s view of your Bb activity compared to your peers? 28% “I was surprised by what it showed me” 12% “It confirmed what I already knew” 42% “I’d have to use it more to see” 16% “I haven’t used it.” 2% did not respond to this question

FA2008 SCI100 Findings: Interest If your instructor published a GDR for past assignments, would you be more or less inclined to use the CMA before future assignments are due? 54% “More inclined” 10% “Less inclined 36% “Not sure”

CMA Student Usage

CMA Users (6/1/10 to 9/8/10) 3.0 and above (34 percent) 2.0 to 2.99 (16 percent) 1.0 to 1.99 (4 percent) .99 and below (only 7 students or less than 1 percent) No GPA – entering freshmen & transfers (46 percent) Interestingly, of the 4 percent earning a 2.0 or lower, nearly a dozen students have used the CMA more than 4 times over two or more 24-hour periods.

LESSONS LEARNED

How We Query Bb: Static Replica Blackboard (Static Replica) Queries (PHP Scripts) Complete copy of database made daily at 1 a.m. for reports Blackboard (Production) Jeffrey Cached Reports www.umbc.edu/blackboard/reports

Code Download & Video Show & Tell Available at www.umbc.edu/blackboard/reports Video Show & Tell “Walkthrough” (same as above)

Next Steps: Functional Quantitative Expand the sample of UMBC courses being studied. Study the demographic backgrounds of students. Qualitative Why do UMBC students use the CMA? Why do they return? Literature Review CMS activity as predictor vs. indicator of success. How do others use the CMS for intervention?

WE CAN’T MAKE THEM DRINK Institutions can't “absolve” students from “at least partial responsibility for their own education. To do so denies both the right of the individual to refuse education and the right of the institution to be selective in its judgments as to who should be further educated. More importantly, it runs counter to the essential notion that effective education requires that individuals take responsibility for their own learning” (p. 144) Vincent Tinto, Leaving College: Rethinking the causes and cures of student attrition (1993)

SOCIAL COGNITIVE THEORY - BANDURA

SELF-REGULATED LEARNING: ZIMMERMAN Observational: students learn to distinguish the major features of a model's skill or strategy. Emulative: a learner's performance enactive approximates the general form of a model's skill or strategy. Self-control: students can perform a skill or strategy based on mental representations of a model's performance. Self-regulation: learners can adapt their skills and strategies systematically as personal and contextual conditions change.

Next Steps: Technical Parse student data by "academic plan” Parse all courses by graduate/undergraduate Show # of active Bb courses vs. total courses. Show Bb activity by final GDR for any tool. Display daily update of Most Active Bb courses Display daily update of student CMA use Correlate other bio/demo data in iStrategy

Next Steps: Technical (cont.) What questions are we trying to answer? Many of the questions involve linking to data already in iStrategy (plan, race, sex, gpa, etc.) How do we link the data? Blackboard course id is not the same as the PeopleSoft course id Need a way to crosswalk between the two Doing manually now

Next Steps: Technical (cont.) from istrategy2.iPSSA.Custom.ReportFactRegistration rfr left outer join istrategy2.iPSSA.Student.DimTerm dt on dt.TermKey = rfr.TermKey left join istrategy2.iPSSA.Custom.ReportFactStudentTerm rfst on rfr.EmployeeID = rfst.EmployeeID and rfr.AcademicCareer = rfst.AcademicCareer and rfr.TermSourceKey = rfst.TermSourceKey left outer join istrategy2.iPSSA.CustomSource.PS_UM_LDAP_XWALK x on x.EMPLID = rfr.StudentSourceKey left outer join BbStatsSource.COURSE_MAIN cm on rfr.TermSourceKey = SUBSTRING(RIGHT(cm.COURSE_ID, 6), 3, 1) + SUBSTRING(RIGHT(cm.COURSE_ID, 6), 5, 2) + CASE LEFT(RIGHT(cm.COURSE_ID, 6), 2) WHEN 'FA' THEN '8' WHEN 'SP' THEN '2' WHEN 'SU' THEN '6' WHEN 'WI' THEN '0' ELSE '-' END and cast(rfr.ClassNumber as varchar) = LEFT(RIGHT(cm.COURSE_ID, 11), 4) left outer join BbStatsSource.ACTIVITY_ACCUMULATOR aa on aa.COURSE_PK1 = cm.PK1 and aa.USER_PK1 = (select u.PK1 from BbStatsSource.USERS u where u.USER_ID = x.USER_OPRID) and aa.EVENT_TYPE = 'COURSE_ACCESS'

Next Steps: Technical (cont.) Initial thought was to add the Course or Class Number dimension Won’t work – Blackboard course id is term specific Typical Blackboard course id – MATH251_6428_FA2010 Last six characters are the term 6428 is the PeopleSoft CLASS_NBR which is regenerated each term

Next Steps: Technical (cont.) A new dimension will probably be needed Term based Hierarchy to include all associated PeopleSoft courses/classes Include cross-listed (combined section) classes Attach to Registration and ClassSchedule fact tables

Next Steps: Technical (cont.) Need to determine additional dimension and fact tables around Blackboard data Need to determine reporting needs – Reporting Services, ProClarity, etc.

Selected References Campbell, J.P., DeBlois, P.B. & Oblinger, D.G. (2007, July/August) Academic analytics: A new tool for a new era. EDUCAUSE Review, 424): pp. 41-57. Retrieved March 3, 2009 from http://connect.educause.edu/Library/EDUCAUSE+Review/AcademicAnalyticsANewTool/44594 Rampell, C. (2008). Colleges Mine Data to Predict Dropouts. The Chronicle of Higher Education, 5/30/08. Retrieved March 6, 2009 from http://chronicle.com/weekly/v54/i38/38a00103.htm#web-course (login required) Young, J. (2009). College 2.0: A wired way to rate professors—and connect teachers. The Chronicle of Higher Education, January 8, 2009. Retrieved April 23, 2009 from http://chronicle.com/free/2009/01/9311n.htm

Educational Technology Framework Exploratory Supported Strategic Mission Critical Transformative Transition 4 Institutional User Growth Transition 3 Transition 2 http://www.gilfuseducationgroup.com/gilfus-model-the-educational-technology-framework This was work done by Blackboard in the early 2000's.  Key contributors (in this order) were: Steve Gilfus, Peter Van Tienen, Todd Gibby, and Jim Hermens. The Academic Technology framework is used to describe institutional growth of academic technologies. The different phases describe growth, maturity, relative importance and positioning within an institution. Each transition initiates a change in thought, related activities as well as investment in the associated technology in order to move on to the next phase. It’s our experience that the most critical timing is related to the actual transitions that exist between phases. Educational Organizations must meet certain requirements in order to move from one phase of the Educational Technology Framework to the other. Organizations should not skip a phase. If an organization does, the phase they tend to skip most frequently is the Strategic phase, putting them in a “Mission Critical without a Mission” situation. Transition 1 Phase I Phase 2 Phase 3 Phase 4 Phase 5 Time

Thanks fritz@umbc.edu www.umbc.edu/blackboard/reports Questions? Comments Thanks fritz@umbc.edu www.umbc.edu/blackboard/reports

BREAK

RESOURCES

"Project ASTRO" Blackboard Greenhouse Grant Eric Kunnen Coordinator of Instructional Technologies Grand Rapids Community College ekunnen@grcc.edu Santo Nucifora Manager of Systems Development and Innovation Seneca College santo.nucifora@senecac.on.ca

STARFISH EARLY ALERT Identify & Detect Intervene & Track Manual Flags Automatic Flags Attendance Intervene & Track Instructor Advisor Groups of Courses and Students Improve & Retain Student Communication and 360 Close Loop More info: http://www.starfishsolutions.com Identify = Detect It costs more to recruit students that it does to retain them.

STARFISH EXAMPLE AUTOMATIC FLAGS BASED ON BLACKBOARD GRADEBOOK/COURSE ACCESS Administrators can set up flags to be raised that are auto-generated. Flags can be raised by the system by grades and average scores and specific gradebook columns in Blackboard. Flags can also be raised based on students’ access to their courses in Blackboard. Additional customization is available through API’s.

SIGNALS: THE BIG kAHUNA Show the following: Signals Demo http://www.itap.purdue.edu/tlt/signals/signals_final/index.htm NBC Nightly News Clip http://www.msnbc.msn.com/id/3032619/vp/32634348#32634348

MORE INFORMATION Project ASTRO http://projects.oscelot.org/gf/project/astro/ Starfish Early Alert Project Site http://www.starfishsolutions.com UMBC’s Blackboard Reports & CMA http://www.umbc.edu/blackboard/reports iStrategy Solutions http://www.istrategysolutions.com Purdue University Signals Project Site http://www.itap.purdue.edu/tlt/signals/

BREAK

PURDUE “SIGNALS” PROJECT Demo PURDUE “SIGNALS” PROJECT

EVIDENCE OF IMPACT SURVEY RESULTS Sneak Peek: EDUCAUSE Learning Initiative EVIDENCE OF IMPACT SURVEY RESULTS

BRAINSTORM

IF WE COULD START FROM SCRATCH . . . What would we want to know from or about our CMS that we don't now? What would we do with this information to alert or intervene with students? How would we determine if the interventions were successful? If successful, how might these interventions change instructor course design or pedagogy?

RESOURCES