1. Kim Arnold, Purdue John Fritz, UMBC October 12, 2010
Using CMS Activity Data of Good Students to Raise Awareness of Underperforming Peers
Kim Arnold, Purdue
John Fritz, UMBC
October 12, 2010

2. Overview Introductions Presentations Break
UMBC’s “Check My Activity” (CMA)
Purdue’s “Signals”
Break
Sneak Peak: ELI’s “Evidence of Impact” Survey
Discussion & Brainstorm
Wrap Up

3. A VISION OF THE FUTURE . . . Winter 2011
Read short “scencario” from beginning of Chapter 9, “Academic Analytics in the Future of Higher Education,” of Phil Goldstein’s 2005 ECAR Publication on Academic Analytics.
A VISION OF THE FUTURE . . .

4. ACADEMIC ANALYTICS: GOLDSTEIN (2005)
Stage 1: Extraction and reporting
Stage 2: Analysis and monitoring
Stage 3: “What-if” decision support (Scenarios)
Stage 4: Predictive modeling and simulation
Stage 5: Automatic triggers and alerts
Stage 1: Extraction and reporting of transaction-level data
Stage 2: Analysis and monitoring of operational performance
Stage 3: What-if decision support (such as scenario building)
Stage 4: Predictive modeling and simulation
Stage 5: Automatic triggers of business processes (such as alerts)

5. GOLDSTEIN’S FUTURE OF ANALYTICS

6. New Tools & Approaches
Typically associated with business and marketing—Amazon uses data mining of other people’s past purchases to suggest books you might be interested in buying next—academic analytics can be used to profile and even predict students who may be at risk, by analyzing demographic and performance data of former students.
But Campbell and others stop short of defining the optimal way to present the lessons institutions learn from their data to current students who may need to know them—in the form of an intervention. Citing concerns some students may have about privacy or Big Brother watching them, he even speculates what an institution’s "ethical obligation of knowing" really can or should be.

7. EXAMPLES OF OTHER CMS “DATA MINING” PROJECTS
5/30/08, Chronicle of Higher Education
Argosy University
Purdue University
Slippery Rock University of Pennsylvania
South Texas College
SUNY Buffalo
Tiffin University
University of Alabama
University of Central Florida
University System of Georgia
Blackboard Greenhouse Grant - Project ASTRO
OSCELOT.org, Advanced System Tracking & Reporting tool
Hofstra University

8. “COLLEGES MINE DATA TO PREDICT DROPOUTS”
“At the University System of Georgia, researchers monitored how frequently students viewed discussion posts and content pages on course Web sites for three different courses to find connections between online engagement and academic success. In the graph below, students who were "successful" received an A, B, or C in the class, and students who were "unsuccessful" received a D, F, or an incomplete.”
- 5/30/08 Chronicle of Higher Ed.
Key (to UMBC) is that CMS usage ALONE is established as an indicator of student success. To date, many academic analytics projects have focused on predictive data models that may have more to do with what students did or where they came from BEFORE stepping foot on campus.

9. Educause Center for Applied Research

10. Online Gradebook: A Symbiotic Relationship?

13. UMBC “CHECK MY ACTIVITY”
Demo
UMBC “CHECK MY ACTIVITY”

14. About UMBC
Founded in 1966
“Research extensive university” Carnegie classification
Fall 2009 Stats
12,870 Students
9,947 undergrad, 2,923grad
730 Faculty
480 FT, 250 PT
Selected Brags
#1 “Up and Coming National University” U.S. News America’s Best Colleges, 2010
1st in undergrad chemistry degrees
awarded to African Americans
One of 50 Best Colleges for Women
7-time National College Chess
Champions
John

15. About Blackboard @ UMBC
Blackboard Learn 9.1
As of Spring 2010 (began using in SP200)
95% of all students
75% of all instructors
65% of all courses (1,645 FA2010)
356 Communities
Includes all student, faculty and staff senates
Support Staff:
2 FTE (Admin & Support)
1 Server Admin
John

16. PROBLEM HOW DO YOU ANSWER?
“So, is Blackboard making a difference?”
Former UMBC Provost Art Johnson in 2002
PROBLEM HOW DO YOU ANSWER?
John

17. Bb System Reporting
John

18. Bb Course Reporting Here’s what I want?
But if I want it, maybe others do, too!

19. Questions Functional Technical
What is the relationship between Blackboard use and teaching and learning?
What tools can we give users to shed light on (and improve) their own performance within the system?
Technical
How do we query the system without breaking it?
How do we scale and maintain the process?
Bb – Core product?
Community – Building Blocks?
Other?
John

20.
GO TO LIVE DEMO HERE – TIME PERMITTING

21. Check My Activity Demo

22. Inside Blackboard

23. Inside myUMBC Campus Portal

26. Inside myUMBC Campus Portal

27. Bb Activity by Grade Distribution

28. Bb Activity by Grade Distribution

29. Bb Activity of D & F students
Based on voluntary participation by instructors in 131 courses, students earning a D or F tend to use Bb 39 percent less than students earning higher grades.
SP courses | 37 percent less
FA courses | 37 percent less
SP courses | 47 percent less
FA courses | 40 percent less
SU courses | 33 percent less
SP courses | 32 percent less
FA courses | 36 percent less
Does the pattern hold true during the semester?
What if students knew this information sooner?
NEW: iStrategy analysis of 1,461 SP2010 courses shows D&F students used Bb 47% less than students earning C or higher.
We did not have access to “official” grades in SIS, so we studied “unofficial” grades in the CMS.

30. FA2008 SCI100 Findings: Utility
How would you describe the CMA’s view of your Bb activity compared to your peers?
28% “I was surprised by what it showed me”
12% “It confirmed what I already knew”
42% “I’d have to use it more to see”
16% “I haven’t used it.”
2% did not respond to this question

31. FA2008 SCI100 Findings: Interest
If your instructor published a GDR for past assignments, would you be more or less inclined to use the CMA before future assignments are due?
54% “More inclined”
10% “Less inclined
36% “Not sure”

32. CMA Student Usage

33. CMA Users (6/1/10 to 9/8/10) 3.0 and above (34 percent)
2.0 to 2.99 (16 percent)
1.0 to 1.99 (4 percent)
.99 and below (only 7 students or less than 1 percent)
No GPA – entering freshmen & transfers (46 percent)
Interestingly, of the 4 percent earning a 2.0 or lower, nearly a dozen students have used the CMA more than 4 times over two or more 24-hour periods.

34. LESSONS LEARNED

35. How We Query Bb: Static Replica
Blackboard
(Static Replica)
Queries
(PHP Scripts)
Complete copy of database made daily at 1 a.m. for reports
Blackboard
(Production)
Jeffrey
Cached Reports

36. Code Download & Video Show & Tell
Available at
Video Show & Tell “Walkthrough” (same as above)

37. Next Steps: Functional
Quantitative
Expand the sample of UMBC courses being studied.
Study the demographic backgrounds of students.
Qualitative
Why do UMBC students use the CMA?
Why do they return?
Literature Review
CMS activity as predictor vs. indicator of success.
How do others use the CMS for intervention?

38. WE CAN’T MAKE THEM DRINK
Institutions can't “absolve” students from “at least partial responsibility for their own education. To do so denies both the right of the individual to refuse education and the right of the institution to be selective in its judgments as to who should be further educated. More importantly, it runs counter to the essential notion that effective education requires that individuals take responsibility for their own learning” (p. 144)
Vincent Tinto, Leaving College: Rethinking the causes and cures of student attrition (1993)

39. SOCIAL COGNITIVE THEORY - BANDURA

40. SELF-REGULATED LEARNING: ZIMMERMAN
Observational: students learn to distinguish the major features of a model's skill or strategy.
Emulative: a learner's performance enactive approximates the general form of a model's skill or strategy.
Self-control: students can perform a skill or strategy based on mental representations of a model's performance.
Self-regulation: learners can adapt their skills and strategies systematically as personal and contextual conditions change.

41. Next Steps: Technical Parse student data by "academic plan”
Parse all courses by graduate/undergraduate
Show # of active Bb courses vs. total courses.
Show Bb activity by final GDR for any tool.
Display daily update of Most Active Bb courses
Display daily update of student CMA use
Correlate other bio/demo data in iStrategy

42. Next Steps: Technical (cont.)
What questions are we trying to answer?
Many of the questions involve linking to data already in iStrategy (plan, race, sex, gpa, etc.)
How do we link the data?
Blackboard course id is not the same as the PeopleSoft course id
Need a way to crosswalk between the two
Doing manually now

43. Next Steps: Technical (cont.)
from istrategy2.iPSSA.Custom.ReportFactRegistration rfr
left outer join istrategy2.iPSSA.Student.DimTerm dt on
dt.TermKey = rfr.TermKey
left join istrategy2.iPSSA.Custom.ReportFactStudentTerm rfst on
rfr.EmployeeID = rfst.EmployeeID and
rfr.AcademicCareer = rfst.AcademicCareer and
rfr.TermSourceKey = rfst.TermSourceKey
left outer join istrategy2.iPSSA.CustomSource.PS_UM_LDAP_XWALK x on
x.EMPLID = rfr.StudentSourceKey
left outer join BbStatsSource.COURSE_MAIN cm on
rfr.TermSourceKey = SUBSTRING(RIGHT(cm.COURSE_ID, 6), 3, 1) + SUBSTRING(RIGHT(cm.COURSE_ID, 6), 5, 2) + CASE LEFT(RIGHT(cm.COURSE_ID, 6), 2) WHEN 'FA' THEN '8' WHEN 'SP' THEN '2' WHEN 'SU' THEN '6' WHEN 'WI' THEN '0' ELSE '-' END and
cast(rfr.ClassNumber as varchar) = LEFT(RIGHT(cm.COURSE_ID, 11), 4)
left outer join BbStatsSource.ACTIVITY_ACCUMULATOR aa on
aa.COURSE_PK1 = cm.PK1 and
aa.USER_PK1 = (select u.PK1 from BbStatsSource.USERS u where u.USER_ID = x.USER_OPRID) and
aa.EVENT_TYPE = 'COURSE_ACCESS'

44. Next Steps: Technical (cont.)
Initial thought was to add the Course or Class Number dimension
Won’t work – Blackboard course id is term specific
Typical Blackboard course id – MATH251_6428_FA2010
Last six characters are the term
6428 is the PeopleSoft CLASS_NBR which is regenerated each term

45. Next Steps: Technical (cont.)
A new dimension will probably be needed
Term based
Hierarchy to include all associated PeopleSoft courses/classes
Include cross-listed (combined section) classes
Attach to Registration and ClassSchedule fact tables

46. Next Steps: Technical (cont.)
Need to determine additional dimension and fact tables around Blackboard data
Need to determine reporting needs – Reporting Services, ProClarity, etc.

47. Selected References
Campbell, J.P., DeBlois, P.B. & Oblinger, D.G. (2007, July/August) Academic analytics: A new tool for a new era. EDUCAUSE Review, 424): pp Retrieved March 3, 2009 from
Rampell, C. (2008). Colleges Mine Data to Predict Dropouts. The Chronicle of Higher Education, 5/30/08. Retrieved March 6, 2009 from (login required)
Young, J. (2009). College 2.0: A wired way to rate professors—and connect teachers. The Chronicle of Higher Education, January 8, Retrieved April 23, 2009 from

48. Educational Technology Framework
Exploratory
Supported
Strategic
Mission Critical
Transformative
Transition 4
Institutional User Growth
Transition 3
Transition 2
This was work done by Blackboard in the early 2000's.  Key contributors (in this order) were: Steve Gilfus, Peter Van Tienen, Todd Gibby, and Jim Hermens.
The Academic Technology framework is used to describe institutional growth of academic technologies. The different phases describe growth, maturity, relative importance and positioning within an institution. Each transition initiates a change in thought, related activities as well as investment in the associated technology in order to move on to the next phase. It’s our experience that the most critical timing is related to the actual transitions that exist between phases. Educational Organizations must meet certain requirements in order to move from one phase of the Educational Technology Framework to the other. Organizations should not skip a phase. If an organization does, the phase they tend to skip most frequently is the Strategic phase, putting them in a “Mission Critical without a Mission” situation.
Transition 1
Phase I
Phase 2
Phase 3
Phase 4
Phase 5
Time

49. Thanks fritz@umbc.edu www.umbc.edu/blackboard/reports
Questions? Comments
Thanks

50. BREAK

51. RESOURCES

52. "Project ASTRO" Blackboard Greenhouse Grant
Eric Kunnen
Coordinator of Instructional Technologies
Grand Rapids Community College
Santo Nucifora
Manager of Systems Development and Innovation
Seneca College

53. STARFISH EARLY ALERT Identify & Detect Intervene & Track
Manual Flags
Automatic Flags
Attendance
Intervene & Track
Instructor
Advisor
Groups of Courses and Students
Improve & Retain
Student Communication and 360 Close Loop
More info:
Identify = Detect
It costs more to recruit students that it does to retain them.

54. STARFISH EXAMPLE AUTOMATIC FLAGS BASED ON BLACKBOARD GRADEBOOK/COURSE ACCESS
Administrators can set up flags to be raised that are auto-generated. Flags can be raised by the system by grades and average scores and specific gradebook columns in Blackboard. Flags can also be raised based on students’ access to their courses in Blackboard. Additional customization is available through API’s.

56. SIGNALS: THE BIG kAHUNA
Show the following:
Signals Demo
NBC Nightly News Clip

57. MORE INFORMATION
Project ASTRO
Starfish Early Alert Project Site
UMBC’s Blackboard Reports & CMA
iStrategy Solutions
Purdue University Signals Project Site

58. BREAK

59. PURDUE “SIGNALS” PROJECT
Demo
PURDUE “SIGNALS” PROJECT

60. EVIDENCE OF IMPACT SURVEY RESULTS
Sneak Peek: EDUCAUSE Learning Initiative
EVIDENCE OF IMPACT SURVEY RESULTS

61. BRAINSTORM

62. IF WE COULD START FROM SCRATCH . . .
What would we want to know from or about our CMS that we don't now?
What would we do with this information to alert or intervene with students?
How would we determine if the interventions were successful?
If successful, how might these interventions change instructor course design or pedagogy?

63. RESOURCES