1. An Amazing Day!

2. New Ways of Showing How Good Students Use a CMS
John Fritz, UMBC
January 21, 2009
ELI Annual Conference
John

3. Terms of Use
"Copyright John Fritz, This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author."

4. Overview About UMBC Problem Demo Perspectives to Consider Students
Faculty
Future Plans
Other Examples & Policy Implications
More Information
Q & A
John
While the demo consists of static screen shots, if there’s time at the end, we can go to the live site to view the public reports and myUMBC portal tools.

5. About UMBC
Founded in 1966
“Research extensive university” Carnegie classification
Fall 2007 Stats
12,041 Students
9,464 undergrad, 2,577 grad
1,319 Faculty
636 FT, 683 PT
Selected Brags
One of 50 Best Colleges for Women
1st in undergrad chemistry degrees
awarded to African Americans
Six-time National College Chess
Champions
John

6. About Blackboard @ UMBC
Learning System Version 7.2
As of Fall 2007
11,614 students (96% of all students)
1,074 Bb course sites (includes multi-section courses)
808 Instructors (61% of all instructors)
356 Communities
Includes all student, faculty and staff senates
Support Staff:
2 FTE (Admin & Support)
1 Server Admin
John

7. PROBLEM HOW DO YOU ANSWER?
“So, is Blackboard making a difference?”
Former UMBC Provost Art Johnson in 2002
PROBLEM HOW DO YOU ANSWER?
John

8. Current Reporting in Blackboard
John

9. Course vs. System “Drill Down” View?
John

10. Questions Functional Technical
What is the relationship between Blackboard use and teaching and learning?
What tools can we give users to shed light on (and improve) their own performance within the system?
Technical
How do we query the system without breaking it?
How do we scale and maintain the process?
Bb – Core product?
Community – Building Blocks?
Other?
John

11. New Tools & Approaches
John

12. UMBC Solution: Transparency
Show faculty what peers are doing through publicly available reports of student use.
Bb Sysadmins shouldn’t have the only “birds eye” view.
“Average hits per student” course rankings don’t favor large courses over smaller ones.
Usage alone is no indicator of quality.
But activity by students piques faculty curiosity.
New user tools build on “activity” as an indicator (not a cause) of student success.
We did not have access to “official” grades in SIS, so we studied “unofficial” grades in the CMS.
John

13. DEMO Most Active Blackboard Courses Rankings (Publicly Available)
Self Service Tool: Check My Activity (UMBC Students Only)
Self Service Tool: Grade Distribution Reports (UMBC Faculty Only)
John
DEMO

14. Recent News

15.
John

16. Three Types of Usage John
Student activity is most interesting (to me) because it reflects activity you generated in your course design (I think).

17. Rankings Based on Usage
John

18. Course Tool Usage
John

19. Rankings by Department
John
This confirmed who we support the most (and helps justify some support we get from the IS department) including Mr. Berman the last two years.
Faculty anecdotes:
Katie Morris use of reports before faculty meeting.
Use of reports to ID depts/programs for hybrid program proposal.
Annual user survey results: "if you could pick one prof who uses Bb well, who would it be and why?” Nice correlation between this "qualitative" response and the "quantitative" data the reports show.

20. Rankings in Context
John

21. How Faculty Can Connect
John
Chris Swan’s comment to Karin after we taped her show & tell screencast:
“I didn’t know you were doing that; can we get together for lunch?”
Sweet. 

22. Best Practice: “Show & Tell”
John
Screencast made with camtasia < 5 min.

23. Best Practice: Q&A (iTunesU)
John
iTunesU Q & A Pedagogy interviews:
More “why” than “how”

24. Self Service Tool: Check My Activity (UMBC Students Only)
John
DEMO

25. All Users Start Outside Bb Using myUMBC
John

26. Students See Personal Report of Activity
John

27. Students See Their Summary to Date
John

28. Comparison with Class Summary to Date
John

29. Grade Distribution Reports (UMBC Faculty Only—For Now)
John
DEMO

30. Grade Distribution Reports by Bb Activity
John

31. Per Item Grade Book Activity Analysis
John

32. Sample Report for Final Grades in a Class
John

33. Student Usage Results John
Not much usage by students--yet. About 10 students since March.

34. FA2007 Faculty GDRs (19 courses)
John

35. After we ran this article . . .
John

36. SP2008 Faculty GDRs (28 courses)
John

37. UMBC Bb Reports: GDR of D & F students
Based on voluntary GDRs run by instructors in 61 courses, students earning a D or F tend to use Bb 35 percent less than students earning a C or higher.
FA2008 (13 courses | 40 percent less)
SU2008 (7 courses | 33 percent less)
SP2008 (26 courses | 32 percent less)
FA2007 (15 courses | 36 percent less)
“What is the institution’s ethical obligation of knowing?”- John Campbell, Purdue

38. PERSPECTIVES TO CONSIDER
John
PERSPECTIVES TO CONSIDER

39. Student Perspective
Check My Activity Tool Hasn’t Been Used As Much As Faculty Grade Distribution Report Tool. Why?
May be too buried in myUMBC portal for students to find it
Not yet integrated into Bb under “My Grades” (B2?)
Leery of “Big Brother” or “Academic Profiling”?
See “Colleges Mine Student Data to Predict Dropouts” (5/30/08 Chronicle of Higher Education)
Univ. of Northern Arizona researchers reported student concerns with “intrusive advising” (“Academic Analytics,” Educause Review, Vol. 42, No. 4, July/August, 2007)
Maybe students don’t think this is what a CMS is supposed to do?
John

40. FA2008 SCI100 Findings
How would you describe the CMA’s view of your Bb activity compared to your peers?
28% “I was surprised by what it showed me”
12% “It confirmed what I already knew”
42% “I’d have to use it more to determine its usefulness”
16% “I haven’t used it.”
2% did not respond to this question

41. Faculty Perspective
Concerns about “course ranking” based on average Bb hits per user?
Not a problem at UMBC (so far)
Cheap feedback to students in terms of faculty effort to produce it.
Better if they “publish” grade distribution reports in their courses, so students see how activity informs success.
Can CMS feedback change student motivation? And behavior (performance)?
John

42. FA2008 SCI100 Findings
If your instructor published a GDR for past assignments, would you be more or less inclined to use the CMA before future assignments are due?
54% “More inclined”
10% “Less inclined
36% “Not sure”

43. Online Gradebook: A Symbiotic Relationship?

44. Jeffrey
FUTURE PLANS

45. myUMBC Blackboard “Flyover”
Jeffrey
We will also make a System Wide Bb Announcement on 10th day of classes.

46. What We Want – Building Block or New App
Jeffrey
If students don’t have to go outside the CMS they may use the check my activity tool more.

47. Promote? Test? Expand Scope?
Continue to promote Check My Activity in Portal.
75 percent of all users login to Bb via myUMBC
Add “Check My Activity” on Blackboard “flyover”
Conduct focus groups and/or individual usability tests with students about Check My Activity.
Alert students who may be at risk?
Who decides?
How would this not come across as “intrusive advising”?
Jeffrey

48. OTHER EXAMPLES AND POLICY IMPLICATIONS
Deb
OTHER EXAMPLES AND POLICY IMPLICATIONS

49. Examples of other CMS “Data Mining” projects
5/30/08, Chronicle of Higher Education
Argosy University
Purdue University
Slippery Rock University of Pennsylvania
South Texas College
SUNY Buffalo
Tiffin University
University of Alabama
University of Central Florida
University System of Georgia
Blackboard Greenhouse Grant - Project ASTRO
OSCELOT.org, Advanced System Tracking & Reporting tool
Hofstra University
Deb

50. “Colleges Mine Data to Predict Dropouts”
“At the University System of Georgia, researchers monitored how frequently students viewed discussion posts and content pages on course Web sites for three different courses to find connections between online engagement and academic success. In the graph below, students who were "successful" received an A, B, or C in the class, and students who were "unsuccessful" received a D, F, or an incomplete.”
- 5/30/08 Chronicle of Higher Ed.
Deb
Key (to UMBC) is that CMS usage ALONE is established as an indicator of student success. To date, many academic analytics projects have focused on predictive data models that may have more to do with what students did or where they came from BEFORE stepping foot on campus.

51. MORE INFORMATION John Fritz|
MORE INFORMATION

52. John or Deb
QUESTIONS?

53. Bb Sysadmin/Developer Perspective
No simple feat to query the system without breaking it. We did (twice).
How do we scale and maintain this process, which is necessary so users can learn to use these “self-service” tools?
Jeffrey

54. INFRASTRUCTURE ISSUES
Jeffrey
INFRASTRUCTURE ISSUES

55. How We Query Bb: Static Replica
Blackboard
(Static Replica)
Queries
(PHP Scripts)
Complete copy of database made infrequently, but as needed for reports
Blackboard
(Production)
Jeffrey
Cached Reports

56. Technical Issues to Consider
Pros
Can query current semester data with no impact on production
Data is up-to-date (at time of the static copy)
Cons
Requires a manual process to make the static copy
Need to know in advance when we want to run queries
Will not allow myUMBC What’s New to run as it needs continually up-to-date data
Jeffrey