Weighing Costs versus Benefits: Evaluating the Impact From Implementing a Course Management System Richard E. West University of Georgia Greg Waddoups, Meghan Kennedy, and Charles R. Graham Brigham Young University

Technology improves everything … right? Calvin and Hobbes by Bill Watterson CMS

Research Questions How has the implementation of a CMS impacted teaching/learning at the university? What are the benefits/challenges from supporting this tool on an institution-wide basis?

Research Context Average university enrollment each semester is 30,000 students 8,000 academic courses for both fall and winter semesters in 2004 There were slightly more than 4,000 Blackboard courses created during each semester Over 300,000 students and faculty are estimated to have used Blackboard at least once

Methods Open-ended/closed-ended survey questions (n=124 instructors; 163 students) administered to students and instructors over two semesters An analysis of the calls reported to the Blackboard support center on campus (n=1,341) Semi-structured interviews with instructors (n=53) and brief intercept interviews with students (n=17) Data was analyzed inductively, looking for patterns within major categories

Findings — Satisfaction

Findings: Satisfaction

Conclusions: Satisfaction When Blackboard works without technical difficulties, most students and instructors at our university reported being satisfied with the tool because it is convenient for them to use, easy to learn, and helpful in their studies. Satisfaction and usage is high for only four features (announcements, , posting of course documents, and gradebook). Satisfaction is low for synchronous chatting tools, discussion boards, assessments, and the digital dropbox

Findings — Stability

Conclusions: Stability Blackboard has been unstable in some of its most critical areas Stability is more than a matter of convenience. It impacts instructor/student relationships and the instructional effectiveness of activities –Case Study A

Findings — Efficiency

Conclusions: Efficiency Using a CMS seems to improve efficiency somewhat for instructors and students, especially in 4 ways: –distributing class materials –updating documents electronically –keeping the instructor and course organized –providing quicker and more frequent feedback to students

Conclusions: Efficiency “The potential for me is that I can change it once in one place and not have to pass out 30 paper.” “It allows (and makes) professors be more organized. For the class that doesn't use Blackboard, I am often confused at what material I should know, assignments, etc.” “It saves time at the end of the semester so you don't get a whole bunch of students saying, ‘C+, I thought I had a B.’”

However, some efficiency traps… Homework creep Unrealistic expectations Instructor reliability Encouragement to miss class Using the entire CMS tool for one small feature

Findings — Learning

Conclusions: Learning Many instructors feel the CMS either did nothing or improved their instruction Most instructors who felt it improved instruction were using it to support teacher-centered instruction –Access to materials –Improved notetaking during lectures –Just-in-time preparation for teaching –Saving in-class time for other activities

Conclusions: Teacher-centered learning “I think one of the biggest things that helps the learning thing is having my lectures online so they can pull them down and review or else they can get it if they miss the class, and that's huge.” “I think it's allowed a little more freedom in the classroom for students to ask questions and not feel they're going to miss something while they're writing it down.” “It's interesting because I can get through a lecture now 30% faster because I don't have to write it all out on the board.... I can get through the material so I can get to discussion and application, which is really nice.”

Findings: Student-centered learning Many of the instructors who do not find Blackboard to be a helpful tool are those instructors who prefer more hands-on, exploratory, or student-centered approaches to instruction. –Case Vignette 2 These instructors often feel that Blackboard’s benefits do not outweigh the costs associated with using the tool These instructors are typically seeking out alternatives to Blackboard to fulfill their online instructional needs, such as blogs, wikis, personal websites, and instant messenging services

Why? All feature controls are open only to the teacher –Examples: Discussions, grading, posting of documents, no outside guests, creating/adding modules Blackboard is not easily adaptable –“It is not flexible”

Overall Conclusions Do the benefits of a CMS outweigh its disadvantages? –It depends! What type of instruction do you want to encourage? Are you satisfied with only four features, or will it only be worth the price tag if every feature is being used? Caution: are you getting “married” to the tool? –Even if they only use one or two features, faculty can grow dependent on the tool, making a switch very difficult.

Future Questions Can a CMS be built to support constructivist, student-centered, problem-based, or other types of pedagogies? How can we help stakeholders properly evaluate a technology for its pedagogical strengths before purchasing it? If we have already purchased one that is not meeting our needs, how can we more easily maneuver towards other tools without isolating the instructors who now feel tied to the tool?

Case Studies “ It looks like it works great, but there are glitches. ” “ It's really cumbersome. ”