How to run “successful” online discussion boards: Some observations from the coalface Jim Waters The iSchool at Drexel

 Jim Waters, Drexel University, 2009 But Asynchronous Discussion Boards are old hat ! Surely Chat is faster and more interactive ? Reflection – knowledge building discourse is not a speed sport Richer peer interactions

 Jim Waters, Drexel University, 2009 Gauging Success Participation Discourse Knowledge Building Consensus Understanding If you cannot measure it, you cannot improve it - Lord Kelvin

 Jim Waters, Drexel University, 2009 Participation Posting messages – how many ? All messages are not equal “ I agree with Fred” “ I Like cats” “ I see you are an eagles fan” “ yeah it’s week 10 !” “ LOL” “ It is nice to see you again” My understanding of participation is not yours Thread length – The K-Tel approach, 25 original hits unrelated messages Think better not more often !

 Jim Waters, Drexel University, 2009 Discourse Student-to-Student interactions Student-faculty interactions Apple for teacher ? Appetite for knowledge ? Deep threads Most participants in a thread Iterations – most dyads Interaction types

 Jim Waters, Drexel University, 2009 Knowledge Building Is any knowledge really new ? Schrödinger's cat Is an inference new knowledge ? Content Analysis Least measured outcome

 Jim Waters, Drexel University, 2009 Consensus “So we are agreed, we will land the Cuban exiles in the Bay of Pigs “ “A mode of thinking that people engage in when they are deeply involved in a cohesive in-group, when the members' strivings for unanimity override their motivation to realistically appraise alternative courses of action – Irving L. Janis, Victims of Groupthink, 1972

 Jim Waters, Drexel University, 2009 Understanding The Turing test ? Eliza Interpretation Inference Exemplification Abstraction

 Jim Waters, Drexel University, 2009 Small Group Discussion Cohesive groups Less cliquey behavior Non-participation is obvious Manageable size for creative project Mixed ability The strong leader An army of followers The dysfunctional group

 Jim Waters, Drexel University, 2009 Research Context  Analyzed online interactions in graduate classes in information systems and information science  Course interactions via discussion board on Blackboard learning system.  15 online courses over a five year period  325 Students  21,000 messages  4000 message threads

 Jim Waters, Drexel University, 2009 Research Method  Collected and analyzed student messages posted to discussion board and small group discussion  Analyzed Thread depth, thread length Cognitive content of message Patterns of message sequences  Student interactions related to outcomes

 Jim Waters, Drexel University, 2009 Findings I  Are all participants created equal ?  Dealing with the Divas ?  Rewarding the loudest ?  Leave-alone or interact ? Harangue Encourage Flatter The dark side of rewards  Questions or self-questions ?  Question design ?  Small group reflections ?  In what context Charlie ?  How many questions ?  Grading discussion board interactions, does it motivate ?

 Jim Waters, Drexel University, 2009 Findings II  Cognitive Outcomes  Do Social posts mean less cognitive outcomes  How many social posts is acceptable  Dyads, cliques and committees – what level of group interaction is most productive ?  How to deal with cliques ?  Breaking up the cartels ?  How truthful to be in feedback/moderation

 Jim Waters, Drexel University, 2009 Participatory democracy vs. benevolent oligarchy  Will an active core of students bring in peripheral participants or exclude them ?  How do we get peripheral participants to join the circle ?  Does it matter if some do not actively participate ? I meant to ask that but…… Vicarious learners or lurkers ?  Should we moderate core participants dominating the discussion or encourage them ?

 Jim Waters, Drexel University, 2009 Moderated vs. Laissez-faire Case Study III  Two sections of an IS course delivered at the same time – same basic syllabus  Same number of students (25)  Selected six “identical” questions on each section  Different Instructor approach  Heavy moderation vs. lightweight moderation

 Jim Waters, Drexel University, 2009 Question Heavy ModerationLow Moderation Systems Analyst as problem solver 6974 Agile methods 9697 Project design Requirements Analysis 9683 Fact Finding 8590 Data Modeling Practice Average Questions and Approach (messages)

 Jim Waters, Drexel University, 2009 Systems Analyst as Problem solver Heavy ModerationLow Moderation Total Messages 6974 Instructor – student messages 17(24%)0 Deep sub-threads ( 4 levels or greater) 46 Student messages 5274 Student-instructor messages 25 Student-student messages 2752

 Jim Waters, Drexel University, 2009 Agile methodsHeavy ModerationLow Moderation Total Messages 9697 Instructor – student messages 16(17%)0 Deep sub-threads ( 4 levels or greater) 108 Student messages 8097 Student-instructor messages 5224 Student-student messages 2873 Deep thread messages (students) 6544

 Jim Waters, Drexel University, 2009 Agile methodsHeavy ModerationLow Moderation Total Messages 9597 Deep sub-threads ( 4 levels or greater) 28 Deep threads without instructor intervention

 Jim Waters, Drexel University, 2009 Project designHeavy ModerationLow Moderation Total Messages Instructor – student messages 44(30%)0 Deep Threads 189 Student messages Student-instructor messages 5323 Student-student messages 5374 Deep thread messages (students) 7846

 Jim Waters, Drexel University, 2009 Goals objectives and scopeHeavy ModerationLow Moderation Total Messages Deep sub-threads ( 4 levels or greater) 39 Deep threads without instructor intervention

 Jim Waters, Drexel University, 2009 So does providing feedback help ? ModeratedUn-moderated Agile methods (Words) Agile methods (Messages) 8097 Project design (Words) Project Design (Messages) SA problem solver (Words) SA problem solver (messages)5274 Tot messages Tot words Average words/student message What about question design ?

 Jim Waters, Drexel University, 2009 Posing questions - general  Sometimes a committed, motivated and interested group with decent moderation will be inert.  Even within the same niche of the same domain some questions just work better than others  Good question design is not trivial even for domain experts  Does the question connect to student experience, real or vicarious  Is the question relevant to the course  Does the question represent a well-structured single knowledge domain goal

 Jim Waters, Drexel University, 2009 Good, bad or average? I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [cooking up a new project] Critically evaluate the author's FAST approach. Is it useful? Practical? What are some alternatives? Is this a "real" model that could be used on "real" projects? [Fast or Slow] I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [fact-finding]

 Jim Waters, Drexel University, 2009 And the Winner is I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [fact-finding] Critically evaluate the author's FAST approach. Is it useful? Practical? What are some alternatives? Is this a "real" model that could be used on "real" projects? [Fast or Slow] I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [cooking up a new project]

 Jim Waters, Drexel University, 2009 Good I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [cooking up a new project] 150 posts (2 nd best thread) Several sub-threads extremely deep (7 or 8 levels) Question is level 1 30% of messages were Instructor to Student 29% of messages were student to instructor Critique, feedback, support and facilitation Well-placed faculty moderation, nudges rather than cattle prods Well-bounded but open-ended problem, students define problem Initial high level of misunderstanding of task (goals vs. objectives) despite material having been formally covered already Students negotiated the meaning of the task collaboratively Deliberate pitching as cooperative

 Jim Waters, Drexel University, 2009 Average I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [fact-finding] 85 posts (Mean was 91) Moderate sub-thread depth (mostly 3 or 4 levels) 31% were messages from Instructor to students 20% were messages from students to instructor Well-placed faculty moderation, focus on challenging assumptions. Reasonably open-ended problem Far less cooperative inter-student activity Not pitched as a cooperative activity Students not answering a common question, but question is defined

 Jim Waters, Drexel University, 2009 Bad Critically evaluate the author's FAST approach. Is it useful? Practical? What are some alternatives? Is this a "real" model that could be used on "real" projects? [Fast or Slow] 46 posts Limited sub-thread depth - mostly 2 (question then single response) 45% were messages from Instructor to students 37% were messages from students to instructor 18% were student-student messages Faculty intervention much more critical (didactic) Five questions in one, one open-ended 4 bounded Very little cooperative inter-student activity Not pitched as a cooperative activity One overarching common question

 Jim Waters, Drexel University, 2009 Final Words: Value of online discussion this was so helpful because often I was struggling with the same thing so I could learn from their errors and gain new information from the answers to their questions I was moved to comment on how refreshing the lack of competition in the Communications for the online classes seemed to me. It was a discussion and a sharing of experiences Honestly, in the second half of the course, I have felt like I must be a pariah. Apart from the professor, I can't get anyone to respond to my posts- a very lonely feeling. I have posted to the the weekly board with little feedback No question that the on line discussion was critical to getting me through the class. There were mostly questions about how to.. I've never done this before. I felt lost and inexperienced most of the time. I have no real full time work experience and I felt I had nothing much to contribute and compared to the rest of the posts mine would feel really insignificant.

 Jim Waters, Drexel University, 2009 Worrying or not ? I felt lost and inexperienced most of the time. I have no real full time work experience and I felt I had nothing much to contribute and compared to the rest of the posts mine would feel really insignificant. Honestly, in the second half of the course, I have felt like I must be a pariah. Apart from the professor, I can't get anyone to respond to my posts- a very lonely feeling. I have posted to the the weekly board with little feedback Stopped posting after week 2 Little opportunity to get drawn in What do you do if someone will not contribute ? Concerned about how posts interpreted Most posts were responded to Most posts positive feedback but no added content Many posts Social rather than task-oriented Rarely posted thread starters

 Jim Waters, Drexel University, 2009 Conclusions I  Tentative findings  Deep engagement can be encouraged by thought-leaders  Useful thought-leader behaviors highly context dependent  Interaction behaviors interact  Faculty feedback improves some forms of engagement but perhaps at the expense of student-student interactions ?  Finite tank of engagement ?

 Jim Waters, Drexel University, 2009 Conclusions II  Question design can be crucial to engagement  Discussion needs to be framed as collaborative not competitive  Students recognize core participants, should we alter our behaviors to fit ?  Committed engagement is possible  So is isolation and inertia !  Sense of community is not automatic

 Jim Waters, Drexel University, 2009 Related References  Waters, J. 'Social Network Behavior, Thought-Leaders and Knowledge Building In An Online Learning Community', Proceedings of Hawaii Intl. Conference on System Sciences (HICSS-41), Knowledge Management Track, Jan Social Network Behavior, Thought-Leaders and Knowledge Building In An Online Learning Community  Gasson, S. and Waters, J. “How (not) to construct ALN course questions that encourage student participation in peer collaboration and knowledge construction,” 40th Hawaii International Conference on System Sciences, Hawaii, January 2007.How (not) to construct ALN course questions that encourage student participation in peer collaboration and knowledge construction  Waters, J., and Gasson, S. "Social Engagement in an Online Community of Inquiry," 27th International Conference on Information Systems (ICIS), Milwaukee WI, 2006.Social Engagement in an Online Community of Inquiry  Waters, J. “Determinants of Engagement in an Online Community of Inquiry,” The 12th Sloan-C International Conference on Online Learning, November 2006, of Engagement in an Online Community of Inquiry  Waters, J., and Gasson, S. "Strategies Employed By Participants In Virtual Learning Communities," Hawaii Intl. Conference on System Sciences (HICSS-38), Collaboration Systems and Technology track, IEEE Software Society, Manua, Hawaii, January 2005, 2005, p. 3b.Strategies Employed By Participants In Virtual Learning Communities  A full list of publications, with full copies of articles, is available at

 Jim Waters, Drexel University, 2009 About Me  I am a Doctoral Candidate at the iSchool at Drexel University, Philadelphia (USA), graduating Summer  My principal research interests lie in Online Collaborative Knowledge Building, Technology-Supported Learning, Student Role-Behavior in Online Learning Communities and HCI. I am currently employed ( ) as a Research Assistant on an IMLS funded project "Toward a Model Curriculum for the Management of Digital Information". Online Collaborative Knowledge Building, Technology-Supported Learning, Student Role-Behavior in Online Learning CommunitiesToward a Model Curriculum for the Management of Digital Information  I received a BA in Psychology at Warwick University, UK (1979), an MSc in Occupational Psychology at the University of Hertfordshire, UK (1991) and a MS in Information Systems at the College of IS&T at Drexel University (2002). Prior to my advanced academic studies I enjoyed a substantial career in Systems Design, Management and IS Consultancy. 