Engaging Students in Distance Learning Jim Waters The iSchool at Drexel

 Jim Waters, Susan Gasson Drexel University, 2009 If students have already paid us, why should we care if they are engaged ?  Altruism – the long view?  Pragmatism – they can always leave taking their tuition money with them  Student feedback and tenure decisions?  Word gets out! – social networks Students can vote with their virtual feet and Their very real wallets

 Jim Waters, Susan Gasson Drexel University, 2009 Agenda  How do I know if students are engaged?  What is the effect of: Question design? Course scaffolding? Instructor moderation? Being an entertaining instructor?  So what?

 Jim Waters, Susan Gasson Drexel University, 2009 Research Study  Analyzed course interactions via discussion board on Blackboard learning system. 12 online MS courses (info. systems./info. Science) 313 Students, 11,497 messages Posts to discussion board + small group discussions  Analyzed Thread depth, thread length, participants Cognitive content of message Interactive intent of message Patterns of message sequences  Examined student outcomes related to interaction Pre and Post questionnaires Demographics and Attitudinal data

 Jim Waters, Susan Gasson Drexel University, 2009 Three modes of learning engagement  Individual Participation  Active Course Involvement  Iterative Social Engagement  Fluid: students can move between modes reacting to drivers

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 1: Individual Participation  The semi-transparent participant Interacts with materials Internalizes knowledge Contractual obligation postings Broadcast messages Superficial learning Hermit!

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 2: Active Course Involvement  Demonstrates (some) genuine interest Interacts with peers (after a fashion) Translates community knowledge Relates posts to own experience or knowledge Internalizes community knowledge Ego-centric approach Small group or clique interactions

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 3: Interactive Social Engagement  Motivated for interactive learning Committed to greater group learning Interacts freely with peers Looped learning cycles Iterative internalizations/externalizations Social construction of knowledge

 Jim Waters, Susan Gasson Drexel University, 2009 How can we tell what is going on?

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 1: Course Participation How Do We Know A Project Is On Track? Instructor 10/21/07 2:36 PM RE:How Do We Know A Project Is On Track? S1810/25/07 12:09 PM RE:How Do We Know A Project Is On Track? S10 10/25/07 6:29 PM RE:How Do We Know A Project Is On Track?S21 10/25/07 8:30 PM RE:How Do We Know A Project Is On Track? S17 10/26/07 7:38 PM RE:How Do We Know A Project Is On Track? S22 10/28/07 6:19 PM RE:How Do We Know A Project Is On Track? S2 10/28/07 7:04 PM RE:How Do We Know A Project Is On Track?S4 10/28/07 10:24 PM RE:How Do We Know A Project Is On Track? S10 10/28/07 10:26 PM RE:How Do We Know A Project Is On Track? S7 10/28/07 10:46 PM RE:How Do We Know A Project Is On Track? S8 10/29/07 12:59 AM RE:How Do We Know A Project Is On Track? S11 12/14/07 11:34 AM

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 1: Course Participation

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 2: Active Course Involvement Unanswered questions Instructor 10/5/07 3:23 PM RE:Unanswered questions S1 10/5/07 6:59 PM RE:Unanswered questions S17 10/6/07 3:41 AM RE:RE:Unanswered questions S19 10/8/07 12:33 AM RE:RE:RE:Unanswered questions S20 10/8/07 10:52 AM RE:Unanswered questions S13 10/6/07 10:52 AM RE:RE:Unanswered questions S18 10/7/07 4:14 PM RE:RE:RE:Unanswered questions S6 10/9/07 9:45 PM RE:Unanswered questions S12 10/6/07 11:04 AM RE:RE:Unanswered questions S20 10/7/07 10:34 AM RE:Unanswered questions S9 10/7/07 6:49 AM RE:Unanswered questions S21 10/7/07 4:36 PM RE:Unanswered questions S10 10/7/07 5:31 PM RE:Unanswered questions S4 10/7/07 10:59 PM RE:Unanswered questions S19 10/8/07 12:07 AM RE:Unanswered questions S8 10/8/07 2:21 PM BOK as a communications/marketing tool S12 10/9/07 1:18 PM RE:Unanswered questions S8 10/8/07 2:48 PM RE:Unanswered questions S14 10/12/07 2:33 PM RE:RE:Unanswered questions S12 10/12/07 3:22 PM RE:Unanswered questions S15 10/12/07 3:26

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 2: Active Course Involvement

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 3: Interactive Social Engagement

 Jim Waters, Susan Gasson Drexel University, 2009 Mode 3: Interactive Social Engagement

 Jim Waters, Susan Gasson Drexel University, 2009 What makes students engage at higher levels in a distance- learning course?

 Jim Waters, Susan Gasson Drexel University, 2009 Question Frequency vs. response

 Jim Waters, Susan Gasson Drexel University, 2009 Question Design I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [Cooking up a new project] Critically evaluate the author's FAST approach. Is it useful? Practical? What are some alternatives? Is this a "real" model that could be used on "real" projects? [Fast or slow] I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [Fact-finding]

 Jim Waters, Susan Gasson Drexel University, 2009 And the Winner is I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [Fact-finding] Critically evaluate the author's FAST approach. Is it useful? Practical? What are some alternatives? Is this a "real" model that could be used on "real" projects? [Fast or slow] I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [Cooking up a new project]

 Jim Waters, Susan Gasson Drexel University, 2009 Good I want you to cook up a systems development project (real or imagined). Describe the goal(s), the objective(s) of the project and the scope of the work the systems analyst for the project. Post your goals, objectives and scope by around Thursday of this week. I'd then like each of you to comment a bit on each other's work. [Cooking up a new project] 150 posts Several sub-threads extremely deep (7 or 8 levels) Critique, feedback, support and facilitation Well-placed faculty moderation, nudges rather than cattle prods  Well-bounded but open-ended: students define problem  Deliberately pitched as a cooperative task  Concrete (well-defined) task  Students negotiate the task meaning collaboratively

 Jim Waters, Susan Gasson Drexel University, 2009 Average I would like each of you to initially focus on one fact finding technique, your contribution should be a critical (but brief) examination of that technique within the domain of systems analysis. [Fact-finding] 85 posts Moderate sub-thread depth (mostly 3 or 4 levels) 31% were messages from Instructor to students 20% were messages from students to Instructor Well-placed faculty moderation, focus on challenging assumptions.  Reasonably open-ended problem  Far less cooperative inter-student activity  Not pitched as a cooperative activity  Students not answering a common question, but question is defined

 Jim Waters, Susan Gasson Drexel University, 2009 Bad Critically evaluate the author's FAST approach. Is it useful? Practical? What are some alternatives? Is this a "real" model that could be used on "real" projects? [Fast or slow] 46 posts Limited sub-thread depth - mostly 2 (question then single response) 45% were messages from Instructor to students 37% were messages from students to Instructor 18% were student-student messages  Faculty intervention much more critical (didactic)  Five questions in one: 1 was open-ended 4 bounded  Very little cooperative inter-student activity  Not pitched as a cooperative activity

 Jim Waters, Susan Gasson Drexel University, 2009 Good questions tended to be  Early rather than later First question in the week Early weeks better than later weeks  Open (scope), but bounded (problem structure) Permitted students to call upon their personal experience with IT or organizations Permitted many ways to approach the issues  Negotiated rather than defined Permitted collaborative interpretation Allowed students to contribute by defining their own take on the question.  Relevance to students helps – war stories, company policies and approaches

 Jim Waters, Susan Gasson Drexel University, 2009 Bad questions tended to be  Following a highly-interactive question or later in the course Later questions much less interactive and constructive across courses than earlier questions  Cognitively complex Containing multiple parts that needed to be considered in turn, or Overly abstract, so students could not draw on their personal experience.  Socially isolating Fewer opportunities for interpretation and collaboration in answering the question.

 Jim Waters, Susan Gasson Drexel University, 2009 How does course scaffolding affect student engagement?

 Jim Waters, Susan Gasson Drexel University, 2009 Scaffolding  Something to hang construction on  Solid foundation for task Materials Discussion Support for knowledgeable peers to contribute  Task requires an extension of prior abilities  Provides a structure on which students can build knowledge  Task must be just beyond current unaided skills but doable with help

 Jim Waters, Susan Gasson Drexel University, 2009 Poor Scaffolding You've been asked to read the ALA Code of Ethics plus two other codes of ethics of your choice. What did you learn from this process? Did any common themes or concerns tend to emerge? What did you relate to in the ALA Code of Ethics? Were there things that seemed problematic, or that you disagreed with? Codes of Ethics Instructor 1/28/08 3:15 PM RE: Codes of Ethics S5 1/31/08 7:00 AM RE: Codes of Ethics S15 1/31/08 5:49 PM RE: Codes of Ethics S13 1/31/08 9:17 PM RE: Codes of Ethics S14 2/1/08 12:05 AM RE: Codes of Ethics S16 2/1/08 12:54 PM RE: Codes of Ethics S9 2/1/08 1:08 PM RE: Codes of Ethics S17 2/3/08 1:20 PM RE: Codes of Ethics S11 2/2/08 3:26 PM RE: Codes of Ethics S18 2/3/08 4:23 PM RE: Codes of Ethics S19 2/4/08 5:33 PM RE: Codes of Ethics S6 2/5/08 6:24 PM RE: Codes of Ethics S1 2/5/08 11:03 PM

 Jim Waters, Susan Gasson Drexel University, 2009 Better Scaffolding Can ethical behavior really be codified by a professional organization? Can ethical behavior be enforced? How?

 Jim Waters, Susan Gasson Drexel University, 2009 Unscaffolded  Learning materials A long list of codes of ethics web-pages Three abstract ethics articles  A body of solid material but this did not directly relate to the posted question or give a framework for answering the question

 Jim Waters, Susan Gasson Drexel University, 2009 Scaffolded  Learning materials A description of ethical models Worksheet for ethical decision making  Actions and consequences  Responsibilities and obligations  A theoretical and pragmatic platform from which discourse could be built Three sparse pages of bullet-points  Materials contained less information but provided a structure for thinking.

 Jim Waters, Susan Gasson Drexel University, 2009 How does instructor moderation affect student engagement?

 Jim Waters, Susan Gasson Drexel University, 2009 Moderated or leave-alone ?  Two sections of an IS course delivered at the same time – same basic syllabus  ~Same number of students (23/24)  Selected six “identical” questions on each section  Different Instructor approach  Heavy moderation vs. lightweight moderation

 Jim Waters, Susan Gasson Drexel University, 2009 Question Heavy Moderation Low Moderation Systems Analyst as problem solver 6974 Agile methods 9697 Project design Requirements Analysis 9683 Fact Finding 8590 Data Modeling Practice Average Tot messages Tot words Average words/student message Questions and Approach (messages)

 Jim Waters, Susan Gasson Drexel University, 2009 Agile methodsHeavy ModerationLow Moderation Total Messages 9697 Instructor – student messages 16 (17%) 0 Student messages 8097 Student-student messages 2873 Student-instructor messages 5224 Deep thread messages (students) 6544 Deep sub-threads ( 4 levels or greater) 108 Deep sub-threads w/o instructor intervention 28

 Jim Waters, Susan Gasson Drexel University, 2009 Instructor interaction and student posts

 Jim Waters, Susan Gasson Drexel University, 2009 Are students more engaged when the Professor is entertaining?

 Jim Waters, Susan Gasson Drexel University, 2009 Deconstructing the Entertaining Professor  Highly knowledgeable industry professional Very Popular Instructor High level of interaction with students Regular internet chat presence Projects personality into discussion  High percentage of social interactions Voluntarily discusses hobbies, weather, music, Disneyworld, cooking, children, Dickens, vintage cars, pets, gardening, insects, Star Wars, birds, Nintendo, Scrabble, foreign films, beer …. Injects lots of jokes

 Jim Waters, Susan Gasson Drexel University, 2009 Student participation StudentsDiscussion Board Visits PostsTopic Threads Posts/ Thread Student Posts Student Posts/ Thread Formal Questions Student Posts/ Question Average Thread Depth per question

 Jim Waters, Susan Gasson Drexel University, 2009 Never mind the width, feel the quality  Threads show evaluation and analysis  Some hypothesis formation  Fairly advanced cognitive activity  Some student-student interaction But, some chaff among the wheat 5% Un-focused Anecdotes 32% fluff posts: “LOL,” “Awesome [dead rock star] story!” “OMG,” “Pictures of gardening implements,” “Lawyer Jokes” 50% contribute to learning 33% student-Instructor messages Still, pretty successful overall High student satisfaction Grades were comparable with Prof. Serious

 Jim Waters, Susan Gasson Drexel University, 2009 Introducing Professor Serious  Same course – Same Syllabus  Highly knowledgeable industry professional Very skilled Instructor Low level of direct interaction with students Strong Topic-focus Little social interaction

 Jim Waters, Susan Gasson Drexel University, 2009 Student participation StudentsDiscussion Board Visits PostsTopic Threads Posts/ Thread Student Posts Student Posts/ Thread Formal Questions Student Posts/ Question Average Thread Depth per question

 Jim Waters, Susan Gasson Drexel University, 2009 Quality ?  More collaborative learning  Messages longer and more detailed  Fairly advanced cognitive activity  Much stronger student-student interaction Stronger awareness of value of peer interactions So what? High task-Focus 2.5% fluff posts 80% substantive knowledge-building posts 15% student-Instructor messages Successful overall Moderate student satisfaction Grades were comparable with Prof. Entertaining

 Jim Waters, Susan Gasson Drexel University, 2009 The tale of the tape  Instructor participation of Prof. Entertaining inflates thread depth slightly More posts, but smaller % focus on knowledge-building  Student-student posts more productive than student-instructor posts  Overall productive (knowledge building) activity was about the same for the two Professors  Student satisfaction slightly lower for Prof. Serious than Prof. Entertaining Fewer posts, but greater % focus on knowledge-building  But Stronger thought leaders for Prof. Serious Explicit kudos for peers in Prof. Serious

 Jim Waters, Susan Gasson Drexel University, 2009 Conclusions  Question design can be crucial to engagement Discussion needs to be framed as collaborative not competitive Relevance to students helps – war stories, company policies and approaches No payoff for frequency of questions  Course scaffolding aids engagement Focus and framing of questions Must support task in concrete manner

 Jim Waters, Susan Gasson Drexel University, 2009 Conclusions  Successful course moderation hinges on quality not quantity or frequency Knowing when to intervene Does not mean “being absent”  Being entertaining is not essential for success Tradeoff between popularity with students and peer-engagement by students May shift focus from peers to instructor Excessive interaction is a lot of work

 Jim Waters, Susan Gasson Drexel University, 2009 Students think they are learning more from peer-interaction this was so helpful because often I was struggling with the same thing so I could learn from their errors and gain new information from the answers to their questions I was moved to comment on how refreshing the lack of competition in the communications for the online classes seemed to me. It was a discussion and a sharing of experiences Honestly, in the second half of the course, I have felt like I must be a pariah. Apart from the professor, I can't get anyone to respond to my posts- a very lonely feeling. I have posted to the the weekly board with little feedback No question that the on line discussion was critical to getting me through the class. There were mostly questions about how to.. I've never done this before. I felt lost and inexperienced most of the time. I have no real full time work experience and I felt I had nothing much to contribute and compared to the rest of the posts mine would feel really insignificant.

 Jim Waters, Susan Gasson Drexel University, 2009 Related References  Waters, J. 'Social Network Behavior, Thought-Leaders and Knowledge Building In An Online Learning Community', Proceedings of Hawaii Intl. Conference on System Sciences (HICSS-41), Knowledge Management Track, Jan Social Network Behavior, Thought-Leaders and Knowledge Building In An Online Learning Community  Gasson, S. and Waters, J. “How (not) to construct ALN course questions that encourage student participation in peer collaboration and knowledge construction,” 40th Hawaii International Conference on System Sciences, Hawaii, January 2007.How (not) to construct ALN course questions that encourage student participation in peer collaboration and knowledge construction  Waters, J., and Gasson, S. "Social Engagement in an Online Community of Inquiry," 27th International Conference on Information Systems (ICIS), Milwaukee WI, 2006.Social Engagement in an Online Community of Inquiry  Waters, J. “Determinants of Engagement in an Online Community of Inquiry,” The 12th Sloan-C International Conference on Online Learning, November 2006, of Engagement in an Online Community of Inquiry  Waters, J., and Gasson, S. "Strategies Employed By Participants In Virtual Learning Communities," Hawaii Intl. Conference on System Sciences (HICSS-38), Collaboration Systems and Technology track, IEEE Software Society, Hawaii, January 2005, p. 3b.Strategies Employed By Participants In Virtual Learning Communities  A full list of publications, with full copies of articles, is available at