1 Evidence-based practices in elearning. Collaborative learning in higher education: empirical evidence. Prof. dr. Martin Valcke Hamburg February 4, 2007

2 Structure Collaborative learning without ICT Setting the scene But does it lead to learning? Group characteristics Task characteristics –Scripting –Roles –Tagging Student characteristics & support: peer tutoring Conclusions

3 Collaborative learning: don’t forget « lessons learned » Collaborative learning is part of larger learning environment Adding structure is the key: roles, scripting, tagging Coaching, tutoring, … has an impact Management issues

4 « Collaborative learning is in the air » « Everyone wants it. It is the instructional strategy, perhaps the strategy of the decade »

5 What do we know about collaborative learning without ICT? What does the research say?

6 Collaborative learning without ICT? Meta-analysis collaborative learning research –Slavin (1996) –Johnson & Johnson (1989) “The research has an external validity and a generalizability rarely found in the social sciences.”

7 Collaborative learning without ICT? Consistent and overwhelming positive impact on performance, motivation, social skills, development of metacognition, etc. But, why has it not been implemented to a larger extent?

8

9 Design guidelines 1.Garantee that there are shared learning objectives in a team 2.Build on team responsibility to reach the goals. 3.Build individual responsibility to reach goals. 4.Guarantee equal opportunities in the team activities. 5.Embed a level of competition and/or comparision.

10 Design guidelines 6.Break down larger tasks into subtasks. 7.Take into account individual differences (level, interest, intentions,...). 8.Blend group activities with face-to-face activities. 9.Develop communication skills. 10.Monitor communication processes.

11 Setting the scene University Large groups of 1st year students (N=286) Online learning environment Computer Supported Collaborative Learning (CSCL): part of this environment Course ‘Instructional Sciences’ 35 groups of 8 students working in online groups

12 Integration larger learning environment

13

14

15

16

17 But does this invoke relevant learning? Collaboration does not lead automatically to high quality learning. There is a need guidance and online support in CSCL settings that is comparable to the need of classroom support in face-to-face settings (Lazonder, Wilhelm, & Ootes, 2003).

18 But does this invoke relevant learning? First generation CSCL-research: –Naive use of cooperative learning –Medium orientation –Neglection of context / individual / objectives –Over-estimation of potential technology

19 Does it invoke relevant learning? First generation: –Management problems –No insight into structure of dicsussion –Low task focus (Henri, 1982) –Low levels of cognitive processing: new facts, concepts; hardly theory construction, application, evaluation –Time on task problem –What with students who are not active? –…

20 But does this invoke relevant learning? Second generation CSCL-research: –Focus on “affordances” –Attention paid to “design guidelines”

21 Applying design guidelines 1.Shared learning objectives 2.Team responsibility 3.Individual responsibility 4.Equal opportunities 5.Level of competition or comparision.

22 Applying design guidelines 6.Subtasks. 7.Individual differences 8.Blend group and face-to-face activities 9.Develop communication skills. 10.Monitor communication processes

23 Design guidelines ~ 3 sets of variables Task characteristics Learner characteristics & support Group Characteristics

24 Design guidelines ~ 3 sets of variables Group: –Size –level of interaction Task characteristics: –Nature of task (open, theme) –Roles (content) –Roles (communication) –Tagging –Timing of role assignment Learner: characteristics and support

25 Learning: Nature of dependendent variables Level of interaction Level of knowledge construction Learning performance (test scores) Level of critical thinking Self & group efficacy

26 Group characteristics

27 Group size Differential impact small (8-10), average (11-13, large (15-18)

28 Level of interaction

29 Task structure

30 Roles Pharmacy education 5th year students 5 months internship Lack of integrated pharmaceutical knowledge

31

32 Roles Content roles: –Pharmacyst –Pharmacyst assistant –Theorist –Researcher –Intern Communication roles: –Moderator –Question-asker –Summarizer –Source researcher

33

34 Exchange

35 S. TIMMERS, M. VALCKE*, K. DE MIL & W.R.G. BAEYENS (in press). The Impact of Computer Supported Collaborative Learning on Internship Outcomes of Pharmacy Students. Interactive Learning Environments ICS Integrated Curriculum Score

36 S. TIMMERS, M. VALCKE*, K. DE MIL & W.R.G. BAEYENS (in press). The Impact of Computer Supported Collaborative Learning on Internship Outcomes of Pharmacy Students. Interactive Learning Environments LKC Level knowledge Construction

37 Timing roles 1ste year course “instructional sciences” N discussion groups Transcripts of the entire 12 week discussion period 4 discussion themes of 3 weeks each About 4818 messages or lines of text

38 Timing: introduction roles

39 Timing: introduction roles

40 Roles Starter: start off the discussion, give new impulses every time the discussions slack off Moderator: monitor the discussions, stimulate other students, ask critical questions, inquire for opinions Theoretician: bring in theory, ensure all relevant theoretical concepts are used in the discusion Source searcher: seek external information on the topics, go beyond the scope of course reader Summarizer: post interim summaries, make provisional conclusions, post final summary

41 Moderator

42 Source

43 Gunawardena, Lowe, & Anderson (1997) Level 1: sharing/comparing of information Level 2: the discovery and exploration of dissonance or inconsistency among ideas, concepts or statements Level 3: negotiation of meaning / co-construction of knowledge Level 4: testing and modification of proposed synthesis or co-construction Level 5: agreement statement(s) / applications of newly constructed meaning

44 Timing: introduction roles

45 Timing: introduction roles Role/No-Role condition reaches significantly higher levels of knowledge construction in two themes Even when the role support is cut back

46 Differential impact roles

47 Differential impact roles

48 “There is a differential impact of the different roles” No role condition Starter Moderator Summarizer Theoretician Source Searcher No role Ref.cat. = = Differential impact roles

49 Tagging

50 Tagging

51 Tagging Aims of tagging: –it obliges students to reflect upon the nature of their contribution and on how it will add to the ongoing discussion –the labels improve the outline of the discussion and indicate the predominance or absence of one or more thinking types Example: De Bono’s (1991) thinking hats in view of developing critical thinking

52 Tagging Garrison (1992) identifies five stages of critical thinking: Problem identification Problem definition Problem exploration Problem evaluation/applicability Problem integration

53 De Bono’s (1991) thinking hats Critical ThinkingThinking hats Problem identificationWhite hat Problem definitionBlue hat Problem explorationGreen hat Problem applicabilityBlack hat Problem integrationYellow hat Red hat

54 Tagging 3th-year university students enrolled for the course ‘Instructional Strategies’ (N=35) 6 groups of 6 team members Experimental condition Control condition 4 groups 23 students 2 groups 12 students Tag posts by a thinking hat No tags to posts required

55 Tagging Evidence for critical thinking in both conditions Significant deeper critical thinking in experimental condition (F(1, 416)= ; p<.001)

56 Tagging Patterns are quite similar for both conditions Experimental condition –more focused discussions (F(1, 415)= ; p<.001) –more new info and ideas (F(1, 352)=21.955; p<.001) –more linking facts ideas (F(1, 31)=3.024; p<.092)

57 Impact of tagging Multinomial logistic regressions indicate that being in the experimental condition increases the probability of engaging in in-depth discussions radically (p<.001) experimental students post 2.73 as many messages adding new problem-related information to the discussion (p=.001) experimental students were 2.95 times more likely to add new ideas for discussion (p=.009). linking ideas and critical assessment occur rarely. When it occurs, it is in the experimental condition.

58 Impact of tagging over time Experimental students show a rather constant level of critical thinking Control students show a decrease during problem identification (F(1, 416)= ; p<.001) and exploration (F(1, 415)= ; p<.001)

59 But …. Studies with freshman: no significant impact. Tagging interferes with knowledge construction process. BUT … tutoring helps

60 But … More critical thinking in labeling condition after correction for the different tutor styles –Overall depth of CT –Importance –Discussion of ambiguities –Input of new information –Linking of information –Critical assessment –Defining the problem –Integrating new knowledge

61 Learner characteristics & learner support

62 Support: peer tutors Given critical results of some CSCL-studies, demand for structure: –Scripting (roles, tagging, …) –Facilitators (Bonk, Wisher, & Lee, 2004; Garrison, Anderson, & Archer, 2000; Rickard, 2004; Salmon, 2000) –Prior research, however, revealed that peer tutors were mainly engaged in social support, while less attention was paid to stimulating ‘knowledge construction’ and ‘personal development’ (De Smet, Van Keer, & Valcke, in press) –Therefore extra support for tutors

63 Method Effect study: Impact of labeling on patterns in tutor support. E-moderating model (Salmon, 2000)

64 Peer tutoring Cross-age peer tutoring blended in with online discussion groups One peer clearly takes a supportive role Fourth-year students help freshmen Ratio = 1/10 Open-ended group assignments 2 weeks discussion per theme 1 trial discussion and 4 discussion themes

65 3 tutor training conditions Control (N=39) All-round instructions, No labeling requirements, No pre-service exercises, Focus groups Labeling (N=18) E-moderating instructions, Labelling requirements Pre-service exercises, Focus groups Non-labeling (N=17) E-moderating instructions, No labeling requirements, Pre-service exercises, Focus groups

66 Labeling tutoring activity Labeling involves self-monitoring E-moderating taxonomy (Salmon, 2000) Access and motivation (Step 1) Socialisation (Step 2) Information-exchange (Step 3) Knowledge construction (Step 4) Personal development (Step 5)

67 Impact of labeling on patterns in Emoderating

68 Impact of labeling in tutoring Multinomial logistic regression analysis >> Variables treated as nominal Independent of the training condition, tutors filled all the roles required of e-moderators In each training condition, vast majority for ‘information-exchange’ (step 3)

69 Impact of labeling in tutoring Compared to the control condition, both the labelling and non-labeling condition positively influenced the adoption of tutoring support that stimulates: –‘socialisation’ (step 2) –‘information-exchange’ (step 3) –‘personal development’ (step 5) Labelling enhanced tutors’ facilitation for ‘personal development’ (step 5)

70 Conclusions Collaborative learning: don’t forget « lessons learned » Collaborative learning is part of larger learning environment Adding structure is the key: roles, scripting, tagging Coaching, tutoring, … has an impact Management issues

71 Publications De Smet, M., Van Keer, H., & Valcke, M. (in press). Blending asynchronous discussion groups and peer tutoring in higher education: An exploratory study of online peer tutoring behaviour. Accepted for publication in Computers and Education. De Smet, M., Van Keer, H., & Valcke, M. (in press). Cross-age peer tutors in asynchronous discussion groups: A study of the evolution in tutor support. Accepted for publication in Instructional Science. De Wever, B., Schellens, T.,Valcke, M & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: a review. Computers & Education, 46(1), De Wever, B., Van Keer, H., Schellens, T., & Valcke, M. (in press). Applying multilevel modelling on content analysis data: Methodological issues in the study of the impact of role assignment in asynchronous discussion groups. Accepted for publication in Learning and Instruction. De Wever, B., Van Winckel, M. & Valcke, M. (in press). Discussing patient management online: The impact of roles on knowledge construction for students interning at the paediatric ward. Accepted for publication in Advances in Health Sciences Education. Schellens, T. & Valcke, M. (2005). Collaborative learning in asynchronous discussion groups: What about the impact on cognitive processing? Computers in Human Behavior, 21(6),

72 Publications Schellens, T. & Valcke, M. (2006). Fostering knowledge construction in university students through asynchronous discussion groups. Computers & Education. 46(4), Schellens, T., Van Keer, H. & Valcke, M. (2005). The impact of role assignment on knowledge construction in asynchronous discussion groups: a multilevel analysis. Small Group Research, 36, Schellens, T., Van Keer, H., & Valcke, M. (2007). Learning in asynchronous discussion groups: A multilevel approach to study the influence of student, group and task characteristics. Accepted for publication in Journal of Behavior and Information Technology. 26(1), Schellens, T., Van Keer, H., De Wever, B., Valcke, M. (in press). Tagging Thinking Types in Asynchronous Discussion Groups: Effects on Critical Thinking. Accepted for publication in International Journal of Interactive Learning Environments. Timmers, S., Valcke, M., De Mil, K. & Baeyens, W.R.G. (in press). CSCLE and internships of pharmacy students - The Impact of Computer Supported Collaborative Learning on Internship Outcomes of Pharmacy Students. Accepted for publication in International Journal of Interactive Learning Environments. Valcke, M. & De Wever, B. (2006). Information and communication technologies in higher education: Evidence-based practices in medical education. Medical Teacher, 28,

73 Evidence-based practices in elearning. Collaborative learning in higher education: empirical evidence. Prof. dr. Martin Valcke Hamburg February 4, 2007