The Role of Peer Review in Supporting the Sustainability of Technology-Enhanced Learning Environments Pantelis M. Papadopoulos, United Nations University Antonio Cerone, United Nations University

2 / 24 Overview Sustainability in Educational Technology Research domain and argument The peer review process Collaboration script Three studies on peer review The role of coercion Free-selection Indirect feedback

3 / 24 Sustainability in Educational Technology Multiple definitions of sustainability in ET Organizational, financial, technical, institutional, etc We focus on pedagogy, using sustainability to refer to Life cycle of a technology-enhanced learning environment (TELE) Ability of a TELE to sustain an active group of students Even with institutional support and financial viability, a TELE may fail, if it does not capture the interest of the students Link between sustainability and student engagement

4 / 24 Research domain and argument Domain Peer review method in computer supported collaborating learning (CSLC) Student engagement can be affected by Individual factors: learning profile, intrinsic motives, etc Learning activity characteristics: peer interaction, scaffolding method, etc Our argument Peer review method can (a) enhance student engagement, (b) support the creation of sustainable a learning community, and consequently support the sustainability of the TELE used by the community Literature already reports multiple benefits from peer review in many levels

5 / 24 The peer review method Four major phases Producing initial work Students work individually/collaboratively to produce the initial draft Assigning reviewers The teacher/system assigns peer work to reviewers Review submission Review comments are sent back to the authors Revision and final version The author revises the initial draft according to reviews and creates the final version

6 / 24 Collaboration script Didactical script guiding students into meaningful learning interactions Even in scripted collaboration, there is a distance between the prescribed task and actual implemented activity External script: the activity as prescribed by the teacher Internal script: the mental representation of the script that the group builds from teacher's prescription Actual script: the actual task and interactions that learners engage into

First Study Peer review and script coercion

8 / 24 First Study: Method Two groups of juniors majoring in Informatics studied in a TELE 20 students in Low Coercion condition 22 students in High Coercion condition We randomly assigned students into same-sex dyads Collaboration script All students worked individually to produce the initial draft Students in dyads reviewed each other's work following a set of guidelines Students had to discuss, reach a consensus, and form a final common answer Low Coercion group: submitting reviews in the TELE was optional High Coercion group: submitting reviews in the TELE was mandatory

9 / 24 First Study: Test results Five phases: pre-test, individual study, collaboration, post-test, interview Pre-test: t-test results showed that the two groups were comparable regarding prior domain knowledge (p>0.05) Post-test: ANCOVA results showed that the High Coercion group significantly outperformed the Low Coercion group (p<0.05) High CoercionLow Coercion MSDnM N Pre-test2.24(0.71)222.13(0.59)20 Post-test7.42(1.30)226.36(0.83)20

10 / 24 First Study: Collaboration patterns Analysis of collaboration patterns based on: (a) student statements in the interviews, (b) comparison of individual and collaborative answers in the TELE, and (c) the TELE log files Collaboration patterns: Ideal, moderate, weak Only 1 out of 10 Low Coercion dyads submitted written reviews in the TELE The others said that they shared review comments during discussion Even if review comments were shared, they would be unstructured and scattered in the discussion Low Coercion group: 2 dyads in the "ideal", 5 dyads in the "moderate", and 3 in the "weak" High Coercion group: 5 dyads in the "ideal", and 6 dyads in the "moderate"

11 / 24 First Study: Conclusions Script coercion can be used by the teacher as a mean to engage students into the activity When left to decide, almost all the students in the Low Coercion group opted-out of the review process as described in the guidelines Students in the High Coercion group demonstrated better collaboration patterns and post-test performance

Second Study The free-selection protocol

13 / 24 Second Study: Method Two groups of sophomores majoring in Informatics and Telecommunications Engineering studied in a TELE 20 students in Assigned-Pairs (pre-defined dyads by the teacher) 22 students in Free-Selection Collaboration script All students worked individually to produce the initial draft Students reviewed each other's drafts through a double-blinded process Students had to review the initial submissions taking into account review comments

14 / 24 Second Study: Review conditions Assigned-Pair Learners in an author-reviewer dyad are assigned exclusively to each other We randomly assigned students into dyads Free-Selection Learners have access to all the submission made by the rest of the class More points of view accessible They can read and review as many submission they like min. at least one review shift to dyad format

15 / 24 Second Study: Test results Five phases: pre-test, study, review & revise, post-test, interview Pre-test: t-test results showed that the two groups were comparable regarding prior domain knowledge (p>0.05) Post-test: ANCOVA results showed that the Free-Selection group significantly outperformed the Assigned-Pair group (p<0.05) Assigned-PairFree-Selection MSDnM N Pre-test2.69(1.07)202.59(0.83)17 Post-test7.71(0.95)208.43(0.81)17

16 / 24 Second Study: Student attitudes Students in the Free-Selection group studied on average 8 and reviewed 2 out of the 16 drafts submitted by their peers Two students did not receive reviews and we asked selected students to provide the missing reviews Interviews: Two main trends in selecting drafts by FS students Find good answers to be able to give nice comments Find weak answers to be able to provide more meaningful reviews Interviews: FS student submit more reviews than asked because: It was a good exercise for them to clarify their thoughts After a while it was easy to do and it would increase the possibility of everyone receiving at least one review ( community culture)

17 / 24 Second Study: Conclusions FS students… Took advantage of the multiple points of view offered in their peers' drafts Engaged deeper in the activity voluntarily far exceeding the minimum effort requirements Explicitly reported a community culture in submitting more reviews Had a more positive opinion about the activity than the AP students, despite the fact that the latter had to work less

Third Study What if there is only indirect feedback?

19 / 24 Third Study: Method Based on the findings of the Second Study Indirect feedback: Students get new insights by reading others' drafts Can the lack of review comments be addressed by indirect feedback and self- review? Two groups of sophomores majoring in Informatics and Telecommunications Engineering studied in a TELE Free-selection protocol 20 students in Self-Review (their drafts were excluded from review and they had to fill in a self-review form before revising their drafts) 18 students in Peer-Review (they all received at least one review) Collaboration script: same as in Second Study

20 / 24 Third Study: Test results Five phases: pre-test, study, review & revise, post-test, interview The two groups were comparable both in pre-test and post-test (p>0.05) Self ReviewPeer Review MSDnM N Pre-test2. 20(1.09)202.04(1.06)18 Post-test8.19(1.37)208.13(1.40)18

21 / 24 Third Study: Student attitudes Same attitudes as in Second Study Students demonstrated strategies that far exceeded the minimum effort requirement Interviews: Same two trends in selecting drafts Find good answers to be able to give nice comments Find weak answers to be able to provide more meaningful reviews

22 / 24 Third Study: Conclusions The study tried to provide evidence that reading other's draft can be equally beneficial to getting peer feedback We supported non-reviewed students with a self-review process and results showed that they were comparable to typical Free-Selection students with peer reviews Since receiving peer review is not the only way to receive feedback, it is easier for a student to participate in a learning community that applies the Free-Selection protocol Even if the student does not get comments from peers, the previously submitted answers can provide valuable feedback Self-review can help the student perform comparative analysis and identify weak points in his own work

23 / 24 General Remarks Peer review process can be an effective tool for structuring the interaction between students When used right, the degree of coercion in scripted collaboration can result to deeper engagement and consequently to better learning outcomes Free-selection protocol allows for more freedom, while it maintains a minimum level of engagement FS protocol has resulted in increased student effort in both studies it was used The community culture of working more to benefit others also appeared in the two studies with FS protocol The indirect feedback in FS makes the protocol even more appropriate for learning communities

24 / 24 Thank you!