1 Phil Davies School of Computing University of Glamorgan “Super U” Peer-Assessment: No marks required, just feedback? Evaluating the Quality of Computerized Peer-Feedback compared with Computerized Peer-Marking

2 Definition of Peer-Assessment?  In describing the teacher.. A tall b******, so he was. A tall thin, mean b******, with a baldy head like a lightbulb. He’d make us mark each other’s work, then for every wrong mark we got, we’d get a thump. That way – he paused – ‘we were implicated in each other’s pain’ McCarthy’s Bar McCarthy’s Bar (Pete McCarthy, 2000,page 68)

3 Why Peer-Assessment  Community of Practice!!  Working together to learn (and be assessed)  Perceptions –Staff > Lecturer getting easy life not marking –Student> Lecturer getting easy life not marking  Student Awareness of Benefits …. IMPORTANT TANGIBILITY  Success dependant upon scalability (computerisation).. “Student Numbers have risen dramatically since 1991 without a concomitant increase in resources” (Pond et al, 1995)

4 Computerised Peer-Assessment  CAP System  Permits students to mark & comment the work of other students. (normally 8)  Also initial self-assess stage (reflection) … used as a standard of expectation  Internet, not Web-based system (developed in Visual Basic / Access)

5

6 Having done the marking, what next?  Students should receive feedback  What feedback? –Marks –Comments  Which is most important? –To students or staff

7

8 AUTOMATICALLY THE MARKER.. ANONYMOUS

9 What should the marker do? Reflect  Look at essay again  Take into account the essay owner’s comments  Further clarification (if it is needed, then is this a ‘black mark’ against the marker?)  Try to ‘appease’ the essay owner?  Modify mark based upon reflection?  Give more feedback

10

11 Must be rewarded for doing the ‘mark for marking’ process.. Based on quality  How to judge?  Standard of expectation (self-assessment)  Marking consistency  Commenting, quality, measure against mark  Discussion Element  Need for additional comments – black mark?  Reaction to requests / further clarification

12 Standard of Expectation Self-Assess = 52% Peer-Assessed as = 58% ESSAY Compensated Median Mark Student Mark Difference(absolute) W65%57% -8% (2) X48%36% -12% (6) Y53%49% -4% (2) Z61%59% -2% (4) -6.5% (3.5)

13

14 How easy to get mark for marking?  Statistically fairly easy to create a mark for marking based upon marks  Take into account high and low markers  Standard of expectation  Consistency … judge against final mark awarded for an essay (compensated median)  What about the comments?

15 Feedback Index  Produce an index that reflects the quality of commenting  Produce an average feedback index for an essay  Compare against marker in a similar manner to marks analysis  Where does this feedback index come from and is it valid?

16 The way to get the feedback index?  Develop an application?? –C-Rater?  Spelling Mistakes  Similar Meanings? –That was cool –Really Choc –Really Good Essay  Manually

17

18 Commonality!!  In the 67 essays that were marked –Only 96 comments –44% positive and 56% negative –Highly critical if something not explained properly (21% of total comments (of which 73% were negative)  Better students were more critical than weaker students.. –Better understanding permitting criticism? –Confidence? –Hostility?  Comments grouped into 10 categories  Need to QUANTIFY these comments.. Feedback index  Create a database holding positive & negative (by category)

19

20

21

22

23

24

25 Time Consuming?  Can we formulate the marking process  Take away need for quantification process of analyzing comments  Is it still peer-assessment if the students are told what to say?

26

27 STUDENT FRED REFERENCES: Positive ……… Negative ……. Personal Valuation 5, 3, 2, 1 3, 1, 2

28 Some points outstanding  What should students do if they identify plagiarism?  Is it ethical to get students to mark the work of their peers?  Is a computerised solution valid for all?  At what age / level can we trust the use of peer assessment?  How do we assess the time required to perform the marking task?

29 Peer-Assessment: No marks required, just feedback?  Not there yet  Feedback index results are very positive  Require more evaluation before totally automated system is possible  Getting there.. Removing subjectivity in marking