1 Phil Davies School of Computing University of Glamorgan “Super U” The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays

2 Computerised Peer-Assessment  CAP System  Permits students to mark & comment the work of other students.  Also self-assess … used as a standard of expectation  Internet, not Web-based system (developed in Visual Basic / Access)

3

4 Having done the marking, what next?  Students should receive feedback  What feedback? –Marks –Comments  Which is most important?

5

6 AUTOMATICALLY THE MARKER.. ANONYMOUS

7 What should the marker do? Reflect  Look at essay again  Take into account the essay owner’s comments  Further clarification (if it is needed, then is this a ‘black mark’ against the marker?)  Try to ‘appease’ the essay owner?  Modify mark based upon reflection?  Give more feedback

8

9 Must be rewarded for doing the ‘mark for marking’ process  How to judge?  Standard of expectation (self-assessment)  Marking consistency  Commenting, quality, measure against mark  Discussion Element  Need for additional comments – black mark?  Reaction to requests / further clarification

10 Does this student deserve a good mark? ESSAY Compensated Median Mark?? Student Mark Difference W65%67%+2% X48%54%+6% Y53%57%+4% Z61%59%-2% +2% on average

11 Standard of Expectation Self-Assess = 68% Peer-Assessed as = 58% ESSAY Compensated Median Mark Student Mark Difference W65%67% +2% (8) X48%54% +6% (4) Y53%57% +4% (6) Z61%59% -2% (12) X +2% (7.5)

12

13 How easy is it to do?  Statistically fairly easy to create a mark for marking based upon marks  Take into account high and low markers  Standard of expectation  Consistency … judge against final mark awarded for an essay (compensated median)  What about the comments?

14 Feedback Index  Produce an index that reflects the quality of commenting  Produce an average feedback index for an essay  Compare against marker in a similar manner to marks analysis  Where does this feedback index come from and is it valid?

15 The way to get the feedback index?  Develop an application?? –C-Rater?  Spelling Mistakes  Similar Meanings? –That was cool –Really Choc –Really Good Essay  Manually

16

17

18

19

20

21

22

23

24 DAI SMITH ESSAY /10 Differences from actual: F/Index Mark Score ESSAY /10 FINAL MARK AWARDED 5/10 …………. … …. …

25 Time Consuming?  Can we formulate the marking process  Take away need for quantification process of analyzing comments  Is it still peer-assessment if the students are told what to say?

26

27 STUDENT FRED REFERENCES: Positive ……… Negative ……. Personal Valuation 5, 3, 2, 1 3, 1, 2

28 Some points outstanding  What should students do if they identify plagiarism?  Is it ethical to get students to mark the work of their peers?  Is a computerised solution valid for all?  At what age / level can we trust the use of peer assessment?  How do we assess the time required to perform the marking task?

29 The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays  Student Acceptance:  Anonymity / Discussion / High & Low Markers  Tangible Reward  I’m more concerned with rewarding ‘higher order skills’  Getting there