1. 1 Phil Davies School of Computing University of Glamorgan “Super U” The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays

2. 2 Computerised Peer-Assessment  CAP System  Permits students to mark & comment the work of other students.  Also self-assess … used as a standard of expectation  Internet, not Web-based system (developed in Visual Basic / Access)

3. 3

4. 4 Having done the marking, what next?  Students should receive feedback  What feedback? –Marks –Comments  Which is most important?

5. 5

6. 6 AUTOMATICALLY EMAIL THE MARKER.. ANONYMOUS

7. 7 What should the marker do? Reflect  Look at essay again  Take into account the essay owner’s comments  Further clarification (if it is needed, then is this a ‘black mark’ against the marker?)  Try to ‘appease’ the essay owner?  Modify mark based upon reflection?  Give more feedback

8. 8

9. 9 Must be rewarded for doing the ‘mark for marking’ process  How to judge?  Standard of expectation (self-assessment)  Marking consistency  Commenting, quality, measure against mark  Discussion Element  Need for additional comments – black mark?  Reaction to requests / further clarification

10. 10 Does this student deserve a good mark? ESSAY Compensated Median Mark?? Student Mark Difference W65%67%+2% X48%54%+6% Y53%57%+4% Z61%59%-2% +2% on average

11. 11 Standard of Expectation Self-Assess = 68% Peer-Assessed as = 58% ESSAY Compensated Median Mark Student Mark Difference W65%67% +2% (8) X48%54% +6% (4) Y53%57% +4% (6) Z61%59% -2% (12) X +2% (7.5)

12. 12

13. 13 How easy is it to do?  Statistically fairly easy to create a mark for marking based upon marks  Take into account high and low markers  Standard of expectation  Consistency … judge against final mark awarded for an essay (compensated median)  What about the comments?

14. 14 Feedback Index  Produce an index that reflects the quality of commenting  Produce an average feedback index for an essay  Compare against marker in a similar manner to marks analysis  Where does this feedback index come from and is it valid?

15. 15 The way to get the feedback index?  Develop an application?? –C-Rater?  Spelling Mistakes  Similar Meanings? –That was cool –Really Choc –Really Good Essay  Manually

16. 16

17. 17

18. 18

19. 19

20. 20

21. 21

22. 22

23. 23

24. 24 DAI SMITH ESSAY 1 4.5 3.8 3/10 Differences from actual: F/Index Mark Score ESSAY2 -3.0 8.5 5/10 FINAL MARK AWARDED 5/10 …………. … …. …

25. 25 Time Consuming?  Can we formulate the marking process  Take away need for quantification process of analyzing comments  Is it still peer-assessment if the students are told what to say?

26. 26

27. 27 STUDENT FRED REFERENCES: Positive ……… Negative ……. Personal Valuation 5, 3, 2, 1 3, 1, 2

28. 28 Some points outstanding  What should students do if they identify plagiarism?  Is it ethical to get students to mark the work of their peers?  Is a computerised solution valid for all?  At what age / level can we trust the use of peer assessment?  How do we assess the time required to perform the marking task?

29. 29 The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays  Student Acceptance:  Anonymity / Discussion / High & Low Markers  Tangible Reward  I’m more concerned with rewarding ‘higher order skills’  Getting there