Phil Davies School of Computing University of Glamorgan “Super U” The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays Phil Davies School of Computing University of Glamorgan “Super U”

Computerised Peer-Assessment CAP System Permits students to mark & comment the work of other students . Also self-assess … used as a standard of expectation Internet, not Web-based system (developed in Visual Basic / Access)

Having done the marking, what next? Students should receive feedback What feedback? Marks Comments Which is most important?

AUTOMATICALLY EMAIL THE MARKER .. ANONYMOUS

What should the marker do? Reflect Look at essay again Take into account the essay owner’s comments Further clarification (if it is needed, then is this a ‘black mark’ against the marker?) Try to ‘appease’ the essay owner? Modify mark based upon reflection? Give more feedback

Must be rewarded for doing the ‘mark for marking’ process How to judge? Standard of expectation (self-assessment) Marking consistency Commenting, quality, measure against mark Discussion Element Need for additional comments – black mark? Reaction to requests / further clarification

Does this student deserve a good mark? ESSAY Compensated Median Mark?? Student Mark Difference W 65% 67% +2% X 48% 54% +6% Y 53% 57% +4% Z 61% 59% -2% +2% on average

Standard of Expectation Self-Assess = 68% Peer-Assessed as = 58% ESSAY Compensated Median Mark Student Mark Difference W 65% 67% +2% (8) X 48% 54% +6% (4) Y 53% 57% +4% (6) Z 61% 59% -2% (12) +2% (7.5)

How easy is it to do? Statistically fairly easy to create a mark for marking based upon marks Take into account high and low markers Standard of expectation Consistency … judge against final mark awarded for an essay (compensated median) What about the comments?

Feedback Index Produce an index that reflects the quality of commenting Produce an average feedback index for an essay Compare against marker in a similar manner to marks analysis Where does this feedback index come from and is it valid?

The way to get the feedback index? Develop an application?? C-Rater? Spelling Mistakes Similar Meanings? That was cool Really Choc Really Good Essay Manually

Differences from actual: F/Index Mark Score DAI SMITH ESSAY 1 4.5 3.8 3/10 ESSAY2 -3.0 8.5 5/10 …………. … …. … FINAL MARK AWARDED 5/10

Time Consuming? Can we formulate the marking process Take away need for quantification process of analyzing comments Is it still peer-assessment if the students are told what to say?

STUDENT FRED REFERENCES: Positive ……… Negative ……. Personal Valuation 5, 3, 2, 1 3, 1, 2

Some points outstanding What should students do if they identify plagiarism? Is it ethical to get students to mark the work of their peers? Is a computerised solution valid for all? At what age / level can we trust the use of peer assessment? How do we assess the time required to perform the marking task?

The Automatic Generation of ‘Marks for Marking’ within the Computerised Peer-Assessment of Essays Student Acceptance: Anonymity / Discussion / High & Low Markers Tangible Reward I’m more concerned with rewarding ‘higher order skills’ Getting there