‘The Philosopher’s Stone’? An Appraisal of Computer Assisted Objective Assessment in English Studies Dr Michael Jardine Dr Matthew Sauvage University of Winchester

L&T Projects at University of Winchester Problems to Solve: –‘Prompt’ return of work –Feedback limitations –Use of ‘Moodle’ –Student reading (set texts) ‘minimalism’ –Student seminar preparation ‘minimalism’ –Student progress difficult to monitor (with summative assessment)

Use of Computer Assisted Assessment (in BA English) Challenges to Introduction of CAA: –Potential colleague resistance: ‘pub Quiz’ reputation; time consuming; lack of necessary skills not career enhancing –Potential student resistance low level associated with distance learning replaces tutors –Limitations of Moodle/Blackboard

CAA Project Findings 1.Development of sophisticated items 2.Introduction of detailed feedback 3.Identifiable impact on formative learning better preparation for addressing summative tasks learning objectives are made clearer much earlier 4.Availability of valuable performance data Identify uncommitted / struggling students Identify areas of student comprehension or lack of 5.Time consuming / time saving 6.Lends itself to Level 4 (team teaching) 7.Dissemination issues

CAA Student experience Students are more likely follow up lecture and seminar material Regular on-line assessments provide material for seminar debate On-line feedback provision has unforeseen pedagogical advantages Creates a climate of intelligent study between students Can help overcome resistance to complex theoretical foundation material

CAA Tutor Experience forces tutor to analyze in great detail their teaching practices and the pedagogy behind methods used for module delivery produces a transferrable set of tools and work process that can drive the creation and remodelling of whole modules based on sound and clear pedagogical principles in all, but particularly in large, teaching team scenarios the use of CAA can benefit both students and teaching staff by bringing homogeneity to delivery, because of the workflow it creates

Designing CAA Mix of questions and item types within an activity – Will vary depending on activity objectives – Desired pace of activity needs to be considered Order of items and item options –Fixed by activity designer –Random item order –Random options order Sections – Grouping of related items –Can be visible or invisible to students –Use these to isolate week areas

Objective test items Complex objective test items can demand the use of and assess the following abilities (not just the regurgitation of facts):  Analysis  Comprehension  Interpretation  Application of knowledge to context  Problem solving  Synthesis  Experiments  Evaluation

Item (question) types that objective assessment software can process 1.Classifying ideas/concepts/data into a certain strict order. 2.Matching/pairing any type of information (e.g., word- definition, concept-example, statements-true/false) given in two columns. 3.Cloze – filling any gaps in text. 4.Exact answer – this is usually a single sentence or just a couple of words but it has to be exactly the same as the answer required. For this, options can also be given - for example, to take account of different ways of spelling. 5.Key terms stipulated by the author of the item. Partial marks can be allocated for each correct term used.

Item (question) types that objective assessment software can process 6.Multiple choice type item – single or multiple key. 7.True/False. 8.Responses to a picture, sound track or audio-visual sequence. 9.Target spotting. 10.Open answer (a short paragraph for example), which has to be marked manually - the idea of manual marking may be a deterrent but the scope and variety this type of question can add as part of the mix is significant.

Item anatomy – using stimuli in the stem With even the simplest stimulus in the stem it is possible to create demanding, exciting and effective items. Use: a text, however short an image or graph a sound file a video sequence

Transforming MCQs into learning activities Decide which of the terms is defined by: “Denoting a concept, person or thing through a term that represents a significant part of these (e.g., crown = monarchy).” A.Personification B.Hyperbole C.Metaphor D.Onomatopoeia E.Metonymy

Transforming MCQs into learning activities

Transforming simple items and CAA into learning activities

Consider the following passage from Terry Eagleton’s Literary Theory: An Introduction (London: Blackwell, 1983, p. 106). You must decide whether each of the numbered elements of the passage is relevant to (a) Ian McEwan’s Saturday, (b) Charlotte Bronte’s Jane Eyre, (c) both, or (d) neither: The narrator may know more than the characters [1], [the narrator may know] less than them [2], or [the narrator may] move on the same level [3]; the narrative may be ‘non- focalized’, referring to situations where the narrator knows more than the character, or more exactly says more than any of the characters know, delivered by an omniscient narrator outside the action [4], or ‘internally focalized,’ recounted by one narrator from a fixed position [6], from variable positions [7], or from several character-viewpoints [8]. 1. a, 2. b, 3. c, 4. c, 5. b, 6. a, 7. c, 8. c

ASSESSMENT on this module is summative: A. 2,000 word essay, due in week 10, chosen from the following: 1) Analyse the nature and function of narrative with reference to at least one text studied on the module.