eAssessment Colin Milligan Heriot-Watt University

Randy Bennett, ETS Like many innovations in their early stages, today’s computerised tests automate an existing process without reconceptualising it to realise the dramatic improvements that the innovation could allow.

Bennett’s 3 Stages 1.Resemble Paper Tests, 2.New Formats – multimedia and constructed responses, 3.Simulations, Virtual Reality – embedding assessment within learning.

Summative Formative Efficient for large student numbers, Consistent marking, Integration with institutional systems, Immediate feedback on performance. Immediate feedback, linked to remediation, Individualisation, Opportunity for multiple attempts, Opportunity for innovation.

It’s all Multiple Choice Easy to author (!), Easy to mark, Supported by VLEs. Assesses lower order skills, Guessability, Not suited to some disciplines.

Beyond Multiple Choice Numerical Assessment, Authentic Assessment, Short Answer Questions, Discussion Fora, Essays, Peer Assessment.

Numerical Assessment Randomisation, Expression evaluation, Partial credit, Avoids pitfalls of closed ‘objective- type’ questions, Matched to current assessment.

Authentic Assessment Instead of testing knowledge recall, tests interpretation and application: –Utilise data, –Manipulate simulation, –Perform a task.

TRIADS (University of Derby)

Pushing the Boundaries of Computer Marking src: Tom Mitchell, Intelligent Assessment Technologies

Short Answer Questions Intelligent Assessment Create model answers based on analysis of samples, Acceptable accuracy from around 50 responses, Optimum accuracy from ~200 samples.

Mean Marking Accuracy Increased Consistency, Marking schemes can be varied to suit content, Built in Moderation.

Essays: e-Rater Claims to assess quality of argument, style, examples etc. Requires thousands of sample essays to develop model answers. ‘Off the shelf’ essay questions.

Assessing Discussion Forum Activity Appropriate for Learning Environments, Assessing increases forum activity, Contributions can be assessed, but be careful with marking scheme – e.g.: –Baseline marks for activity, –Extra marks for worthwhile contributions – e.g followed up. Explain marking scheme clearly, Human marking required – intensive.

Low Stakes Peer Assessment Students submit task – e.g. critique of paper. Anonymised and sent out to others in group, Computer used for organisation, collation and distribution of content and feedback, Encourages reflection, Combines well with Discussion Forum activities.

Item Banking and Analysis MCQs will predominate, Question Banks, Highlights good and bad questions, Analysis of Question Quality: IRT –Difficulty, –Discrimination, –Guessability.

Item Characteristic Curve (ICC)

Difficulty: the displacement A more difficult question is less likely to be answered correctly by students of a given ability. more difficult

Discrimination: the slope High discrimination is good only if matched correctly to student ability.

Guessability: the baseline Poor distractors can raise the effective guessability.

Question Design All assessment can suffer from poor question design, Authoring systems make it easy to create (poor) questions, Multi-media presentation provides more opportunity for poor design.

standards orig. src: Charles Duncan, Intrallect

standards Question and Test Interoperability –Reuse questions, –Use others’ questions, –Change delivery systems. Other standards –Integrate with other systems, –Embed questions within content, –Identify questions by metadata, –Match questions to content.

Student Attitudes Resistant to change – so manage it. Sell the benefits – –immediate access to results, –remediation/feedback, Carefully integrate – don’t bolt-on – explain the rationale, Provide opportunity for practice.

Staff Attitudes Resistant to change – so manage it Sell the benefits: –Automation, –Access to others’ questions, –Tried and tested assessment, –Authenticity of Assessment, –Long-term efficiency.

Security, Plagiarism and Reliability Computers are subjected to far more rigorous scrutiny, See assessed work in context of the whole course, Adapt assessment to integrate understanding and bring in personal experience, Get students to rate themselves, Always have a backup plan.

Implementation Cost is front loaded: –Technology Investment, –Assessment Design (Pedagogical Staff Development), –Planning and Piloting (Systems, procedures and policies, Convince the stakeholders), Use computers where appropriate, For what they are good at, Important Issues go beyond the technology.

Where are we going? Assessment integrated with learning, blurring the distinction. Adaptive and diagnostic assessment. Rich feedback, tailored study Portfolios, Assessment of teamwork. Can the curriculum change? Will the curriculum change?

Further Information CAA Conference, Loughborough LTSN Generic Centre CAA CAA/index.phtml Slides and Notes for this Talk

Colin Milligan