1. SCROLLA 16th June 2004 The Future of Computer Aided Assessment Helen Ashton Ruth Thomas Heriot-Watt University

2. SCROLLA 16th June 2004 Why Assess? Diagnosis Support for Learning Certification of Learning Quality Assurance

3. SCROLLA 16th June 2004 Assess How? Traditional exams, open book exams, quiz(multiple choice/true-false), essays, reviews, reports, practical work, portfolios, presentations, vivas, performances, group work, self assessment, peer assessment...

4. SCROLLA 16th June 2004 Computerising – why? Reduction of teacher workload Flexibility Immediacy Accessibility Reuse

5. SCROLLA 16th June 2004 Computerising – what? Which bits can be automated? –Delivery –Data Storage/Collection –Customisation/Reproduction –Marking –Feedback

6. SCROLLA 16th June 2004 PASS-IT Online delivery of NABS Online, incremental data collection Randomisation for reuse Some automatic marking (including partial credit) Immediate & post assessment feedback

7. SCROLLA 16th June 2004 Process Development with Subject Specialists –aligning learning objectives Validation & verification procedures –marking schemes –usability Formative practice -> summative exam

8. SCROLLA 16th June 2004 “Like many innovations in their early stages, today’s computerised tests automate an existing process without reconceptualising it to realise the dramatic improvements that the innovations could allow” Randy Bennett (1998)

9. SCROLLA 16th June 2004 How can reflection be developed? (learner and teacher) Are there new opportunities to assess a wider range of learning outcomes (in CAA)? Are there approaches to assessment that were impossible/impractical?

10. SCROLLA 16th June 2004 Developing Reflection During assessment –Immediacy –Some Automation Post assessment –Immediacy –Repurposing (reporting, reflection) Example

11. SCROLLA 16th June 2004

14. Developing Reflection A wealth of information! –What do we record? –How do we report it? Further developments –Assessing features of answers to give individualised feedback –Individual appropriate experiences Where else can we go? How do we get to “the guide on the side”?

15. SCROLLA 16th June 2004 Higher Order Skills Automation of existing techniques: –Objective testing –Mostly of lower order skills What about –understanding, –application, –analysis, –evaluation and –creation.

16. SCROLLA 16th June 2004 Higher Order Skills Simulations – –exploratory environment, realism, interactivity. Aim total integration –2 way communication –Configure simulation starting state –Simulation state available for marking and storage. Example

17. SCROLLA 16th June 2004 SolarGeom example Randomisation of state Learner must devise strategy to determine latitude. –Day length, seasonal differences, etc. Hypothesis, prediction, test, revise More than one skill involved –Understanding, analysis, evaluation

18. SCROLLA 16th June 2004 Authentic assessment Realistic task – greater complexity and duration. Assessing in the same environment in which learning takes place. Example

19. SCROLLA 16th June 2004 Authentic assessment example Calculating the activation energy of a reaction The same simulation can be used with different experimental data Issues –Support –Feedback

20. SCROLLA 16th June 2004 Questions What should we be assessing? How do we integrate learning and assessment? Can CAA ever be as good as a 1:1 session with a teacher/expert? By concentrating on automating the current process are we strangling innovation?