E-assessment: a risk-based approach to success Dr Chris Ricketts, Sub-Dean (Teaching Enhancement), Faculty of Technology, University of Plymouth and Director of Assessment, Peninsula College of Medicine and Dentistry

Overview l history l institutional strategy l risk analysis l examples of success l discussion

History (1) l over 20,000 students l 6 faculties l 6 sites over 150 miles l some use of CAA on local servers

History (2) l Needs analysis in April 1999 l 12 staff, 5 faculties, over 3,000 students l Strategic introduction of CAA (University-wide)

Strategy (1) Zakrzewski’s (1999) ‘Catherine-wheel’ model â Module > Department > Faculty > University We chose University-wide availability, but â controlled pilot use in modules â undertook risk analysis â roll-out to University

Strategy (2) What to pilot? â formative assessment â summative (in-course) assessment â end of year examinations â a variety of subject areas

Strategy (3) l Decision 1: A steering group which involved staff from all affected parts of the University l Decision 2: Academic Board support l Decision 3: Staff training and support l Decision 4: Easy introduction to students l Decision 5: Get student feedback - and act on it

Strategy (4) Decision 6: the big one! Don’t treat computer aided assessments differently â management â quality processes â security

General experience l bed of roses?…not l we did prepare carefully and try to consider all risks

Risk analysis (Ricketts & Zakrzewski, AEHE 2005) Types of risk l Pedagogic (P) l Operational (O) l Technical (T) l Web-based (W) l Financial (F)

Risk analysis l Define the risks l Estimate likelihood l Estimate severity (who is affected?) l Concentrate on the severe problems l How to avoid l What if it happens?

Defining the risks l The literature can help, but… … these need to be YOUR risks, not someone else’s l Estimate likelihood l Estimate severity (who is affected?) l Concentrate on the real problems l How to avoid l What if it happens?

Risk example 1 l P1: Assessment method not integrated into the curriculum l Likelihood? M l Who affected? Students, Academic staff l How much? Module l When? Before assessment This was not a high severity risk for us.

Risk example 2 l O7: Module size too large for number of workstations available l Likelihood? H l Who affected? Students, Academic staff, Support staff l How much? University l When? Before assessment This was the most high severity risk for us.

Risk management example O17: Different invigilation requirements for CAA not recognized l Liaise with examinations office l Produce guidelines for invigilators l Ensure support team in exam includes technical support staff l Academic staff to be present at start of on- line examination

Issues l load testing the system for large-scale summative assessments l link between student records and assessment databases l student computing mistakes l adequate computer facilities on all sites l wayward staff

Some findings (from Roy Lowry) l Introduction of formative tests each week after lecture l Students are motivated enough to use the system l Although requiring some effort to set up, the system can be re-used l CAA in a formative mode has a significant impact upon learning

Next step l Expanded to cover all material l Use of MC, MR, drop-down boxes and numeric questions l Used for the end-of-module test l 103 students in Babbage open access area l Students obtained their marks immediately after they finished the test

Student use l 30% of students attempted all of the tests (average mark: 72%) l 65% attempted some of the tests (average mark: 53%) l 5% did not use the system (average mark: 45%)

Benefits Benefits for staff... â easier to give frequent feedback â no marking! Benefits for students… â more frequent feedback â instant marking â more self-assessment

“Cost effectiveness” l Thanks to James Wisdom (ELEN Conference 2000) for the following

Cost benefit analysis l Useful when benefits can be expressed in monetary terms eg.Saving in staff time (hence money) etc. l Is this why we use CAA?

Cost effectiveness l Useful when outcomes cannot be expressed in monetary terms l looks at outcomes in relation to goal “on time, on budget, to quality” l Is this why we use CAA?

Pedagogic effectiveness l Learners learning,and learning better l Must be part of cost effectiveness l Is this why we use CAA??

Questions l Over to you