How should we measure the costs (and benefits?) of computer aided assessment? Chris Ricketts, Paul Filmore, Roy Lowry & Sally Wilks University of Plymouth

Overview l Why cost CAA? l Some problems l Different approaches l Activities to cost l Using different staff l Discussion (plenty,I hope!)

Why cost CAA? (1) Emphasis on technology in teaching and learning Costs of teaching can be quantified Costs of learning technology becoming understood (no savings!) Funding bodies expect cost allocation

Why cost CAA? (1) No longer enthusiast-led Strategic introduction of CAA (University-wide) Leads to change in resource usage

Problems l Cost of current assessment systems not quantified (in universities) l If universities were commercial examiners, perhaps this would change! l Costs of on-line systems variable (commercial or in-house products,or by- product of VLE)

Literature l Pollock (2000) CAA Conference: introduction into mathematics course l Other CAA Conferences - nothing l “Computer Assisted Assessment in Higher Education” (1999) -nothing l Do you know differently?

Possible approaches l Cost-benefit analysis? or l Cost-effectiveness?

“Cost effectiveness” l Thanks to James Wisdom (ELEN Conference 2000) for the following

Cost benefit analysis l Useful when benefits can be expressed in monetary terms eg.Saving in staff time (hence money) etc. l Is this why we use CAA?

Cost effectiveness l Useful when outcomes cannot be expressed in monetary terms l looks at outcomes in relation to goal l “on time, on budget, to quality” l Is this why we use CAA?

Pedagogic effectiveness l Learners learning,and learning better l Must be part of cost effectiveness l Is this why we use CAA??

Cost-benefit analysis l We suggest that the costs and benefits need to be examined from the view of three ‘stakeholders’ in assessment 1 - the University 2 - academic staff 3 - students

The University Costs l Hardware (server etc) for CAA l Software for CAA l Support staff for maintenance of server and software upgrades l Provide staff development l Additional load on IT infrastructure Benefits l Can compensate for lack of staff resources (time) l Less paper and printing l Better retention of students l Increased cross- departmental working

Academic(?) staff Costs to staff l Time to learn the system Benefits l No marking l Easy recording of marks l Additional mode in assessment strategy l May improve staff- student contact l Better information on students l Better information on assessments

Students Benefits l More formative assessment (currently limited by staff resources) l Fast feedback l Repeatability l Clarity of assessment criteria l Flexibility in place l Flexibility in time l Apparent fairness l May improve staff-student contact Costs l ?

How to measure costs Bacsich and colleagues (1999) propose activity-based costing Pollock and colleagues (2000)looked at three main activities - setting time - invigilation time - marking time

Assessment activities(1)

But whose time? Need to know who does what before CAA Need to know who does what after CAA Does institutional support change the balance?

Assessment activities (2a)

Assessment activities (2b)

Questions l How do we find out the costs of assessment? l How many of the ‘benefits’ can really be costed? l Do we use CAA because it saves money? l Do we use CAA because of pegagogic effectiveness?