Piloting CAA: All Aboard Gavin Sim & Phil Holifield

2 Overview Introduction CAA at UCLAN Key Challenges Staff Uptake Framework Staff Development Students Other Stakeholders Conclusions and Discussions

3 Introduction Teaching and Learning strategy incorporated e-learning – mainly content development First summative CAA test summer 2003 WebCT TRIADS and Questionmark evaluated for pilot study Questionmark - adopted felt easier for staff develop their own questions

4 Introduction Technical Infrastructure analysed concerns  Scalability – expansion over time  Connectivity – internal and external colleges  Bandwidth – 10 Mbps available, multimedia Purchased dedicated server host Questionmark, Internet Information Server & SQL Server Integration with other systems concern but addressed later Piloting Within Department of Computing

5 Key Challenges Encouraging Staff Uptake Staff Development Stakeholder Acceptance – e.g. management CAA perceived ability to test range of cognitive skills Practical Issues - Labs

6 Methodology Questionnaire Staff n=34 response rate 64%  Views in relation to CAA, support and training Framework developed based on Blooms Taxonomy, 6 staff, 8 modules Questionnaire Students n=86 response rate 94%  Acceptance of technique  Question styles  Language Used  Usability

7 Staff Uptake Computing encompasses range of subjects technical networking, subjective HCI CAA may readily lend itself assessment specific disciplines Questionnaire revealed only five members of staff used CAA, 3 actively using it Encourage uptake CAA being incorporated into department’s strategy, all level 1 formative and summative being optional

8 Staff Uptake Five staff now using CAA within the department Questionnaire revealed  91% use CAA formative  56% Summative Difference could be attributed level lecturer teaches

9 Framework Analysing structure of module identify how CAA could be incorporated into modules Bloom’s Taxonomy Learning Outcomes Syllabus Other Assessment format CAA

10 Framework Number of Learning Outcomes at each level of Blooms Taxonomy Level 1Level 2Level 3Level 4 CO1652C01802C01804CO2751CO2752CO2601CO3707CO4707 Knowledge11 Comprehension31111 Application3262 Analysis132 Synthesis11121 Evaluation1213

11 Framework Variations between number of Learning Outcomes from 3 – 8 Level 1 modules at lower Cognitive Level Level 2 Module CO2601 (Technical Solutions and Business) requires students to demonstrate similar ability found on CO3707 Next is to identify elements of syllabus and relationship to Learning Outcomes Prevent unrelated content being integrated into exam

12 Framework Example for CO3707 Identify the parts of the syllabus that relate to the learning outcomes. ABCD 1 Consideration of primary users XXXX 2 Introduction to Multimedia XX 3 Introduction to human systems XXXX 4 Multimedia Technology XXX 9 Importance of evaluation and choice metrics X

13 Framework Number of syllabus elements at each level of Bloom’s Taxonomy Level 1Level 2Level 3Level 4 CO1652C01802C01804CO2751CO2752CO2601CO3707CO4707 Knowledge21 Comprehension Application Analysis133 Synthesis321 Evaluation5619

14 Framework Is going to be used on MSc Web Development Module Module is all coursework Formative test in first semester  Enable students gain early feedback  Lecturer obtain early indication of their progress Framework shows how staff can integrate CAA into modules but further development necessary

15 Staff Development Asked staff ‘ Would you be prepared to input the questions into the software yourself?’  80% Yes  May not reflect attitude staff in other departments

16 Staff Development Lecturers need support in question design 74% LDU organised staff development in CAA  An introduction to Computer Assisted Assessment CIF bid for funding pay developer to work with staff develop multimedia questions 81% more time required to write questions  Question banks and experience reduce time 61% lecturers help invigilation (essential)

17 Staff Development Informal Focus Groups Discuss problems and share experiences  How accommodate students special needs  Invigilation issues  Risk issues e.g. server fails Without this students experience may be different from module to module

18 Students Attitude measured through series of questionnaires Students asked ‘ Would you find this format of assessment an acceptable replacement for part of your final exam?’ 5 Point Likert Scale, Strongly Disagree=0, Strongly Agree=4 Mean=2.9, SD=.9, 99% Conf. Interval ± 0.26 Indicates reasonable level of support

19 Students Research into computer anxiety and CAA (Liu et al. 2001; Zakrzewski & Steven 2000) Concerns, students no prior experience of QM ‘This format of assessment is more stressful than a paper based test’ Mean=.99, SD=.987, Conf. Interval ± 0.28 Comments ‘I prefer completing a test in this way as it is less intimidating’ ‘As a computer geek I feel more at ease in front of a computer.’ (final exam)

20 Students ‘Did you have any difficulties accessing the test?’  14% Yes Majority problems copying password from with white space Software could trim white spaces Authentication could be achieved through LDAP process

21 Students Questionmark used question by question delivery Standard Templates Question the suitability of a number of templates e.g. scrolling, navigation Idea have a template bank

22 Students Series of questions relating to the interface QuestionMeanStandard Deviation The test was easy to use It is easy to read the characters on the screen The screen layout is clear The screen layout is consistent The navigation was clear I always know where I am in the software The button location is consistent The order of the navigation buttons is logical The button names are meaningful The on-screen navigation is easily distinguished from the questions

23 Students 81 Students completed questionnaire 3o provided qualitative feedback Requested facility go directly back to previous question (11 times) ‘Proceed’ button felt inappropriate near main navigation Features incorporated into forthcoming test and further analysis will be conducted

24 Other Stakeholders Information System Services and Management informed through steering committee Responsibility report finding of the evaluation for institutional wide deployment Without support of management additional resources will not be made available

25 Conclusions and Discussions Scepticism about CAA appropriateness at level 3,4 for summative assessment Framework showed how it may be incorporated further research required Adopting CAA into departments strategy increased uptake but staff development necessary Students responded positively to experience Logging in process could be improved Comparison of WebCT and Questionmark planned

26 Questions