© 2001 By Default! A Free sample background from Slide 1 CAA Pedagogy, standards and interoperability LTSN 27 th Jan 2003 Niall Sclater Manager, Learning Services University of Strathclyde

© 2001 By Default! A Free sample background from Slide 2 What is CAA? The use of computer-based tools for: the creation, storage and delivery of assessments to candidates the capture, marking, storage and analysis of their responses the collation, return and analysis of results

© 2001 By Default! A Free sample background from Slide 3 Types of CAA Diagnostic Formative Summative

© 2001 By Default! A Free sample background from Slide 4 Why use CAA? Improving learning Objective marking Reducing marking load Automated results collation, analysis and return Improving access Reducing opportunities for plagiarism

© 2001 By Default! A Free sample background from Slide 5 Costs Insitutional VLE? Obtaining or creating questions and tests Staff development – –Management – –Teaching staff – –Support staff Technical support Hardware

© 2001 By Default! A Free sample background from Slide 6 Authentication and security Knowledge Ownership Characteristic Impersonation without candidate’s permission Impersonation with permission –Use real time questioning –Get to know writing style Restricting access to other resources Restricting access to other resources

© 2001 By Default! A Free sample background from Slide 7 Selecting a CAA system Is your VLE adequate? Quality of vendor Navigation facilities Usability Help facilities Security features Candidates’ responses Interoperability Accessibility Accessibility

© 2001 By Default! A Free sample background from Slide 8 MCQ Name the capital of Saudi Arabia. Mecca Medina Jiddah Riyadh

© 2001 By Default! A Free sample background from Slide 9 True/false Yes/no Sydney is the capital of Australia True False Is Sydney the capital of Australia? Yes No

© 2001 By Default! A Free sample background from Slide 10 Short answer A body will continue in its state of rest or uniform velocity unless acted upon by an unbalanced force. What is this law known as?

© 2001 By Default! A Free sample background from Slide 11 Fill in the blank / gap fill Complete the following: Now is the of our discontent. Toledo es a setenta kilómetros de Madrid. está

© 2001 By Default! A Free sample background from Slide 12 Fill in the blank / gap fill Toledo es a setenta kilómetros de Madrid. está Justify your choice of es or está in the above sentence.

© 2001 By Default! A Free sample background from Slide 13 Matching Associate the following plants with the most appropriate type: lysichiton americanus evergreen perennial cornus alba deciduous shrub phormium tenax marginal aquatic perennial verbascum bombyciferumbiennial or short-lived evergreen perennial

© 2001 By Default! A Free sample background from Slide 14 Hot spot Click the stigma of this flower:

© 2001 By Default! A Free sample background from Slide 15 Sequencing Put the following colours of the rainbow into the correct order. Orange Yellow Indigo Blue Red

© 2001 By Default! A Free sample background from Slide 16 Adding complexity to MCQs An inelastic collision takes place between a moving object and a stationary object. Both objects have the same mass. In this situation, which of the following quantities is/are conserved for this system? 1 Momentum 2 Total energy 3 Kinetic energy 1 and 2 2 and 3 1, 2 and 3

© 2001 By Default! A Free sample background from Slide 17 Metadata Contact details of creator Date of creation Description Level

© 2001 By Default! A Free sample background from Slide 18 Automated text marking Keyword analysis Natural language processing

© 2001 By Default! A Free sample background from Slide 19 Scoring strategies Make candidates aware of strategy MCQs usually allocate one mark for a correct response Negative marking Multiple response scoring

© 2001 By Default! A Free sample background from Slide 20 Holding invigilated test sessions Develop evaluation procedures Trial system Register candidates for system Give candidates practice Give candidates prior information Prepare room and equipment Ensure adequate staffing Ensure security Specify technical failure procedures Specify emergency procedures Authenticate candidates

© 2001 By Default! A Free sample background from Slide 21 Why do I need to know about QTI? I’m authoring a bank of questions in my subject area I’m authoring a bank of questions in my subject area I’m involved in the selection of a CAA system for my institution I’m involved in the selection of a CAA system for my institution I have a general interest in online learning and assessment I have a general interest in online learning and assessment

© 2001 By Default! A Free sample background from Slide 22 The basics IMS QTI is a means for exchanging assessment information IMS QTI is a means for exchanging assessment information It uses XML It uses XML 2 key parts of the spec: 2 key parts of the spec: –Assessment Section Item –Results Report Also QTI lite Also QTI lite Can be implemented separately Can be implemented separately

© 2001 By Default! A Free sample background from Slide 23 Rendering example

© 2001 By Default! A Free sample background from Slide 24 True false example

© 2001 By Default! A Free sample background from Slide 25 True false example XML This is a simple True/False multiple-choice example. The rendering is a standard radio button style. No response processing is incorporated. Paris is the Capital of France ? True False

© 2001 By Default! A Free sample background from Slide 26 Where does QTI fit in with other specs? IMS Meta-data Specification IMS Meta-data Specification IMS Content Packaging Specification IMS Content Packaging Specification IMS Learner Information Packaging Specification IMS Learner Information Packaging Specification IMS Digital Repositories Interoperability Specification IMS Digital Repositories Interoperability Specification

Elect Drafts Scope IMS Spec Development Process IMS Members Technical Board Public Working Group Final Scope Base Document Public Draft Feedback Approval

© 2001 By Default! A Free sample background from Slide 28 Products which say they are implementing QTI E-Test 3, RIVA Technologies Inc E-Test 3, RIVA Technologies Inc Question Mark Perception Question Mark Perception Can studios Can studios Oracle i-learning Oracle i-learning Blackboard Blackboard WebCT WebCT Ultimate assessment Ultimate assessment E3AN E3AN TOIA TOIA

© 2001 By Default! A Free sample background from Slide 29 QTI: some problems For vendors For vendors –implementing QTI is costly –stops them from protecting their user base Vendors want simple specifications; users want broad, well-defined ones Vendors want simple specifications; users want broad, well-defined ones It is difficult to define specifications which do not have an effect on functionality It is difficult to define specifications which do not have an effect on functionality

© 2001 By Default! A Free sample background from Slide 30 Finally: why bother with QTI? Avoid lock in Avoid lock in –allow data to be moved in standard formats –allow one system to replace another Integrate systems from different vendors; allow choice of best assessment tool Integrate systems from different vendors; allow choice of best assessment tool Facilitate exchange of questions and tests between institutions, allow market to develop Facilitate exchange of questions and tests between institutions, allow market to develop Enable separation of content from presentation Enable separation of content from presentation Standard assessment records will facilitate student mobility Standard assessment records will facilitate student mobility

© 2001 By Default! A Free sample background from Slide 31 Further information BS7988 “A code of practice in the use of IT for the delivery of assessments”