© 2001 By Default! A Free sample background from Slide 1 Does interoperability actually work? A case study carried out with Boon Low and Niall Barr Niall Sclater Manager, Educational Systems University of Strathclyde

© 2001 By Default! A Free sample background from Slide 2 The IMS Question and Test spec For the exchange of questions and tests For the exchange of questions and tests Processes include: Processes include: –Authoring –Storage –Administration –Rendering

© 2001 By Default! A Free sample background from Slide 3 The packages tested A leading commercial CAA package (Questionmark Perception) A leading commercial CAA package (Questionmark Perception) The two leading virtual learning environments (WebCT and Blackboard) The two leading virtual learning environments (WebCT and Blackboard) A lesser known but highly innovative rendering tool (Canvas Arena) A lesser known but highly innovative rendering tool (Canvas Arena) A university-built tool developed for testing IMS QTI (CETIS Rendering Tool) A university-built tool developed for testing IMS QTI (CETIS Rendering Tool)

© 2001 By Default! A Free sample background from Slide 4 Scope of the evaluation Concentrates on the exchange of questions between systems Concentrates on the exchange of questions between systems Question creation Question creation Exporting Exporting Importing Importing

© 2001 By Default! A Free sample background from Slide 5 Question 1 Question 1: Which electronic component allows alternating current (AC) to flow but prevents direct current (DC) flowing?  A transistor  A resistor  A capacitor [correct]  An inductor

© 2001 By Default! A Free sample background from Slide 6 Feedback for question 1 Transistors are used to amplify or switch both AC and DC currents, the correct answer is capacitor. Transistors are used to amplify or switch both AC and DC currents, the correct answer is capacitor. Resistors impede the flow of both AC and DC currents, the correct answer is capacitor. Resistors impede the flow of both AC and DC currents, the correct answer is capacitor. Yes, capacitors prevent DC current flowing. Yes, capacitors prevent DC current flowing. Inductors have high impedance for AC currents and low impedance for DC currents, the correct answer is capacitor. Inductors have high impedance for AC currents and low impedance for DC currents, the correct answer is capacitor.

© 2001 By Default! A Free sample background from Slide 7 Question 2 Which of these symbols represents a Field Effect Transistor (FET)?

© 2001 By Default! A Free sample background from Slide 8 Criteria Question creation – the ease with which the two model questions can be created and the degree of accuracy of question rendering Question creation – the ease with which the two model questions can be created and the degree of accuracy of question rendering Exporting – the ease and accuracy with which the following applications export questions into the IMS QTI specification Exporting – the ease and accuracy with which the following applications export questions into the IMS QTI specification Importing – the degree with which the suite of test questions can be imported and how accurately the imported questions are rendered. Importing – the degree with which the suite of test questions can be imported and how accurately the imported questions are rendered.

© 2001 By Default! A Free sample background from Slide 9 Results of the evaluation Question authoring Perception Perception –q1 entered without difficulty –q2 not so easy but workarounds found WebCT WebCT –q1 fine –q2 – feedback reduced to a single section for each answer Blackboard Blackboard –Feedback restricted to one section for the correct answer and one for the incorrect ones

© 2001 By Default! A Free sample background from Slide 10 Results of the evaluation (2) Exporting Perception Perception –Worked fine WebCT WebCT –Requires use of a command line tool –Some minor errors in the exported QTI code Blackboard Blackboard –Questions exported within an IMS content manifest but in a proprietary format

© 2001 By Default! A Free sample background from Slide 11 Results of the evaluation (3) Importing Perception Perception –No problems Blackboard Blackboard –Didn’t work WebCT WebCT –Content migration tool only for systems admins –Extremely difficult to use even for them Rendering Rendering

© 2001 By Default! A Free sample background from Slide 12 Rendering: WebCT - Perception

© 2001 By Default! A Free sample background from Slide 13 Conclusions VLEs very disappointing VLEs very disappointing None of the other systems had problems with q1 None of the other systems had problems with q1 Questionmark has been more rigorous than the VLE vendors in implementing QTI Questionmark has been more rigorous than the VLE vendors in implementing QTI –But even it had some problems Does this bode well for interoperability? Does this bode well for interoperability?

© 2001 By Default! A Free sample background from Slide 14 Conclusions (2) Still in early days Still in early days Most problems easy to correct Most problems easy to correct Need cross-vendor collaboration Need cross-vendor collaboration Stress-testing programmes Stress-testing programmes VLE vendors may leave QTI compliance to QuestionMark VLE vendors may leave QTI compliance to QuestionMark

© 2001 By Default! A Free sample background from Slide 15 Further info