1. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Security Strategies for Every Stage of the Testing Process DeDe Hedlund, Creighton University Jeff Place, Questionmark Evangelist NCTA – Minneapolis, MN Friday Aug. 3, 2012, 11:30-12:30 pm

2. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. There are no shortage of test security challenges At each stage of the assessment process (Authoring, Scheduling, Delivery, and Reporting) solutions exist to meet security challenges Lets Identify the threats to watch out for Show you today how certain product functionality can meet your test security needs by walking through a typical examination process and some real life examples Goals of this Session Slide 2

3. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Impersonation Content Theft Cheating Threats Slide 3

4. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Issues Fraud TriangleThreats Rationalization Opportunity Motivation Impersonation Content Theft Cheating

5. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Higher Stakes High Medium Low Medium Stakes Low Stakes Low/High to High/High Stakes Tests Slide 5 Whats at Stake? Life and Limb Promotion & Jobs & Legal Concern Educational Exams Tests Elearning & Surveys

6. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Personable, consistent Monitor vulnerable to unfair influence Test Centers Convenient, personable, consistent Monitor vulnerable to unfair influence Events (Classrooms or Conventions) Monitor is less vulnerable New and not yet widespread Remote Real-time 360 cam Monitor is less vulnerable Easy to deploy for use at home Remote Real-time webcam Seems secure Nothing for content theft Record & Review 360 or webcam Works for employees Nothing for content theft Un-Monitored Monitoring Tests Securely Slide 6 ID Fraud Protect Content Minimizes Cheating X XX

7. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Test Centers Events (Classrooms or Conventions) Remote Real-time 360 cam Remote Real-time webcam Record & Review 360 or webcam Un-Monitored Providers Slide 7 ID Fraud Protect Content Minimizes Cheating X XX

8. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Create Collaborative, secure authoring of surveys, quizzes, tests and exams Deliver Blended delivery to support a broad range of devices and security needs Analyze Evaluate results and inform stakeholders with timely, meaningful reports Assessment Management Process

9. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 9 Authoring: Item Banking and Security Security impacts Validity of assessment results Fairness to participants One approach: security rights that are role- based Set access permissions by role or profile Limit an authors or reviewers access to the various functions Limit access to topic (item) folders Limit what the author may do in those folders

10. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 10 Scheduling Create participants and schedule assessment delivery Manage participants individually or in hierarchical groups Schedule access to assessments: Limit dates Limit attempts Require proctor/invigilator Deliver to certified test center Online or offline delivery Email Broadcast: Notifications to Participants

11. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Formative Quizzes Course Evaluations Diagnostic Tests Low Stakes Tests Medium Stakes Exams High Stakes Exams Levels of Monitoring Blended Delivery Stakes of Assessment

12. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 12 Questionmark Secure Stops people from printing questions Stops people from typing in their own URL Always display in full screen and its not possible to maximize or minimize Stops people starting a new task Does not provide menu options or icons Disables control keys and task switching Disables right-click menu options Disables the ability to start new applications Prevents going backwards to a previous page Stops people exiting in a high stakes, proctored, environment Hides the HTML source Provides an API to control certain functions of a browser from the server Server can command Questionmark Secure to display a toolbar Secure Browser Regular Browser

13. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 13 Browser Check Ensures Test Takers use a compatible browser

14. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. All participant scoring is conducted on the Perception server (rather than on client/participant PCs) Scoring algorithms are not sent in any way to participant PCs Perception can be configured to use SSL to ensure communication between participants and the server are secure Data in item repositories are encrypted (QML is encrypted) providing added item bank security Delivery Security Slide 14

15. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Delivery Security Features

16. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Question by Question (QxQ) delivery options limit the amount of content shown at one time Randomization of questions and choices to prevent cheating

17. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Score List Report IP addresses (as expected?) Time taken (too quick, too long?) Easy flagging of participants who get perfect scores for further investigation Coaching Report Drill down on an individual participant, comparison to benchmark, use in appeal process Item Analysis Report Determine psychometric performance and examine item difficulty drift over time Test Analysis Report Assessment defensibility review and test difficulty drift Reporting Slide 17

18. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Case studies from Creighton University a. What their test security needs were b. How our software addressed those needs c. Demo of solutions Case studies Slide 18

19. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Creighton University School of Pharmacy & Health Professions Private Jesuit University located in Omaha, NE 2001 started the first and only accredited truly distance based full time Doctor of Pharmacy program Issues to resolve and security requirements 1. How to securely deliver and administer exams to students located throughout the country 2. Paper and pencil were too expensive and its difficult to distribute and collect approximately 5600 exams per semester 3. Control who sees what test and when 4. Lock down the computer to eliminate printing, instant messaging, browsing the internet during the tests 5. Ability for students to mark an identified area of a graphic

20. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Solutions offered to meet requirements: 1. Provided online testing solution and eliminated almost all paper and pencil tests to reduce costs 2. 22 Question types including Hotspot allowing online exams to replicate past paper exam formats with the additional ability to provide instant feedback. 3. Ability to randomly deliver different variations of an exam 4. Manage large item banks of questions including security for collaborative authoring among faculty 5. Lock down browser 6. Scheduling (right test to right student) Demos 1. Hotspot question type & feedback text/images 2. Password system & variations of an exam 3. Item Banking and security for collaborative authoring 4. Questionmark Secure 5. Scheduling Creighton University School of Pharmacy & Health Professions Slide 20

21. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Demonstration

22. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Variations of an Exam Slide 22

23. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Item Banking Slide 23

24. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Additional security measures 1. Established guidelines for choosing new proctor sites 2. Formalized the process of communicating what is expected from a proctor site 3. Proctor Certification forms are required from every student who completes an exam 4. Exam specific instructions are emailed to the proctors Creighton University School of Pharmacy & Health Professions Slide 24

25. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Laptops and PCs Custom Hardware Remote Video Monitoring Slide 25

26. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Candidate (Participant) Logs in As Software Secure and Questionmark integration is for higher stakes exams a Participant will be referred to as a Candidate Slide 26

27. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Candidate selects exam and Schedules Appointment Schedule Appointment Exam Exams (4)Surveys (1) Slide 27

28. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Schedule Day and Time Slide 28

29. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. If the exam is to be schedule in the next 3 days or if the participant might be expected to pay some of the cost of the exam then there are more steps Review Schedule Slide 29

30. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Payment Process Slide 30

31. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Candidate Takes Exam When It Was Scheduled

32. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Candidate (Participant) Logs in Slide 32

33. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Selects Exam to Take Exams (4)Surveys (1) Slide 33

34. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Camera and Microphone Access Identity Checks Captures Photo, Proctor Checks against files Proctor checks has Participant scan the room ProctorU / Software Secure

35. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. After Authentication Candidate takes Exam Video and screens are monitored during exam Security Checks Complete Candidate Takes Exam Slide 35

36. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Potential Applications for Mobile Delivery Deliver to the device in your hand Observational assessments Mobile test centers

37. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Mobile Test Centers Slide 10 Use notebooks/3G laptops/iPads to create mobile test centers Enables greater flexibility in where you deliver assessments Assessments can be conducted in on location environments

38. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Securing an iPad… Slide 11 An app should prevent candidates from easily getting to other URLs But – access to home button allows task- switching Hardware solutions are available to enable use of iPad in a kiosk mode

39. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Considerations: Slide 12 The higher the stakes, the higher the propensity to cheat –still require human monitoring! Bandwidth – Be sure to test the wifi and/or 3G signal strength prior to event

40. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Cheat-resistant questions and assessments

41. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Randomization Refresh items regularly Build large item banks Track item parameters Tests that are unique for each test taker make it hard to share answers Adaptive tests provide different questions based on responses to previous questions Considerations for cheat-resistant questions and assessments Slide 41

42. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Answer choices should be roughly the same length and kept as short as possible. Provide a minimum of three answer choices and a maximum of five. Four is considered optimal. Keep your writing clear and concise – youre testing knowledge, not reading comprehension. Make sure that youre putting the correct answer in the first two positions as often as the last two positions. Improve Your Multiple Choice Questions Slide 42

43. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Avoid Previously Delivered Questions Slide 43

44. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Extended Matching Question Type The number of answer options depends on the logical number of realistic options for the test taker.) The same answer choice could be correct for more than one question in the set Some answer choices may not be the correct answer for any of the questions So it is difficult the answer this type of question correctly by chance. A well-written lead-in question is so specific that students understand what kind of response is expected, without needing to look at the answer options. Slide 44

45. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Scenario Based Questions Slide 45

46. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 46

47. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Scan a QR Code with your mobile to start an assessment! Slide 47

48. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Closing and questions Thank you for your time and attention! Questions/Comments? Visit our booth to talk to us more We have a whitepaper on this topic (Delivering Computerized Assessments Safely and Securely) Visit our website to download: http://www.questionmark.com/us/whitepapers/index.aspx http://www.questionmark.com/us/whitepapers/index.aspx

49. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Thank you for attending! DeDe Hedlund dhedlund@creighton.edu Jeff Place jeff@questionmark.com