1 Examination Accreditation Program Exam Security: Preventing Exam Espionage Saturday, October 1 Baltimore – Maryland - USA

2 Exam Security: Introducing the Presenters During Test Development Lawrence J. Fabrey, Ph.D. During Test Administration Dave Moore After the Test Chris Beer Tadas Dabsys

3 Benefits of ARELLO-Accreditation Independent Evaluation Confidence and Protection Defensible Standards Reasonable Consistency in Quality Facilitates License Recognition (ARELLO Guidelines for Accreditation)

4 ARELLO Additional Resources About Exam Development An Accreditation Overview ARELLO ® Matters to Consider in Exam RFP ARELLO ® Top Ten Questions and Complaints on Licensing Exams ARELLO ® Supreme Court Case - Exam Validity ARELLO ® Examination Accreditation Program - White Paper on Determining Minimum Passing Scores

5 Security During Examination Development Physical Environment Electronic Access Staffing Inventory Storage Subject Matter Experts Other Functions Translation, Printing, Artwork, etc.

6 Development – Physical Access Secure Room/Area Vault – Key Access Limited Authorized Access Chain-of-Custody Shredding – including notes/drafts

7 Development – Electronic Access Passwords & Screensavers Selective Access Permission Specific Time Period Read-only, non-print, non-export Webinars

8 Development - Staffing Staff Confidentiality Agreements Conflict of Interest Audits

9 Development - SME’s and Other Functions Subject Matter Experts Translators Artwork Supplies for Practical Exams Printers Couriers

10 Development – Inventory and Distribution Printing (Serial #’s, Shrink wrap) Shipping (Traceable, Armed Courier) Electronic Transmission Web Access

11 Security During Exam Administration Paper & Pencil Computer-based Testing Practical Exams Open-book Exams Special Accommodations Facilities & Staff Candidates

12 Administration – Paper & Pencil Printing Shipping Receipt and Inventory Storage Contingency Plan Return Shipment & Inventory

13 Administration - CBT CBT Centers Restricted Login Access Restricted Time Periods Restricted Printing Cameras, DVR’s

14 Administration – Special Administrations Practical Exams Raters (SME) – Conflict of Interest Supplies

15 Administration – Open-book Open-book Exams Notes Highlight/Underline Post-its/Dividers Screening

16 Administration - Facilities Limited Access Spacing Candidates Monitor Restroom Breaks Secure Storage - Supplements Special Accommodations

17 Administration - Staffing Conflict of Interest Self-regulating Restricted Access ( , Internet) Audits Training – Active Proctoring Termination

18 Administration - Special Special Accommodations Interpreters

19 Administration - Candidates Candidates Identification Roster Signature Fingerprint Login Clothing/Purses Activity/Mannerisms

20 Available through SkyMall and Web

21 Caught on Camera Smart phone Notes

22 I know I will be Caught

23

24 Show Me the Money!

25 Security After the Test Data Forensics Monitoring Websites Item Updating Facility Screening Audits Incident Reporting

26 After – Data Forensics - Items p-value: Percentage of candidates who answer an item correctly pbis:Correlation between performance on an item and performance on the overall exam option stats: Percentage of candidates who answer each option correctly option pbis: Correlation between performance on the options and the overall exam

27 After – Data Forensics - Items p-value: Too high, too low pbis:Too low options:Distractors with no responses Distractors with more responses than key O pbis:Distractors higher than key OmitsAbove expectations

28 After – Data Forensics - Candidates Suspicious Candidate Activity Test for significant differences in pass rates Better performance on harder items High scores in low test times Suspicious improvement in scores

29 After – Monitoring Websites Cram Schools Sample Exams False Advertising Social Networking

30 After – Item Updating Monitor Item Statistics Replace Items Add New Items

31 After – Facility Screening Look for Notes in Testing Station Screen Restrooms and Hallways

32 After - Audits Test Center Audits “Secret Shopper” Programs Inventory Continuing Education for Proctors

33 After – Incident Reporting Documentation Evidence Reporting to Client

34 Examples Cheating Incidents

35 ARELLO Accredited Providers Applied Measurement Professionals  Pearson VUE  PSI Services LLC 

36 QUESTIONS? COMMENTS! Fabulous Prize Giveaway