1. THE FRAUD AND PIRACY THREAT TO THE FAIRNESS AND VALIDITY OF HIGH STAKES TESTS Dutch Examinations Association Conference November 26, 2009

2. THE FRAUD AND PIRACY THREAT TO THE FAIRNESS AND VALIDITY OF HIGH STAKES TESTS John Fremer Caveon Test Security

3. Organization Devoted to Helping Testing Organizations: –Protect their intellectual property –Deter and/or detect cheating –Three primary services Security Audit – used by CITO Data Forensics – statistical methods to detect “testing irregularities” (Caveon Test Detective) Web Patrol – searching the Internet for your items or offers to take your test

4. The Problem Cheating on tests is a widespread problem:problem Unfair to the majority of honorable test takers Cheaters don’t acquire the skills/knowledge to perform Cheating undermines the value of testing as protection mechanism Cheating threatens the viability of a testing program Test items

5. EXTENT OF FRAUD AND PIRACY (cont.) Donald McCabe Surveys: Cheating is widespread Students find it easy to rationalize cheating The Internet is raising new questions Students feel that many teachers ignore cheating, at least on occasion Students cheat for a variety of reasons.

6. EXTENT OF FRAUD AND PIRACY (cont.) Caveon Experience, We Have Found: –Actual test content –Discussions about specific items –Offers to provide proxy exam takers –Descriptions of performance-based items –Extremely similar exam items –Guarantees to pass the tests

7. EXTENT OF FRAUD AND PIRACY (cont.) Strategies for Combating Cheaters and Pirates (Overview) –Make clear that adherence to rules is required –Tighten up policies and procedures –Data forensics analyses –Search the Internet –Employ more secure designs

8. Strategies for Combating Cheaters and Pirates # 1 Make clear that adherence to rules is required Honesty contracts do work –Dan Ariely, Predictably Irrational: people will be honest if you remind them to be and this applies to pirates as well as candidates

9. Strategies for Combating Cheaters and Pirates # 2 –Tighten up policies and procedures: Caveon Test Security Audit Systematic review by experienced professionals 250 security standards Immediate feedback - practical and specific Combination of review of materials and interviews

10. Caveon Test Security Audit Areas (Part 1) Security Plan Roles & Responsibilities Budget & Funding Legal Precautions & Agreements Test & Item Design Test & Item Development & Maintenance Test Publication Test Administration

11. Security Audit Areas (Part 2) Test Scores and Results Physical Security Information Security Web & Media Monitoring Security Awareness & Training Security Incidence Response Plan

12. Strategies for Combating Cheaters and Pirates # 3 Data forensics analyses Identify and measure test fraud threats Supports objective decision making –Test results don’t lie –Test item performance Extend the life of items Maximize test life span

13. Appeal of Statistical Analysis Strong Evidence of Interest in Data Forensics Analyses –ATP Conference workshop attendance –Interest in demos –Questions about use and interpretation Support from Lawyers about Use

14. Success Story – Education 72% reduction – 2 years New testing manuals, test admin monitoring Students “confess” Districts follow through

15. Strategies for Combating Cheaters and Pirates # 4 Web Search Basics: –Determine list of words to search on –Use search engines/send alerts –Continuous monitoring – not a part time job –Use experts to infiltrate forums and torrents –Pilot Caveon Web Patrol

16. Strategies for Combating Cheaters and Pirates # 5 Employ more secure designs We can design tests that have security built in –Impara & Foster (2006) and pre-testing security through design –Multiple forms recommended by Tippins et al. (2006) –CAT alone is not the answer but other models are very promising (e.g. linear-on-the-fly testing or LOFT) –Hollinger & Lazna-Kandice (1996) report that 80% of students reported the “scrambling” of items as the most effective anti-cheating strategy –Trojan Horse items

17. Trojan Horse Items EMC exams available on over 200 websites –How were they getting there? –Who was using them? –Where are we certifying people who did not understand the content? Concept – Liz Burns and colleagues - EMC

18. Trojan Horse Items (cont.) STEP ONE - Design Trojan Horse Items Un-scored items designed to identify candidates who: 1.Memorized the answer key, AND 2.Do not understand the material Added benefit - identifies when an exam first hits the web

19. Trojan Horse Items (cont.) STEP TWO – Track Individual Results Perform monthly forensics looking for TH “hits” Actions taken based on probabilities –1 in 1 million threshold to revoke exam results and notify employer –1 in 500,000 threshold for warning letter –Second incident – banned from program Average 10 per month – 1% of overall exam attempts

20. Trojan Horse Items (cont.) STEP THREE – Track IP Theft Trojan Horse “hits” mean exam is available Review exam activity between release date and first TH “hit” date to identify unusual activity –Reg/Cancel –Testing by candidate that does not “fit” –Anomalous response patterns Immediately close testing center where unusual activity identified Several identified and closed: “Dear: Not technical problem, the reason from the test center. Whenever we download any EMC exam, after this EMC exam will be removed by prometric. So now we cannot got the emc exam data, so we also cannot do the exam for you. Thanks

21. Trojan Horse Items (cont.) Ensure Ongoing Validity and Success Perform regular item analysis on TH items –Ensure they are operating as expected Consistent and firm in taking action –Many will plead – decisions based on data not judgment Gain stakeholder buy-in –1 in a million probability –Analysis by third party forensic analysis Support with strong Candidate Agreement

22. RESOURCES (Overview) Professional Standards: –ANSI Standards –ATP Best Practices Work Group Caveon Test Security –Test security standards –Top Ten Cheaters –Cheating in the News

23. IN CONCLUSION Questions? John Fremer Caveon Test Security John.fremer@caveon.com Phone: (215) 805-3007 Fax: (609) 404-0274