CLEAR 2008 Annual Conference Anchorage, Alaska “Computerized Mastery Testing A Testing Architecture” F. Jay Breyer, Prometric Bob Riley, NCTRC

CLEAR 2008 Annual Conference Anchorage, Alaska Decision Error: The Problem CMT Was Designed to Solve

CLEAR 2008 Annual Conference Anchorage, Alaska What is CMT and How Does it Work? Examinees take the test in stages (separately timed sections): –Stage 1 is longer than subsequent stages –Stage 2 to the last or k th stage consists of a single packet of test questions called a testlet Following each stage (except the last) one of three decisions is made: –Pass –Continue –Fail After the last stage one of two decisions is made: –Pass –Fail

CLEAR 2008 Annual Conference Anchorage, Alaska From Psychometrics: Testlets – Packets of Test Questions Divide the content of a test into the smallest number of questions or tasks possible so that each –covers the entire test specifications consistently –has the same difficulty –spreads people out similarly –from 10 to 25 questions – 15 questions is most common Also useful for tasks –All testlets are equal to each other in content, form, and difficulty and have no repeated items or tasks

CLEAR 2008 Annual Conference Anchorage, Alaska From Psychometrics: Testlets – Packets of Test Questions Build testlets from client full-length test forms –Doesn’t use IRT Does use empirical Bayes small sample procedures and the psychometrics of testlets –requires a minimum of 125 candidates per original test form We can start with between 3 to 5 linear CBT forms administered in a single window Or we can divide up a performance test into comparable sections

CLEAR 2008 Annual Conference Anchorage, Alaska From Manufacturing Engineering Destructive Testing: Sequential Analysis

CLEAR 2008 Annual Conference Anchorage, Alaska From Mathematical Statistics: Bayesian Loss Functions At Each Stage Calculate a loss value at each raw score for –Passing a nonmaster –Failing a master Calculate a cost value of exposing an additional testlet at each raw score

CLEAR 2008 Annual Conference Anchorage, Alaska From Mathematical Statistics: Bayesian Loss Functions The Technical Details There is a science behind this that uses Bayesian loss functions and Two weights (A:B) symbolizing the client’s perception of the seriousness of making decision error (i.e., 100:50) –I (A) and –II (B) These two weights implement the client’s view of making a decision error into the cuts used at each stage of testing except the last – the last stage uses the cut for a full-length test

CLEAR 2008 Annual Conference Anchorage, Alaska How Do We Get the Cuts at One Stage of Testing?

CLEAR 2008 Annual Conference Anchorage, Alaska What do the Raw-Score Cuts Look Like in a Program? With 165-items in a full-length test With 75 items in the base test Sequential Testing gives us the concept of testing to a limit and making a decision Stage Of TestingFail BelowPass At or Above 1 (75 items – 86 min.)* (90 items – 100 min.) (105 items – 114 min.) (120 items – 128 min.) (135 items – 142 min.) (150 items – 156 min.) (165 items – 170 min.) * Pretest questions are included here

CLEAR 2008 Annual Conference Anchorage, Alaska Simulated Examinee Sample Based on Statistics from Year 2001 May 1-5 and July 10-August Tested Sample Stage PassFailClassifiedContinuePassFailClassifiedContinue 161 % 6%67%33%61%7%68%32% 25%2%7%26%3%4%7%25% 34%1%5%21%6%1%7%18% 43%2%5%16%4%1%4%14% 51%2%4%12%2%4%6%8% 63% 6% 2%0%3%5% 73%4%6%0%3%2%5%0% 80%20%100% N=10,00081%19%100%N = 225 A Simulation Result

CLEAR 2008 Annual Conference Anchorage, Alaska Who Gets Longer Tests? Borderline Examinees

CLEAR 2008 Annual Conference Anchorage, Alaska Who Would Use It? CMT is useful for clients who –value the time of their candidates and wish to reduce testing time for most examinees –wish to reduce the exposure of their test questions (increase in test security) results of year 2001 assembly – same as 2006 window –don’t have the money or resources for CAT CAT is resource intensive – staff, items, examinees –desire a CBT method more powerful than a linear test (CLT) helps control classification errors –want accurate assignment of candidates to pass/fail status

CLEAR 2008 Annual Conference Anchorage, Alaska Who Should Not Use It? CMT is not recommended for clients where –small social groups share items with each other –item harvesting is a known issue –group culture studies old test items

CLEAR 2008 Annual Conference Anchorage, Alaska Where Else Can We Apply This? How about a Performance Test –Imagine a performance test that consists of multiple separate observations of a candidate across different occasions –For example: 5 separate observations each worth 10 points The observations are accumulated (summed) But they are expensive – since you need two observers

CLEAR 2008 Annual Conference Anchorage, Alaska Where Else Can We Apply This? Let’s imagine that out of a total test worth 50 points –You want to at a minimum make two observations – then a decision –Let’s also assume that the raw cut has been established as 34 out of 50 About 68% of the points in the test

CLEAR 2008 Annual Conference Anchorage, Alaska A Performance Test Example 1 With 5 stages and a minimum of 2 observations, and a cut of 34: Stage Of Testing Fail Below Pass At or Above 1 (1 st two observations) (third observation) (fourth observation) (Fifth observation)3434

CLEAR 2008 Annual Conference Anchorage, Alaska A Performance Test Example 2 With 5 stages and a minimum of 2 observations, and a cut of 34: Stage Of Testing Fail Below Pass At or Above 1 (1 st two observations) (third observation) (fourth observation) (Fifth observation)3434

CLEAR 2008 Annual Conference Anchorage, Alaska A Performance Test Example 3 With 5 stages and a minimum of 2 observations, and a cut of 34: Stage Of Testing Fail Below Pass At or Above 1 (1 st two observations) (third observation) (fourth observation) (Fifth observation)3434

CLEAR 2008 Annual Conference Anchorage, Alaska Discussion & Reactions

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Mission Statement “To protect the consumer of Therapeutic Recreation Services by promoting the provision of quality services offered by NCTRC certificants”

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Profile NCTRC was incorporated in 1981 as an independent nonprofit organization Internationally recognized credentialing body for therapeutic recreation Accredited in 1993 by National Commission for Certifying Agencies (NCCA) 15,000 member CTRS Registry Approximately 1200 exam candidates per year

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Board Exam Involvement The NCTRC Board: Attended a demo of CMT at Prometric HQ Supported CMT because –Limited exposure of item pool –Overall cost effectiveness –Better value to the candidate Participated in Cut Score Process Appoint Exam Management Committee

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Testing Program Began in 1990 with 200 item written exam Based on Job Analysis of Certified Therapeutic Recreation Specialist (CTRS) Transformed in 2001 to a Linear Computer Based Exam (200 items) Transformed in 2002 to a Computer Mastery Exam

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Exam Administered three times each year 5 day testing window Conducted at Prometric Testing Centers across the US, Canada and Puerto Rico Offered to qualified candidates that have been granted professional eligibility by NCTRC

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Job Analysis Assures test specifications and the exam are related to the practice of Therapeutic Recreation Delineates the important tasks and knowledge deemed necessary for competent practice Job Tasks - practical experience Knowledge Areas - theoretical knowledge Conducted in 1987, 1997, and 2007

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC EMC Exam Management Committee Function: To monitor and make revisions to NCTRC’s testing procedures To work with and monitor the administration of NCTRC’s tests, such administration may be contracted for with private testing services To collect data necessary to periodically check for adverse impact or inadvertent bias To collect data necessary to demonstrate reliability and validity of the testing procedures To ensure reasonable accommodation of testing procedures for individuals with disabilities

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC EMC The responsibilities of the Exam Management Committee include: Item writing committee Item review committee Updating and maintaining exam reference list Review current items in operational pool for overlap and currency Job analysis Update and maintain practice tests

CLEAR 2008 Annual Conference Anchorage, Alaska Customer Preparation Certification standards Conference workshops Exam content outline Practice exam Sample items Reference list

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC CMT Base test consists of 90 multiple choice items (87 minutes) 15 items are pre-test items that are not part of score Depending on performance candidate can receive up to 6 additional testlets (14 minutes) One testlet equals15 items Each testlet is mirror reflection of Exam Content Outline (same proportions as full exam)

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Exam Content Outline Content AreasPercent of Exam No. of Test Items (per testlet) Foundational Knowledge 33.3%5 Practice of TR/RT 46.7%7 Organization of TR/RT 13.3%2 Advancement of the Profession 6.7%1 Total100%15

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Exam Experience Immediate feedback Faster test time Candidate satisfaction Some dissatisfaction and confusion Less exposure to item pool via random assignment of testlets Computer-base exam a “plus” with candidates Positive feedback re: NCTRC prep material

CLEAR 2008 Annual Conference Anchorage, Alaska NCTRC Special Accommodations NCTRC approval process Relatively large percent of special accommodations Advanced registration with designated reservation Ability to offer a wide range of accommodations CBT and CMT conducive to special needs

CLEAR 2008 Annual Conference Anchorage, Alaska Speaker Contact Information F. Jay Breyer, Ph.D. Executive Director of Psychometric Consulting Services Prometric 2000 Lenox Drive Lawrenceville, NJ

CLEAR 2008 Annual Conference Anchorage, Alaska Speaker Contact Information Bob Riley, Ph.D., CTRS NCTRC Executive Director 7 Elmwood Dr New City, NY