1. The Jimmy A Young Memorial Lecture July 17, 2014 7:00 to 8:30 AM Marco Island, FL 1

2. The NBRC has honored Jimmy’s memory and the contributions he made to respiratory care through this program since 1978. Jimmy Albert Young, MS, RRT 1935 –1975 2

3. Jimmy Albert Young, MS, RRT was one of the profession’s most outstanding and dedicated leaders – 1935 – born in South Carolina – 1960 – 1966 – served as Chief Inhalation Therapist at the Peter Bent Brigham Hospital in Boston – 1965 – earned the RRT credential, Registry #263 – 1966 – 1970 – served in several roles including director of the education program at Northeastern University in Boston – 1970 – became director of the Respiratory Therapy Department at Massachusetts General Hospital – 1973 – became the 22 nd President of the American Association of Respiratory Care – 1975 – was serving as an NBRC Trustee and member of the Executive Committee when he passed away unexpectedly In a 15-year career, -achieved the RRT -directed an education program -directed a hospital department -served as AARC President -served as an NBRC trustee 3

4. The Clinical Simulation Examination Then and Now 4

5. Presenter Robert C Shaw Jr PhD RRT FAARC – NBRC Assistant Executive Director and Psychometrician 5

6. Conflict of Interest I have no real or perceived conflicts of interest that relate to this presentation. Any use of brand names is not meant to endorse a specific product, but to merely illustrate a point of emphasis.

7. Learning Objectives Compare elements of the current RRT credentialing system to elements of the system that is planned for January 2015 Compare the value of information that has been provided by results from the Clinical Simulation Examination to other elements of the RRT credentialing system Describe features of the 20-problem Clinical Simulation Examination for which candidates should be prepared by January 2015

8. OBJECTIVE 1 Compare elements of the current RRT credentialing system to elements of the system that is planned for January 2015

9. Compare Current to Future RRT Program Elements Number Current January 2015 Hours of testing time97 Examinations32 Testing sessions for a candidate who passes on first attempts32 Sets of test scores42 Passing points43 Multiple-choice items to assess competencies broadly240140 Patient management problems to assess competencies deeply1020 Typical number of sections in a patient management problem10 (8-12)5 (4-6)

10. OBJECTIVE 2 Compare the value of information that has been provided by results from the Clinical Simulation Examination to other elements of the RRT credentialing system

11. QUESTION Is there a measurement reason for the Clinical Simulation Examination to exist?

12. RESEARCH HYPOTHESIS 1 Scores from the Clinical Simulation Examination added information beyond the information from multiple-choice examination scores when predicting membership in three groups for candidates who sought the RRT credential.

13. What defined the three groups of candidates? Credential StatusLabel for group Examination Outcome CRTWritten RegistryClinical Simulation CRT certification pass fail certification +1passfail pass RRTregistrationpass

14. Defining the Population Date range for examination attempts – October 22, 2009 through February 27, 2012 A subset of 9,081 candidates had achieved CRT and made a first attempt at the remaining examinations for RRT (and were not outlying cases) – Written Registry – Clinical Simulation Information gathering (IG) Decision making (DM)

15. Statistical Model and Method Step-wise discriminant analysis with automatic variable selection – Predict group membership from multiple variables, each of which is continuously distributed – Dependent variable certification, certification+1, and registration groups – Independent variables First run included four sets of scores – CRT, Written Registry, Clin Sim IG, and Clin Sim DM Second run included two scores – CRT and Written Registry

16. Standardizing Examination Scores Raw score ranges – CRT = 0 to 140 – Written Registry = 0 to 100 – Clinical Simulation, varied by test form IG = a variable-min to a max in the range of 200-300 DM = a variable-min to a max in the range of 140-170 Each raw score was converted to a z-score where z = (x – mean) / S

17. Results from Run 1 StepTest Scores Wilk’s Lambda - Proportion of unexplained variance F Test for Entry Valuep 1Clinical Simulation DM.4216251.4<.0001 2Written Registry.2963803.5<.0001 3Clinical Simulation IG.2952540.7<.0001 4 CRT.2951907.8<.0001 Predictions about memberships in the registration group were accurate for 92.4% of the cases

18. Discriminant Score Equation Discriminant score = 1.026 (Clin Sim DM z-score) + 0.975 (Written Registry z-score) + 0.091 (CRT z-score) -0.010 (Clin Sim IG z-score) -0.689 Clin Sim DM and Written Registry scores were nearly equal and the dominant contributors to predictions about group memberships

19. Results from Run 2 Predictions about memberships in the registration group were accurate for 85.4% of the cases Step Test Scores Wilk’s Lambda - Proportion of unexplained variance F Test for Entry Valuep 1Written Registry.4475607.3<.0001 2CRT.4332361.6<.0001.42.30 92.4%

20. Conclusions The research hypothesis was accepted – Scores from the Clinical Simulation Examination add information about RRT achievement beyond what is available from multiple-choice examination scores If the Clinical Simulation Examination was removed from the system, there would be a 7% loss of accurate RRT classifications – Incompetent candidates would become RRT – Competent candidates would be denied RRT

21. RESEARCH HYPOTHESIS 2 Although there were four sets of test scores, three tests, and two types of tests, RRT competencies were based on only one cognitive construct.

22. Examination Type Characteristics CharacteristicMultiple-ChoiceClinical Simulation Option-response scoringdichotomous (0 or 1)polytomous (-3 to 3) Linkages between stimulus-response elements independent itemsindependent problems, dependent sections Potential for branching units to which a subset of candidates are directed noyes Cut point determination method external to test development Integrated with test development Cost to produce$$$

23. Risks from Using Multiple Examinations with Different Characteristics SimulationMultiple- Choice Type of Examination AdvancedEntry Level of Examination

24. Statistical Model and Method Principal components analysis with cross- validation – Explore the underlying variance structure within four sets of test scores CRT Written Registry Clinical Simulation – IG – DM – Is useful for confirming a hypothesis, in this case the assertion that there is a common characteristic expressed by the four test scores

25. Preliminary Result 1 As an indicator of sampling adequacy -KMO should be at least.50 -Sig value should indicate statistical significance As indicators of positive cross- validation -KMO values should be about the same Samples whole random split 1 random split 2 sample size9,0814,5574,224 Kaiser-Meyer-Olkin Measure of Sampling Adequacy.775.777.772 Chi- Square 18259.569358.258913.45 Bartlett's Test of Sphericity df666 Sig..000

26. Preliminary Result 2 As indicators of making a sufficient contribution to the principal component solution -Communality values should be at least.50, otherwise a variable should be removed As indicators of positive cross- validation -Values across each row should be similar

27. Primary Result The threshold for a consequential eigenvalue is 1.0 or Components at the inflection point and beyond lack consequence

28. Conclusions -The research hypothesis was accepted There was only one principal component to which all four sets of test scores were linked -Potential risks associated with using a multiple- examination system were avoided

29. Summary from Both Studies Within the population of new RRTs each year, accurate classifications occur more often because there are multiple examinations Risks associated with a credentialing system based on multiple examinations were avoided

30. Study Limitations These were population studies involving a recent period of more than 2 years Unless characteristics of candidates or examinations change, I expect these results will generalize into the future – Candidates: program admission criteria, program duration, program intensity – Examinations: number of instruments, types of measurements

31. OBJECTIVE 3 Describe features of the 20-problem Clinical Simulation Examination for which candidates should be prepared by January 2015

32. Rationale for Changing the Simulation Examination Instant scoring demands selection of problems for each new test form that have not changed – After a decade, keeping examination content current became an increasing challenge 32

33. Solution Give the examination committee smaller content elements from which test forms are assembled – Halve the number of sections in problems – Double the number of problems Hold testing time the same at 4 hours 33

34. ENHANCE PSYCHOMETRIC PROPERTIES As long as other changes will be made... 34

35. Standardize Test Forms More Thoroughly 35 Type of Problem Specifications Current 10-ProblemFuture 20-Problem A1. COPD Conservative Care1 or 22 A2. COPD Critical Care1 or 22 B. Trauma1 or 23 C. Cardiovascular1 or 23 D. Neurological / Neurosurgical1 or 22 E. Pediatric12 F. Neonatal12 G. General Medical / Surgicaloptional4

36. Problems Each Candidate Will See 4 about COPD 4 about children 4 about general medical / surgical 3 about trauma 3 about cardiovascular 2 about neuro – Likely one neuromuscular and one neurologic

37. Simulation Examination Scores

38. Advantages of a one score and one cut system A test with more items and more points than its predecessors will yield more accurate scores as indicators of candidates’ abilities – Pass and fail decisions become more accurate Accuracy is gained without an increase in test administration time – Fee for the Clinical Simulation Examination stays the same

39. A Potential Disadvantage of a Combined Score Compensation can occur unless the cut score policy is changed – Someone within a few points of passing based on decision making performance could pass by acquiring a higher percentage of available information gathering points

40. New Cut Score Policy Section Type Cut Score Range CurrentNew DM60% to 70% IG60% to 70%77% to 81% The cut score for a test form must be the sum of MPLs from the two types of sections such that those section MPLs fall within the two ranges shown in the table Implementation has mandated addition of options labeled as required among positively-scored options in IG sections

41. Conforming to the Policy One IG Section OptionCurrentJanuary 2015 1-2 211R 32R 4 52R 6 7-2 811 92R 10-2 1111 12 MPL, Max, %6, 9, 67%7, 9, 78%

42. WHY THE CUT SCORE POLICY CHANGE MATTERS Illustrations that follow came from one test form

43. MPL range remains 60%-70%

44. MPL range has been 60%-70%

45. Pass No case in this quadrant

46. People in this quadrant would pass under the current system Pass

47. Highlights for Students The numbers of problems by patient type will be constant for each candidate Testing time remains 4 hours – 22 problems will be presented – Results will be based on responses to 20 problems As a result of a problem-splitting procedure – Some problems will not offer IG sections – Candidates will see the same number of IG sections across the whole examination as they currently see

48. Highlights for Students (cont.) Responses will be summed across IG and DM sections that a candidate enters to produce one score to which a cut score will be compared – The cut will equal the sum of MPL values across sections along the critical path Compared to the current examination, responses in IG sections will be consequential – Reduced tolerance for errors

49. QUESTIONS