ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference A Ten-Year Review of CBT By Casey Marks, Ph.D., NCSBN Geraldine Piskorski, MBA, ASCP Board of Registry Anthony R. Zara, Ph.D., PV Presented at the 2004 CLEAR Annual Conference: 21 st Century Regulation: “Show Me” What Works September 30, 2004

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Session Agenda 1.Speaker backgrounds 2.Before transition to CBT 3.Transition 4.Operations 5.Refinement 6.CBT lessons learned 7.Conclusions

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Speaker Background Casey Marks, NCSBN Geraldine Piskorski, ASCP Board of Registry Anthony Zara, Pearson VUE

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference What We’re Trying to Accomplish 1.Share experiences 2.Compare and contrast large vs. small programs’ issues 3.Share solutions 4.Outline successful practices

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Some Industry Trends 1.Exploding numbers of programs 2.More complicated item types 3.Security concerns 4.Globalization 5.Maturing technologies, e.g., Internet 6.Increased choices for program sponsors

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Issues Before Transition to CBT Small program ASCP/BOR background Size and depth of item bank Variable vs. fixed length test Opportunity to review Size of Examinee population Availability of computer testing facilities Cost of delivery Large program NCSBN background NCLEX facts Pre-CAT research program Barriers?

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Foundational Pre-CBT Research Dimensionality IRT calibration Scale stability Comparability of results Early on, much psychometric research was necessary; now a bit less, but still need to validate findings with your program. Mode of administration – What flavor of CBT? (CAT not right for everyone)

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Operational Research Examinee interface Stopping rules Test length Item pre-testing plan Test timing Candidate impact Beta testing Here is where your program’s specifics get analyzed and determined

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Transition Issues Small program Transition from testing two days each year to year-round testing Staffing Stakeholder communications Large program CBT benefits Create roadmap to success Organizational processes Stakeholder communications

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Benefits of CBT More frequent administrations – (daily testing) Measurement accuracy and efficiency Advanced item types can be used Scheduling on demand Increased security Instant candidate results Professional testing environment Reduced time between retakes

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Roadmap for CBT Needs analysis Stakeholder analysis Building consensus Working decision processes Communications Leverage the experience of others when analyzing your program.

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Organizational Processes Organizational Decision Process – Understanding needs – Options for solutions Contracting Process – RFP or directed proposals – Provider options – Focus on outcomes

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Operations Issues Small program Psychometrics Administration Stakeholder support Large program Batch to continuous model – Test development – Psychometrics – Administration – Stakeholder support – Staffing

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Operations Batch to Continuous: Different Mindset – Test Development Constant pool rotation -- ability to introduce content more quickly More items needed Continuous item development/reviewing Item writer recruitment

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Operations Batch to Continuous: – Psychometrics Constant item maintenance Different item exposure concerns Candidate volume spikes/dips due to content changes and passing standard changes Ability to remove underperforming problematic/underperforming items during administration

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Operations Batch to Continuous: – Test Administration Individual issues/incidents Ongoing test center monitoring Ratio of investigations to candidates increases Time from exam registration to result Time-dependent registration policies –Application process –Fee changes –Rescheduling issues

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Operations Interface with stakeholders – Test accommodations – Communicating program changes – Procedure/documentation changes Stakeholder Support – Direct support – Information/communication Accessible Timely Constant Rumors/urban legends

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Operations Batch to Continuous: – Program Staffing Dedicated Specialized Outsourcing – Customer Service The on-demand mentality Expectations

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference CBT Program Refinement Small program Immediate pass/fail results at test center Providers - more options Quality of graphics Large program Ongoing research Item types Providers Service

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Refinement - Research Ongoing Research – Methodology and content – Staff and/or test service – External Implementation dates seemingly “arbitrary”

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Refinement – Item Types Advanced item types – Media – Stimuli – Response formats

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Refinement - Providers Service Provider Change – Between 1994 and 2002 same CBT provider – New test development and administration provider (2 for 1) in 2002 – New center network – New test driver/candidate interface – Candidate support – Data reconciliation between test services – New application/administration/reporting/results processing systems – Seamless…

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Program Refinement - Service Innovative item types (beyond MCQ) – Utilizing same exam administration parameters More frequent administrations – 2 to 4 to 8 per year Additional testing time – Opportunity to demonstrate competence under optimal exam termination rules International administration – Outside U.S. and territories starting in 2005

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference CBT Lessons Learned Small program Importance of good communication with all stakeholders Size of examinee population Improved benefits to examinees Large program Benefits obtained worth it? Program complexities Stakeholder satisfaction? Service? Etc.

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Challenges Large program, yet perfection is needed/expected daily… – More moving pieces, more expertise needed – Change is harder to integrate – The more complex the program, the more difficult to educate stakeholders – Timing of program changes never convenient for all stakeholders

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference General Conclusions Any large vs. small program things? Any “gotchas”? Need to consider all program stakeholders when considering procedure changes

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Conclusions CAT is a good thing for the NCLEX… – Boards of Nursing enjoy: Convenience and speed of system The ability to focus on exam content not psychometric and/or exam administration – Candidates like: Quick registration, scheduling and results receipt Scheduling options Personalized service – NCSBN appreciates: Security benefits The ability to test content in new and novel ways

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Discussion

ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, CLEAR Conference Contact Information Casey Marks, Ph.D. Director of Testing Services NCSBN Gerry Piskorski, MBA Director, Applicant, Certificant and IT Activities ASCP Anthony R. Zara, Ph.D. Vice President Testing Services Pearson VUE