1. What goes into developing a high-quality credentialing test?

2. Speakers Heidi Leenay Anthony Marini, PhD Chad Kinart, ATC, MBA
Director of Education
OptumHealth
Anthony Marini, PhD
Founder/Psychometrician
Martek
Chad Kinart, ATC, MBA
Director of Partner Services
Assessment Systems

3. Agenda The landscape of psychometrics and credentialing
OptumHealth: How we started out
The Test Development Cycle
OptumHealth: How we achieved our goals

4. Landscape What is our goal?
Assessments that provide accurate information about candidate competence in a profession.
You know the profession and the candidates. We’re here to talk about what it means to “provide accurate information”

5. Landscape First: Definitions Accreditation Credentialing Certification
Licensure
Certificates
Education
Microcredentials
First: Definitions
Certification: Recognition of broad skills/knowledge with extensive assessment. Voluntary for candidate.
Example – Certified Ophthalmic Assistant
Licensure: Recognition of broad skills/knowledge with extensive assessment. Mandatory for candidate. (usually by Govt)
Example – Certified Ophthalmic Assistant (required by law in State X)
Certificates: Narrower of broad skills/knowledge with moderate assessment. Voluntary for candidate.
Example – Certified LASIK assistant (LASIK is obviously a portion of ophthalmology)
Education: Just recognition that you have completed learning, with no indication of mastery.
Example – Completed a 6 month training program in ophthalmic assisting at a community college
Microcredentials: Very narrow recognition of skills/knowledge. Often tied to specific products.
Example – Took course and passed test to fit contact lenses from Brand X
Accreditation

6. Landscape More on Accreditation…
A stamp of approval that you developed your credentialing program aligned with best practices
External to your organization (third party)

7. Landscape More on Accreditation…
A stamp of approval that you developed your credentialing program aligned with best practices
External to your organization (third party)

8. Landscape Accreditation: Who makes the standards?
Additional standards:
AERA/APA/NCME
ITC
Govt (in USA, Uniform Guidelines)

9. Landscape What are in these standards? Policies Test Development
Board Governance
Financials
Test Development
Policies
Recertification
Eligibility
Landscape
What are in these standards?
There’s more than this yet!
We focus on this one aspect today.
It is relevant even if you do not get accredited, because it means you are following industry standards.
This makes your assessment much, much more defensible and effective.

10. OptumHealth: How We Started
Heidi, tell your story here

11. The Test Development Cycle
So, how do we develop a credentialing test that is aligned with best practices?
This section of the presentation will provide an overview of major steps in this process.

12. Item Development Construction and Review
Where do test items/tasks come from?
Defining content for a high stakes exam

13. Step 1: Define what the test covers
Scope of practice
How role fits in profession
High level KSAs needed
Sometimes defined for you?
In some cases, the licensure/govt regulations define this ahead of time

14. Step 1: Define what the test covers
Job Analysis
You can’t just pick the topics
Need real data to drive decisions
Different ways to do this, but a common one is Job Task Analysis

15. Step 1: Define what the test covers
Job Task Analysis
Define domains and tasks in them
Survey incumbents on importance and time spent
Analyze which domains have more tasks and higher ratings

16. Step 1: Define what the test covers
Job Task Analysis: Example
Domain
Tasks
Mean TxI
Sum TxI
Percent A 14
4.21
58.94
24.02 B 11
3.64
40.04
16.32 C 21
3.75
78.75
32.09 D 17
3.98
67.66
27.57
Domain A doesn’t have many tasks, but they are quite important or frequent, so it gets an edge.
Domain C has the most tasks, so it gets the largest weight on the test.

17. Item Development, Construction and Review
Who develops the items/tasks for examinations and how are they selected?
How should writers be selected and what characteristics should they possess?
What conflict of interest issues should be considered in selecting item writers?
What methods of compensation exist for acknowledging the work of writers?

18. Item Development, Construction and Review
What is the nature of training for preparing item/task writers?
What document resources are necessary to support writers?
How do you enhance the sustainability of a writing team?

19. Item Development, Construction and Review
What processes are available to help ensure the relevancy and quality of testing materials?
What practices should be implemented to ensure the security of examination materials during the writing process?

20. Item Development, Construction and Review
How can the principles of universal design help to ensure maximum accessibility?
How can we ensure that important issues such as linguistic complexity are recognized in item development?
If there is a need for a second language what challenges does exam translation create?

21. Item Development, Construction and Review
What contribution can technology make in supporting both item development and exam construction activities?

22. Evaluating Evaluation
Feedback from students
Feedback from item analysis
Difficulty index
Discrimination index

23. Item Analysis Example Prop. Answering Correctly: 0.74
Discrimination Index: 0.42
Point Biserial: 0.40
Alt.
Total Ratio
Low (27%)
High (27%)
Point Biserial
Key A
0.74
0.44
0.86
0.40 * B
0.09
0.17
0.03
-0.13 C
0.04
0.06
0.08
-0.03 D
0.13
0.33
-0.38 E
0.00
Other

24. Item Analysis Example Prop. Answering Correctly: 0.18
Discrimination Index: -0.03
Point Biserial: -0.01
Alt.
Total Ratio
Low (27%)
High (27%)
Point Biserial
Key A
0.34
0.27
0.44
0.23 ? B
0.18
0.22
0.19
-0.01 * C
0.12
0.06
0.17
0.10 D
0.36
0.47
-0.29 E
0.00
Other

25. Standard Setting
It is legally indefensible to pick an arbitrary cutscore like 75%
Also indefensible to be quota-based
Should be criterion-referenced

26. Standard Setting Some of the acceptable methods:
Modified Angoff (most common)
Nedelsky/Ebel (superseded by Angoff)
Bookmark
Contrasting Groups
Borderline Group
Hofstee
Scalemark

27. Standard Setting Modified Angoff: driven by SME panel
Define minimally competent candidate
Rate items on % of MCCs that would get correct
Discuss items with poor agreement
Re-rate
Evaluate the average rating… average score you’d expect from MCC

28. Linking/Equating
If you have more than one form (required for NCCA!) they must be statistically equated
If scores on Form 2 are higher, is it because the items are easier or it happened to have smarter candidates?

29. Linking/Equating Easier items on A Smarter candidates on A Form
Mean score
Mean score on anchors A 72 14 B 70
Form
Mean score
Mean score on anchors A 72 15 B 70 14
100 item test with 20 anchors
First example: same score on anchor items so we know one group is NOT smarter than the other
Second example: Score on anchors is higher so they are smarter

30. Feedback Reports What misconceptions exist What candidates want
What can we offer and what disclaimers are required

31. Feedback Report for MCQ

32. Feedback Report for OSCE

33. Accommodations
Providing accommodations in accordance with the Americans with Disabilities Act (ADA), Accessibility for Ontario with Disabilities Act (AODA)
Documentation
Common accommodations
Other accommodations
On site requirements

34. Retake Policy The current landscape (limited vs unlimited)
From the candidates perspective
From the examining board or regulatory perspective
From the psychometric perspective

35. OptumHealth: How We Achieved Our Goals
Heidi, tell your story here

36. Speaker Contact Information
Anthony Marini, Ph.D.
Senior Faculty Developer, Carleton University
President, Martek Assessments Ltd.
Ottawa Ontario, Canada