1 Wallace Judd APA Testing Standards: RFC

2 The APA Testing Standards: Request for Comment Wallace Judd, Ph.D. President of Authentic Testing Corporation President-Elect of the PTC Webinar Sponsored by the PTC and by Authentic Testing Corporation.

3 The APA Testing Standards: Request for Comment Comments due October 15,

4 TestStandards.org

5 Webinar is organized in 2 parts: Part I. 20-minute intro to the Standards: How they were formulated. Why they’re important. How they’re likely to impact your program. Problems with the current Standards.

6 Webinar is organized in 2 parts: Part II. Forty minutes of discussion… Are these issues relevant to you? What (if anything) do you want to do? What are Next Steps?

7 NCME Session Title

8 What The Standards Address: Importance of the Standards – 30,000 copies sold. Represents testing practice worldwide. Basis for ANSI ISO enforcement Basis for NCCA Accreditation

9 Why Standards Are Important ANSI – required by DOD for security certifications NCCA – required by Maine for state accreditations More coming…

10 Who Created the Standards? 75 Organizations provided input, volunteer time, editing, ideas. They were from: Original Sponsors: AERA, APA, NCME Academic Institutions Non-Profit Associations State & Federal Government Agencies Test Publishers

11 Not a single sponsor or reviewer was from corporate America.

12 Current Standards Reviewed, revised Adopted Published, disseminated 1999.

13 Contents of the Standards The Standards were written for the professional and for the educated layperson

14 The Standards address professional and technical issues of test development and use in education, psychology and employment. (APA.org, home page)

15 The Standards are the Bible of Testing in the U.S.

16 Standards = The Bible

17 Contents of the Standards This book is a vitally important reference for professional test developers, sponsors,sponsors, publishers, users, policymakers, employers,employers, and students ( home page)

18 But the Standards clearly target educational & psychological testing.

19 Few Job-Related Standards Educational – measuring and predicting student achievement Psychological – measuring and evaluating psychological traits Sections on Performance Testing and corporate / industrial testing – minimal.

20 Initial Issues

21 Problem: Bylaws that the governance structure, policies, and procedures that have been established protect against undue influence Standard 2. A. The certifying program must show that the governance structure, policies, and procedures that have been established protect against undue influence that could compromise the integrity of the certification process.

22 Problem: Governance Structure The governance structure must provide for autonomy in decision making Standard 2. B. The governance structure, policies, and procedures must provide for autonomy in decision making regarding important aspects of the certification program, such as eligibility standards; the development, administration, and scoring of the assessment instruments; selection of personnel; and operational processes.

23 Problem: No Training & Testing In the Same Organization the certification agency must not also be responsible for accreditation of educational or training programsStandard 2. D. To avoid conflicts of interest between certification and education functions, the certification agency must not also be responsible for accreditation of educational or training programs or courses of study leading to the certification

24 Problem: Board Composition governing committee of the certification program must include individuals from the certified populationStandard 3. The certification board or governing committee of the certification program must include individuals from the certified population, as well as voting representation from at least one consumer or public member.

25 Problem: No Place for Gating Items Gating Items – Items Required to Pass Test Pilot’s Test – Land the plane Veterinarian’s Test – Spay a cat Crane Operator’s Test – Put ball in barrel Word Test – Save a file System Admin. Test – Add a new user

26 Problem: No Performance Items fit of the model is evaluated … by evaluating unidimensionality, and equality of slope parametersStandard 3. (p. 39) When [using IRT], the fit of the model is evaluated … by evaluating unidimensionality, local independence, speededness, and equality of slope parameters.

27 Problem: Expert Judges demographic characteristics of expert judges should also be documentedStandard 3.6 The qualifications, relevant experience, and demographic characteristics of expert judges should also be documented.

28 Problem: Content Domain document the extent to which the content domain of a test represents the defined domainStandard 3.11 Test developers should document the extent to which the content domain of a test represents the defined domain and test specifications.

29 Problem: Publish Scoring Criteria criteria for scoring …should be provided to the test takers Standard 3.20 When appropriate, …criteria for scoring …should be provided to the test takers prior to the administration of the test.

30 Why Do I Care? I’ve developed tests since 1981 at Xerox, Oracle, Honeywell, McGraw Hill, Kelly Services, and my tests were the foundation for Microsoft’s MOUS tests. I believe that skills testing and certification deserves as much recognition and validation as psychological or educational testing.

31 Why Does the PTC care? It has many members who have testing programs, and who want them to exemplify best practices in testing. PTC meets October in Scottsdale. PTC will formulate its own response to the RFC. PTC is at PerformanceTest.org

32 How important is compliance with the Standards to your program? 1.Insignificant 2.A small factor 3.Significant 4.A large factor 5.Critical

33 How important is NCCA accreditation to your program? 1.Insignificant 2.A small factor 3.Significant 4.A large factor 5.Critical

34 How important is ANSI accreditation to your program? 1.Insignificant 2.A small factor 3.Significant 4.A large factor 5.Critical

35 We have two routes to go. What should we do? 1.Formulate separate standards for corporate-sponsored testing 2.Work with AERA, APA, NCME, to revise the Standards. 3.Both

36 Which would you participate in? (You can vote for more than 1.) 1.Individual corporate response 2.Industry-specific group response 3.Response representing corporations with testing programs

37 Plug: The PTC Where: Scottsdale, Arizona When: October 17 – 19, 2007 Who: –Graham Brent – Crane Operators’ Exam –Ric Luecht – Engineering Assessments –Brian Rice – How Virtualization Works

38 Plug: Negotiating the Standards Workshop October 16 th, PTC Summit in Phoenix.

39 Plug: AuthenticTesting.com Automated Job Task Analysis Automated Exam & Item Analysis Performance Test Design

40 Wallace Judd The Performance Testing Council