1. University of North Carolina Wilmington
May 1, 2010
Update on the Revisions to the Standards for Educational and Psychological Testing
Antonio E. Puente, Ph.D.
University of North Carolina Wilmington
Joint Committee for the Revision to the Standards for Educational and Psychological Testing
Update on Revisions to the Test Standards

2. Mission & Brief History
“The purpose for publishing the standards is to provide criteria for the evaluation tests, testing practices, and the effects of tests use”. (1999)
Established in 1954; revised in 1955, 1966, 1999 and currently ( )
American Psychological Association, American Educational Research Association, National Council on Measurement in Education
11/19/2018
Update on Revisions to the Test Standards

3. May 1, 2010
Update on the Revisions to the Standards for Educational and Psychological Testing: Overview of the Revision Process
Update on Revisions to the Test Standards 3

4. Update on Revisions to the Test Standards
May 1, 2010
Scope of the Revision
Based on comments each organization received from invitation to comment
Summarized by the Management Committee in consultation with the Co-Chairs
Wayne Camara, Chair, APA
Suzanne Lane, AERA
David Frisbie, NCME
11/19/2018
Update on Revisions to the Test Standards 4
Update on Revisions to the Test Standards

5. Joint Committee Members
May 1, 2010
Joint Committee Members
Lauress Wise, Co Chair, HumRRO
Barbara Plake, Co Chair, University of Nebraska
Linda Cook, ETS
Fritz Drasgow, University of Illinois
Brian Gong, NCIEA
Laura Hamilton, Rand Corporation
Jo-Ida Hansen, University on MN
Joan Herman, UCLA
11/19/2018
Update on Revisions to the Test Standards 5
Update on Revisions to the Test Standards

6. Joint Committee Members
May 1, 2010
Joint Committee Members
Michael Kane, ETS
Michael Kolen, University of Iowa
Antonio E. Puente, UNC-Wilmington
Paul Sackett, University of MN
Nancy Tippins, Valtera Corporation
Walter (Denny) Way, Pearson
Frank Worrell, Univ of CA- Berkeley
11/19/2018
Update on Revisions to the Test Standards 6
Update on Revisions to the Test Standards

7. Five Identified Areas for the Revisions
May 1, 2010
Five Identified Areas for the Revisions
Access/Fairness
Accountability
Technology
Workplace
Format issues
11/19/2018
Update on Revisions to the Test Standards 7
Update on Revisions to the Test Standards

8. Update on Revisions to the Test Standards
May 1, 2010
Timeline
First meeting January, 2009
Three year process for completing text of revision
Release of draft revision following December 2010 JC meeting
Open comment/Organization reviews
Projected publication Summer, 2012
11/19/2018
Update on Revisions to the Test Standards 8
Update on Revisions to the Test Standards

9. Anticipated Review Process
Draft to be released December, 2010
90 Day (+/-) Review Period
Public Portal for Submitting Comments
Sponsoring Organizations with Publicize Opportunity for Comments
Committee will Summarize Key Issues from Reviews and Provide Response(s)
11/19/2018
Update on Revisions to the Test Standards

10. Publication Options Management Committee responsibility
May 1, 2010
Publication Options
Management Committee responsibility
Goal is for electronic access
Pursuing options for Kindle, etc.
Concerns about retaining integrity and financial support for future revision efforts
11/19/2018
May 1, 2010
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 10
Update on Revisions to the Test Standards 10

11. May 1, 2010
Update on the Revisions to the Standards for Educational and Psychological Testing: Fairness for Diverse Populations
Update on Revisions to the Test Standards

12. Update on Revisions to the Test Standards
May 1, 2010
1999 Approach
Standards related to fairness appear throughout many chapters
Concentrated attention in:
Chapter 7: Fairness in Testing and Test Use
Chapter 8: Rights and Responsibilities of Test Takers
Chapter 9: Testing Individuals of Diverse Linguistic Backgrounds
Chapter 10: Testing Individuals with Disabilities
11/19/2018
Update on Revisions to the Test Standards 12
Update on Revisions to the Test Standards

13. Update on Revisions to the Test Standards
May 1, 2010
Committee Charge
Five elements of the charge focused on accommodations/modifications
Impact/differentiation of accommodation and modification
Appropriate selection/use for English learners (ELs) and examinees with disabilities (EwD)
Attention to other groups, e.g., pre-K, older populations
Flagging
Comparability/validity of accommodated scores
One element focused on adequacy and comparability of translations
One element focused on Universal Design
11/19/2018
Update on Revisions to the Test Standards 13
Update on Revisions to the Test Standards

14. Update on Revisions to the Test Standards
May 1, 2010
Revision Response
Fairness is fundamental to test validity: include as foundation chapter
Fairness and access are inseparable
Same principles of fairness and access apply to all individuals and regardless of specific subgroup
From three chapters to a single chapter that describes core principles and standards
Examples drawn from ELs, EwD, and other groups (young children, aging adults adults, etc)
Comments point to applications for specific groups
More particular standards retained where appropriate (e.g., test translations)
In previous versions of the Standards, fairness and the assessment of individuals from specific subgroups of test takers, such as individuals with disabilities and individuals with diverse linguistic and cultural backgrounds, were presented in separate chapters. In the current version of the Standards, these issues are presented in a single chapter to emphasize the fact that the same fairness and accessibility issues are relevant for all individuals and subgroups of the intended population of test takers and that fairness demands that all test takers be treated equitably. In previous versions of the Standards, fairness and the assessment of individuals from specific subgroups of test takers, such as individuals with disabilities and individuals with diverse linguistic and cultural backgrounds, were presented in separate chapters. In the current version of the Standards, these issues are presented in a single chapter to emphasize the fact that the same fairness and accessibility issues are relevant for all individuals and subgroups of the intended population of test takers and that fairness demands that all test takers be treated equitably.
The committee’s response to its charge was guided by a number of concerns.
First and foremost we want to emphasize that fairness in testing, as the standards conceive the issue, is fundamentally a validity issue: that is, tests that are bias do not provide accurate inferences for all individuals and subgroups for whom they are intended.
Secondly, fairness cannot be separated from accessibility; examinees should be able to understand and respond to an assessment without their performance being unduly influenced by individual characteristics that are not relevant to the construct(s) the test is intended to measure. Fairness in testing requires accessibility to the constructs being measured by all members of the testing population: that is, all examinees that the test is intended for should have an unobstructed opportunity to demonstrate their standing on the construct(s) being measured by the assessment. This includes, for example, young children, aging adults, examinees from rural or urban settings, individuals with disabilities and individuals from diverse linguistic and cultural backgrounds.
Thirdly the same principles of access, freedom from bias, accommodation where necessary and safeguarding use apply regardless of the special population which may characterize an individual examinee.
We combined the three chapters; Fairness in Test Use, Testing Individuals with Diverse Linguistic and Cultural Backgrounds, and Testing Individuals with Disabilities, to underscore this point and to emphasize the fact that accessibility is not something that relates only to certain subgroups of the population but is a fundamental right of all members of the intended test taking population.
As we combined the chapters into a unified set of principles, we tried to maintain sensitivity to different populations by providing specific examples across a range of groups, including comments to the standards that highlighted special concerns and even retaining some special standards that may apply only to specific groups, e.g., translation.
I should note that Rights and Responsibilities of Test Takers chapter, which previously was included in the fairness section of the 1999 standards has now been moved to the applications section, in updated form.
11/19/2018
Update on Revisions to the Test Standards 14
Update on Revisions to the Test Standards 14

15. Overview to Fairness Chapter
May 1, 2010
Overview to Fairness Chapter
Section I: General Views of Fairness
Section II: Threats to the Fair and Valid Interpretations of Test Scores
Section III: Minimizing Construct Irrelevant Components Through the Use of Test Design and Testing Adaptations
Section IV: The Standards
11/19/2018
Update on Revisions to the Test Standards 15
Update on Revisions to the Test Standards 15

16. Four Clusters of Standards
May 1, 2010
Four Clusters of Standards
Use test design, development administration and scoring procedures that minimize barriers to valid test interpretations for all individuals.
Conduct studies to examine the validity of test score inferences for the intended examinee population.
Provide appropriate accommodations to remove barriers to the accessibility of the construct measured by the assessment and to the valid interpretation of the assessment scores.
Guard against inappropriate interpretations, use, and/or unintended consequences of test results for individuals or subgroups.
The draft standards have been organized into four themes or clusters:
1. Use test design, development administration and scoring procedures that minimize barriers to valid test interpretations for all individuals.
2. Conduct studies to examine the validity of test score inferences for the intended examinee population.
3. Provide appropriate accommodations and/or modifications to remove barriers to the accessibility of the construct measured by the assessment and to the valid interpretation of the assessment scores.
4. Guard against inappropriate interpretations, use, and/or unintended consequences of test results for individuals or subgroups.
11/19/2018
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 16

17. May 1, 2010
Update on the Revisions to the Standards for Educational and Psychological Testing: Other Issues
Update on Revisions to the Test Standards 17

18. Other Issues in Committee Charge
Increased use of tests for accountability
Impact of technology on testing
Format and style issues
11/19/2018
Update on Revisions to the Test Standards

19. Increased Use of Testing for Accountability
May 1, 2010
Increased Use of Testing for Accountability
Use of tests for accountability has expanded
Most notably in education but also in other areas such as behavioral health
Facilitated by increasing availability of data and analysis tools
Recent and impending federal and state initiatives will likely lead to further expansion
Use of test scores in policy and program evaluations continues to be widespread
Reinforced by groups that fund and evaluate research (e.g., IES, What Works Clearinghouse)
May 1, 2010
11/19/2018
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 19 19
Update on Revisions to the Test Standards 19

20. Organization of Accountability Material
May 1, 2010
Organization of Accountability Material
Chapter on policy uses of tests focuses on use of aggregate scores for accountability and policy
Chapter on educational testing addresses student-level accountability (e.g., promotional gates, high school exit exams) and interim assessment
Validity, reliability, and fairness standards in earlier chapters apply to accountability testing as well
11/19/2018
May 1, 2010
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 20
Update on Revisions to the Test Standards 20

21. Update on Revisions to the Test Standards
May 1, 2010
Impact of Technology
Technological advances are changing the way tests are delivered, scored, interpreted and in some cases, the nature of the tests themselves
The Joint Committee has been charged with considering how technological advances should impact revisions to the Standards
As with the other themes, comments on the standards that related to technology were compiled by the Management Committee and summarized in their charge to the Joint Committee
11/19/2018
Update on Revisions to the Test Standards 21
Update on Revisions to the Test Standards

22. Key Technology Issues Included in our Charge
May 1, 2010
Key Technology Issues Included in our Charge
Reliability & validity of innovative item formats
Validity issues associated with the use of:
Automated scoring algorithms
Automated score reports and interpretations
Security issues for tests delivered over the internet
Issues with web-accessible data, including data warehousing
11/19/2018
Update on Revisions to the Test Standards 22
Update on Revisions to the Test Standards

23. Reliability & Validity of Innovative Item Formats
May 1, 2010
Reliability & Validity of Innovative Item Formats
What special issues exist for innovative items with respect to access and elimination of bias against particular groups? How might the standards reflect these issues?
What steps should the standards suggest with regards to “usability” of innovative items?
What issues will emerge over the next five years related to innovative items/test formats that need to be addressed by the standards?
11/19/2018
Update on Revisions to the Test Standards 23
Update on Revisions to the Test Standards

24. Automated Scoring Algorithms
May 1, 2010
Automated Scoring Algorithms
What level of documentation/disclosure is appropriate and tolerable for automated scoring developers/vendors?
What sorts of evidence seem most important for demonstrating the validity and “reliability” of automated scoring systems?
What issues will emerge over the next five years related to automated scoring systems that need to be addressed by the standards?
11/19/2018
Update on Revisions to the Test Standards 24
Update on Revisions to the Test Standards

25. Automated Score Reports and Interpretation
May 1, 2010
Automated Score Reports and Interpretation
Use of computer for score interpretation
“Actionable” reports (e.g., routing students and teachers to instructional materials and lesson plans based on test results)
Documentation of rationale
Supporting validity evidence
11/19/2018
Update on Revisions to the Test Standards 25
Update on Revisions to the Test Standards

26. May 1, 2010
Format Issues
Consideration of ways to identify of “Priority Standards”
More parallelism between chapter
Tone
Complexity
Technical language
Organization of Chapters
11/19/2018
May 1, 2010
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 26
Update on Revisions to the Test Standards 26

27. Possible Ways to Identify “Priority Standards”
May 1, 2010
Possible Ways to Identify “Priority Standards”
Clustering of Standards into thematic topics
Over-arching Standards/ Guiding Principles
Application Chapters
Connection of standards to previous standards
11/19/2018
May 1, 2010
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 27
Update on Revisions to the Test Standards 27

28. Organization of Chapters
May 1, 2010
Organization of Chapters
1999 Testing Standards
Three sections
Foundation: Validity, Reliability, Test Development, Scaling & Equating, Administration & Scoring, Documentation
Fairness: Fairness, Test Takers Rights and Responsibilities, Disabilities, Linguistic Minorities
Applications: Test Users, Psychological, Educational, Workplace, Policy
11/19/2018
May 1, 2010
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 28
Update on Revisions to the Test Standards 28

29. Possible New Organization
May 1, 2010
Possible New Organization
Section 1:
Core Principles
Validity, Reliability, Fairness
Testing Operations
Test Design and Development, Scaling & Equating, Test Administration & Scoring, Documentation, Test Takers, Test Users
Section 2
: Applications
Psychological, Educational, Workplace,
Policy and Accountability
11/19/2018
May 1, 2010
Update on Revisions to the Test Standards
Update on Revisions to the Test Standards 29
Update on Revisions to the Test Standards 29

30. Additional Important Information
Documentation
Third Party Observers
Standardized Administration/Scoring
Norms
Technicians
Supervision
Diverse Populations
11/19/2018
Update on Revisions to the Test Standards

31. Update on Revisions to the Test Standards
Feedback Requested
Feedback Information
Web-based
Necessary due to its impact
Feedback Timetable
Comment period- Early 2011
Revision period- Mid to late 2011
Publication expected (APA)
Next revision probably in 15 or so years
11/19/2018
Update on Revisions to the Test Standards