University of North Carolina Wilmington May 1, 2010 Update on the Revisions to the Standards for Educational and Psychological Testing Antonio E. Puente, Ph.D. University of North Carolina Wilmington Joint Committee for the Revision to the Standards for Educational and Psychological Testing www.apa.org/science/programs/testing/standards.aspx Update on Revisions to the Test Standards

Mission & Brief History “The purpose for publishing the standards is to provide criteria for the evaluation tests, testing practices, and the effects of tests use”. (1999) Established in 1954; revised in 1955, 1966, 1999 and currently (2009-2011) American Psychological Association, American Educational Research Association, National Council on Measurement in Education 11/19/2018 Update on Revisions to the Test Standards

May 1, 2010 Update on the Revisions to the Standards for Educational and Psychological Testing: Overview of the Revision Process Update on Revisions to the Test Standards 3

Update on Revisions to the Test Standards May 1, 2010 Scope of the Revision Based on comments each organization received from invitation to comment Summarized by the Management Committee in consultation with the Co-Chairs Wayne Camara, Chair, APA Suzanne Lane, AERA David Frisbie, NCME 11/19/2018 Update on Revisions to the Test Standards 4 Update on Revisions to the Test Standards

Joint Committee Members May 1, 2010 Joint Committee Members Lauress Wise, Co Chair, HumRRO Barbara Plake, Co Chair, University of Nebraska Linda Cook, ETS Fritz Drasgow, University of Illinois Brian Gong, NCIEA Laura Hamilton, Rand Corporation Jo-Ida Hansen, University on MN Joan Herman, UCLA 11/19/2018 Update on Revisions to the Test Standards 5 Update on Revisions to the Test Standards

Joint Committee Members May 1, 2010 Joint Committee Members Michael Kane, ETS Michael Kolen, University of Iowa Antonio E. Puente, UNC-Wilmington Paul Sackett, University of MN Nancy Tippins, Valtera Corporation Walter (Denny) Way, Pearson Frank Worrell, Univ of CA- Berkeley 11/19/2018 Update on Revisions to the Test Standards 6 Update on Revisions to the Test Standards

Five Identified Areas for the Revisions May 1, 2010 Five Identified Areas for the Revisions Access/Fairness Accountability Technology Workplace Format issues 11/19/2018 Update on Revisions to the Test Standards 7 Update on Revisions to the Test Standards

Update on Revisions to the Test Standards May 1, 2010 Timeline First meeting January, 2009 Three year process for completing text of revision Release of draft revision following December 2010 JC meeting Open comment/Organization reviews Projected publication Summer, 2012 11/19/2018 Update on Revisions to the Test Standards 8 Update on Revisions to the Test Standards

Anticipated Review Process Draft to be released December, 2010 90 Day (+/-) Review Period Public Portal for Submitting Comments Sponsoring Organizations with Publicize Opportunity for Comments Committee will Summarize Key Issues from Reviews and Provide Response(s) 11/19/2018 Update on Revisions to the Test Standards

Publication Options Management Committee responsibility May 1, 2010 Publication Options Management Committee responsibility Goal is for electronic access Pursuing options for Kindle, etc. Concerns about retaining integrity and financial support for future revision efforts 11/19/2018 May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 10 Update on Revisions to the Test Standards 10

May 1, 2010 Update on the Revisions to the Standards for Educational and Psychological Testing: Fairness for Diverse Populations Update on Revisions to the Test Standards

Update on Revisions to the Test Standards May 1, 2010 1999 Approach Standards related to fairness appear throughout many chapters Concentrated attention in: Chapter 7: Fairness in Testing and Test Use Chapter 8: Rights and Responsibilities of Test Takers Chapter 9: Testing Individuals of Diverse Linguistic Backgrounds Chapter 10: Testing Individuals with Disabilities 11/19/2018 Update on Revisions to the Test Standards 12 Update on Revisions to the Test Standards

Update on Revisions to the Test Standards May 1, 2010 Committee Charge Five elements of the charge focused on accommodations/modifications Impact/differentiation of accommodation and modification Appropriate selection/use for English learners (ELs) and examinees with disabilities (EwD) Attention to other groups, e.g., pre-K, older populations Flagging Comparability/validity of accommodated scores One element focused on adequacy and comparability of translations One element focused on Universal Design 11/19/2018 Update on Revisions to the Test Standards 13 Update on Revisions to the Test Standards

Update on Revisions to the Test Standards May 1, 2010 Revision Response Fairness is fundamental to test validity: include as foundation chapter Fairness and access are inseparable Same principles of fairness and access apply to all individuals and regardless of specific subgroup From three chapters to a single chapter that describes core principles and standards Examples drawn from ELs, EwD, and other groups (young children, aging adults adults, etc) Comments point to applications for specific groups More particular standards retained where appropriate (e.g., test translations) In previous versions of the Standards, fairness and the assessment of individuals from specific subgroups of test takers, such as individuals with disabilities and individuals with diverse linguistic and cultural backgrounds, were presented in separate chapters. In the current version of the Standards, these issues are presented in a single chapter to emphasize the fact that the same fairness and accessibility issues are relevant for all individuals and subgroups of the intended population of test takers and that fairness demands that all test takers be treated equitably. In previous versions of the Standards, fairness and the assessment of individuals from specific subgroups of test takers, such as individuals with disabilities and individuals with diverse linguistic and cultural backgrounds, were presented in separate chapters. In the current version of the Standards, these issues are presented in a single chapter to emphasize the fact that the same fairness and accessibility issues are relevant for all individuals and subgroups of the intended population of test takers and that fairness demands that all test takers be treated equitably. The committee’s response to its charge was guided by a number of concerns. First and foremost we want to emphasize that fairness in testing, as the standards conceive the issue, is fundamentally a validity issue: that is, tests that are bias do not provide accurate inferences for all individuals and subgroups for whom they are intended. Secondly, fairness cannot be separated from accessibility; examinees should be able to understand and respond to an assessment without their performance being unduly influenced by individual characteristics that are not relevant to the construct(s) the test is intended to measure. Fairness in testing requires accessibility to the constructs being measured by all members of the testing population: that is, all examinees that the test is intended for should have an unobstructed opportunity to demonstrate their standing on the construct(s) being measured by the assessment. This includes, for example, young children, aging adults, examinees from rural or urban settings, individuals with disabilities and individuals from diverse linguistic and cultural backgrounds. Thirdly the same principles of access, freedom from bias, accommodation where necessary and safeguarding use apply regardless of the special population which may characterize an individual examinee. We combined the three chapters; Fairness in Test Use, Testing Individuals with Diverse Linguistic and Cultural Backgrounds, and Testing Individuals with Disabilities, to underscore this point and to emphasize the fact that accessibility is not something that relates only to certain subgroups of the population but is a fundamental right of all members of the intended test taking population. As we combined the chapters into a unified set of principles, we tried to maintain sensitivity to different populations by providing specific examples across a range of groups, including comments to the standards that highlighted special concerns and even retaining some special standards that may apply only to specific groups, e.g., translation. I should note that Rights and Responsibilities of Test Takers chapter, which previously was included in the fairness section of the 1999 standards has now been moved to the applications section, in updated form. 11/19/2018 Update on Revisions to the Test Standards 14 Update on Revisions to the Test Standards 14

Overview to Fairness Chapter May 1, 2010 Overview to Fairness Chapter Section I: General Views of Fairness Section II: Threats to the Fair and Valid Interpretations of Test Scores Section III: Minimizing Construct Irrelevant Components Through the Use of Test Design and Testing Adaptations Section IV: The Standards 11/19/2018 Update on Revisions to the Test Standards 15 Update on Revisions to the Test Standards 15

Four Clusters of Standards May 1, 2010 Four Clusters of Standards Use test design, development administration and scoring procedures that minimize barriers to valid test interpretations for all individuals. Conduct studies to examine the validity of test score inferences for the intended examinee population. Provide appropriate accommodations to remove barriers to the accessibility of the construct measured by the assessment and to the valid interpretation of the assessment scores. Guard against inappropriate interpretations, use, and/or unintended consequences of test results for individuals or subgroups. The draft standards have been organized into four themes or clusters: 1. Use test design, development administration and scoring procedures that minimize barriers to valid test interpretations for all individuals. 2. Conduct studies to examine the validity of test score inferences for the intended examinee population. 3. Provide appropriate accommodations and/or modifications to remove barriers to the accessibility of the construct measured by the assessment and to the valid interpretation of the assessment scores. 4. Guard against inappropriate interpretations, use, and/or unintended consequences of test results for individuals or subgroups. 11/19/2018 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 16

May 1, 2010 Update on the Revisions to the Standards for Educational and Psychological Testing: Other Issues Update on Revisions to the Test Standards 17

Other Issues in Committee Charge Increased use of tests for accountability Impact of technology on testing Format and style issues 11/19/2018 Update on Revisions to the Test Standards

Increased Use of Testing for Accountability May 1, 2010 Increased Use of Testing for Accountability Use of tests for accountability has expanded Most notably in education but also in other areas such as behavioral health Facilitated by increasing availability of data and analysis tools Recent and impending federal and state initiatives will likely lead to further expansion Use of test scores in policy and program evaluations continues to be widespread Reinforced by groups that fund and evaluate research (e.g., IES, What Works Clearinghouse) May 1, 2010 11/19/2018 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 19 19 Update on Revisions to the Test Standards 19

Organization of Accountability Material May 1, 2010 Organization of Accountability Material Chapter on policy uses of tests focuses on use of aggregate scores for accountability and policy Chapter on educational testing addresses student-level accountability (e.g., promotional gates, high school exit exams) and interim assessment Validity, reliability, and fairness standards in earlier chapters apply to accountability testing as well 11/19/2018 May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 20 Update on Revisions to the Test Standards 20

Update on Revisions to the Test Standards May 1, 2010 Impact of Technology Technological advances are changing the way tests are delivered, scored, interpreted and in some cases, the nature of the tests themselves The Joint Committee has been charged with considering how technological advances should impact revisions to the Standards As with the other themes, comments on the standards that related to technology were compiled by the Management Committee and summarized in their charge to the Joint Committee 11/19/2018 Update on Revisions to the Test Standards 21 Update on Revisions to the Test Standards

Key Technology Issues Included in our Charge May 1, 2010 Key Technology Issues Included in our Charge Reliability & validity of innovative item formats Validity issues associated with the use of: Automated scoring algorithms Automated score reports and interpretations Security issues for tests delivered over the internet Issues with web-accessible data, including data warehousing 11/19/2018 Update on Revisions to the Test Standards 22 Update on Revisions to the Test Standards

Reliability & Validity of Innovative Item Formats May 1, 2010 Reliability & Validity of Innovative Item Formats What special issues exist for innovative items with respect to access and elimination of bias against particular groups? How might the standards reflect these issues? What steps should the standards suggest with regards to “usability” of innovative items? What issues will emerge over the next five years related to innovative items/test formats that need to be addressed by the standards? 11/19/2018 Update on Revisions to the Test Standards 23 Update on Revisions to the Test Standards

Automated Scoring Algorithms May 1, 2010 Automated Scoring Algorithms What level of documentation/disclosure is appropriate and tolerable for automated scoring developers/vendors? What sorts of evidence seem most important for demonstrating the validity and “reliability” of automated scoring systems? What issues will emerge over the next five years related to automated scoring systems that need to be addressed by the standards? 11/19/2018 Update on Revisions to the Test Standards 24 Update on Revisions to the Test Standards

Automated Score Reports and Interpretation May 1, 2010 Automated Score Reports and Interpretation Use of computer for score interpretation “Actionable” reports (e.g., routing students and teachers to instructional materials and lesson plans based on test results) Documentation of rationale Supporting validity evidence 11/19/2018 Update on Revisions to the Test Standards 25 Update on Revisions to the Test Standards

May 1, 2010 Format Issues Consideration of ways to identify of “Priority Standards” More parallelism between chapter Tone Complexity Technical language Organization of Chapters 11/19/2018 May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 26 Update on Revisions to the Test Standards 26

Possible Ways to Identify “Priority Standards” May 1, 2010 Possible Ways to Identify “Priority Standards” Clustering of Standards into thematic topics Over-arching Standards/ Guiding Principles Application Chapters Connection of standards to previous standards 11/19/2018 May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 27 Update on Revisions to the Test Standards 27

Organization of Chapters May 1, 2010 Organization of Chapters 1999 Testing Standards Three sections Foundation: Validity, Reliability, Test Development, Scaling & Equating, Administration & Scoring, Documentation Fairness: Fairness, Test Takers Rights and Responsibilities, Disabilities, Linguistic Minorities Applications: Test Users, Psychological, Educational, Workplace, Policy 11/19/2018 May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 28 Update on Revisions to the Test Standards 28

Possible New Organization May 1, 2010 Possible New Organization Section 1: Core Principles Validity, Reliability, Fairness Testing Operations Test Design and Development, Scaling & Equating, Test Administration & Scoring, Documentation, Test Takers, Test Users Section 2 : Applications Psychological, Educational, Workplace, Policy and Accountability 11/19/2018 May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 29 Update on Revisions to the Test Standards 29

Additional Important Information Documentation Third Party Observers Standardized Administration/Scoring Norms Technicians Supervision Diverse Populations 11/19/2018 Update on Revisions to the Test Standards

Update on Revisions to the Test Standards Feedback Requested Feedback Information Web-based Necessary due to its impact Feedback Timetable Comment period- Early 2011 Revision period- Mid to late 2011 Publication expected- 2012 (APA) Next revision probably in 15 or so years 11/19/2018 Update on Revisions to the Test Standards