1 Some Key Points for Test Evaluators and Developers Scott Marion Center for Assessment Eighth Annual MARCES Conference University of Maryland October 11-12, 2007

Marion. Center for Assessment. MARCES Key Points Evaluating the technical quality of AA- AAS must focus on the validity of the test based inferences Specifying the interpretative argument will help create the validity evaluation plan –Consequences are essential Prioritize and get started!

Marion. Center for Assessment. MARCES It’s About Validation The purpose of technical documentation is to provide data for critically evaluating the inferences from the AA-AAS scores and the logic of the interpretative argument

Marion. Center for Assessment. MARCES What about alignment, reliability, comparability, etc? Reliability and comparability might be overrated, especially in the narrow way that reliability has been applied in AA-AAS evaluations Alignment, if pursued at a deep level, can provide useful evidence about the nature of the construct

Marion. Center for Assessment. MARCES Specifying the argument Simply saying that the AA-AAS is being designed to fulfill NCLB and IDEA requirements is inadequate Like a theory of action, the interpretative argument makes the values explicit and specifies the logical connections among the various components of the system, for example: –observed scores will increase for students in response to high quality instruction –the quality of school level instruction will increase as a result of appropriate use of test scores to target professional development –as a result of decisions based on AA-AAS scores, the educational opportunities of students will improve These all lead to important validity inquires

Marion. Center for Assessment. MARCES Consequences are Central With AA-AAS we are trying to balance technical rigor with social justice There is a fundamental belief that educational opportunities for students with significant cognitive disabilities will improve when included in test based accountability systems –If this is a value of the system—and I believe it is for most state AA-AAS systems—than the evaluation of technical quality must include a critical examination of the intended positive and unintended negative effects of the uses of the test scores in this system

Marion. Center for Assessment. MARCES Prioritizing and Focusing Several authors, including Kane (2006), Haertel (1999), Lane (2003), Ryan (2002), and Shepard (1993) have offered suggestions for prioritizing and focusing validity evaluations –There is no “right” way, but many wrong ways States need to develop a validity studies plan, based upon the interpretative argument (or other legitimate organizing framework) to organize both the studies and the synthesis of evidence –Many of these studies—particularly consequential studies—need early planning and some initial data collection

OBSERVATION INTERPRETATION COGNITION  Student Population  Academic Content  Theory of Learning  Assessment System  Test Development  Administration  Scoring  Reporting  Alignment  Item Analysis/DIF/Bias  Measurement Error  Scaling and Equating  Standard Setting VALIDITY EVALUATION  Empirical Evidence  Theory and Logic (argument)  Consequential Features A heuristic to help organize and focus the validity evaluation (Marion, Quenemoen, & Kearns, 2006)