Open Hearing on Revising the Standards for Educational and Psychological Testing National Council on Measurement in Education March 25, 2008 New York, NY

Participants in Revision Process Management Cmte -Wayne Camara -Suzanne Lane -David Frisbie Technical Cmte –Barbara Plake (co- chair) –Lauress Wise (co- chair)

Presenters Suzanne Lane – Overview of Revision Bob Brennan – Purpose and Use Eva Baker – Educational Policy Michael Kane – Validity Michael Kolen – Linking, Scaling & Equating Robert Linn – Large Scale Assessment Randy Bennett – Current & Future Technology Martha Thurlow – Special Populations Joan Herman – Assessments & Learning Dan Eignor – Reflections and Advice David Frisbie – Moderate questions and answers

2.) Purpose and Use of the Standards in Educational Testing Robert L. Brennan University of Iowa

Some Background Issues Revising the Standards has the distinct advantage of forcing the profession to examine the entire field Educational Measurement should inform the Standards but not constrain or dictate to them. What is missing from the current Standards? –NCLB issues –More attention to linking, growth, accountability, standard setting, and causal inference

Do we have Standards? Clearly, we have an extensive set of guidelines that, in some sense, represent the consensus in the field as of From some perspectives, however, we do not have standards, at least not with teeth, since: –There is no enforcement mechanism –Measurement professionals are not certified or held accountable by any external agency

Standards (page 2) The Standards is based on the premise that effective testing and assessment require that all participants in the testing process possess the knowledge, skills and abilities relative to their role in the testing process...They should also obtain any appropriate, supervised experience…

Have the 1999 Standards led to improved educational measurement? The evidence is mixed at best. Standards are referenced in many places (e.g. RFPs), but it often seems that they are not given serious attention (e.g., technical manuals). Standards are cited in court cases and public documents, but it is not clear how much weight they are given. Aspirational or prescriptive: 1999 Preface seems ambiguous to me.

All Things Considered … Standards seem to serve primarily as guidelines, checklists, and/or instructional material. Should the Standards play a more heavy duty role? I suggest, Yes --- to an extent.

Must do (no excuses) Standards The title of a test, the description of what it measures, and the inferences users are invited to make about test scores should be supported by validity evidence. Appropriate and meaningful reliability evidence should be provided for each and every score reported to any user. For every test, a technical manual should be easily accessible, integrated, and up-to-date, with revisions provided at least every three years.