Wayne Camara College Board AERA - NCME 2007 Revising the Standards for Educational and Psychological Testing: The Next Generation Wayne Camara College Board AERA - NCME 2007

Performance: College-bound seniors redux The Last Revision (1999) Planning began in early 1992 The Joint Cmte was appointed and first met in 1993 The Cmte was comprised of 15 members and staffed by APA There was a call for comments, conference, and significant input before the first draft was released. Two drafts were distributed for comments and the Joint Cmte incorporated suggested changes in the final document The final draft was completed in late 1998, but there was an extensive legal review and governance review by AERA, APA and NCME which delayed publication another 12 months Sales of about 30,000 copies (through 12/06) vs 62,700 copies of the 1985 Standards (or 51,000 copies through same 8 yearas) Using our taxonomy of college student performance, we developed two measures that are popular in the I/O world: situational judgment tests and biodata. Define: As there is some debate as to whether either of these measures assesses specific constructs, the content for both measures was based on our taxonomy of performance.

SPONSORING ORGANIZATIONS – AERA, APA, NCME MANAGEMENT COMMITTEE Budget/Timeline Scope/Solicit Input Cmte. Appointment/Staffing Price JOINT COMMITTEE Internal procedures or criteria for changes Communicate to sponsoring organizations Technical changes PUBLISHER Promote and fulfill orders Prepare budgets, staff Man. Cmte. Maintain development fund Publish and distribute OTHER (SPONSORS) Financial Contact Support Technical Contact

Master Agreement - Current Standards Executed 12/31/04 Management Cmte – Terms of 3, 4 and 5 yrs with reappointment and Appointed by each sponsoring organization Suzanne Lane (AERA), Barbara Plake (NCME) and Wayne Camara (APA) Will oversee the financial aspects of the publication of the Standards (pricing, ads) Will oversee the Development fund, representing both the interests of the sponsoring organizations and Standards Will make all decisions by unanimous consent (mediation) Travel expenses paid by each organization

Master Agreement – Revised Standards Will confer with the sponsoring associations and determine the need for a revision Once a need is determined, they will appoint representatives from sponsoring organizations to the Joint Committee All details about the charge, terms, and funding of the Joint Committee will be set forth in a separate document that must be appoved by the CEOs of the sponsoring organizations Circumstances of revision (timeline, scope) Will establish pricing, ad schedule, publication strategies Approve expenses above $2,500

Initial Issues – Lessons Learned Spoke with staff, co-chairs, and others involved in the 1999 Revision General perceptions were: Process was too long Devoted too much time and resources to soliciting comments from external groups and individuals Many comments were repetitive and not very helpful Need to establish general scope of revision effort Select members of the cmte to meet needs of areas in need of greatest revision Limit revision to areas where there have been substantive changes (professional practices, technological or scientific advancements, use of assessment)

Decisions Decisions on Process Separate solicitation of input and comments from the technical work Identify major areas in need of revision and select cmte members with corresponding emphasis (as opposed to representing all areas equally) May include individuals employed by a test publisher (expand beyond just academics) Strive for more efficient and quicker revision General consensus that a major revision is not required – only updating – the Devil is in the Details Greater role for associations in reviewing and prioritizing comments Constraints and Sensitivities Cmte and Standards must represent expertise in Educational Assessment (instructional, large scale, admissions, special education), Psychological (career, counseling, clinical, I-O, school, neuropsychological) and other areas (certification, licensure). Ensure adequate representatives with expertise in psychology as well as education Confine membership to manageable number Face validity – credibility of cmte within specialty areas and measurement community Volunteers – limited time on task vs association desire for quick process Scope creep Writing assignments

Thank You!