1. Wayne Camara College Board AERA - NCME 2007
Revising the Standards for Educational and Psychological Testing: The Next Generation
Wayne Camara
College Board
AERA - NCME 2007

2. Performance: College-bound seniors redux
The Last Revision (1999)
Planning began in early 1992
The Joint Cmte was appointed and first met in 1993
The Cmte was comprised of 15 members and staffed by APA
There was a call for comments, conference, and significant input before the first draft was released.
Two drafts were distributed for comments and the Joint Cmte incorporated suggested changes in the final document
The final draft was completed in late 1998, but there was an extensive legal review and governance review by AERA, APA and NCME which delayed publication another 12 months
Sales of about 30,000 copies (through 12/06) vs 62,700 copies of the 1985 Standards (or 51,000 copies through same 8 yearas)
Using our taxonomy of college student performance, we developed two measures that are popular in the I/O world: situational judgment tests and biodata.
Define:
As there is some debate as to whether either of these measures assesses specific constructs, the content for both measures was based on our taxonomy of performance.

3. SPONSORING ORGANIZATIONS – AERA, APA, NCME
MANAGEMENT COMMITTEE
Budget/Timeline
Scope/Solicit Input
Cmte. Appointment/Staffing
Price
JOINT COMMITTEE
Internal procedures or criteria for changes
Communicate to sponsoring organizations
Technical changes
PUBLISHER
Promote and fulfill orders
Prepare budgets, staff Man. Cmte.
Maintain development fund
Publish and distribute
OTHER (SPONSORS)
Financial Contact
Support
Technical Contact

4. Master Agreement - Current Standards
Executed 12/31/04
Management Cmte – Terms of 3, 4 and 5 yrs with reappointment and Appointed by each sponsoring organization
Suzanne Lane (AERA), Barbara Plake (NCME) and Wayne Camara (APA)
Will oversee the financial aspects of the publication of the Standards (pricing, ads)
Will oversee the Development fund, representing both the interests of the sponsoring organizations and Standards
Will make all decisions by unanimous consent (mediation)
Travel expenses paid by each organization

5. Master Agreement – Revised Standards
Will confer with the sponsoring associations and determine the need for a revision
Once a need is determined, they will appoint representatives from sponsoring organizations to the Joint Committee
All details about the charge, terms, and funding of the Joint Committee will be set forth in a separate document that must be appoved by the CEOs of the sponsoring organizations
Circumstances of revision (timeline, scope)
Will establish pricing, ad schedule, publication strategies
Approve expenses above $2,500

6. Initial Issues – Lessons Learned
Spoke with staff, co-chairs, and others involved in the 1999 Revision
General perceptions were:
Process was too long
Devoted too much time and resources to soliciting comments from external groups and individuals
Many comments were repetitive and not very helpful
Need to establish general scope of revision effort
Select members of the cmte to meet needs of areas in need of greatest revision
Limit revision to areas where there have been substantive changes (professional practices, technological or scientific advancements, use of assessment)

7. Decisions Decisions on Process
Separate solicitation of input and comments from the technical work
Identify major areas in need of revision and select cmte members with corresponding emphasis (as opposed to representing all areas equally)
May include individuals employed by a test publisher (expand beyond just academics)
Strive for more efficient and quicker revision
General consensus that a major revision is not required – only updating – the Devil is in the Details
Greater role for associations in reviewing and prioritizing comments
Constraints and Sensitivities
Cmte and Standards must represent expertise in Educational Assessment (instructional, large scale, admissions, special education), Psychological (career, counseling, clinical, I-O, school, neuropsychological) and other areas (certification, licensure).
Ensure adequate representatives with expertise in psychology as well as education
Confine membership to manageable number
Face validity – credibility of cmte within specialty areas and measurement community
Volunteers – limited time on task vs association desire for quick process
Scope creep
Writing assignments

8. Thank You!