Overview of Standard Setting Leslie Wilson Assistant State Superintendent Accountability and Assessment August 26, 2008

Purpose of Presentation To present the standard setting process To provide background information to the Board in preparation for setting standards for – Modified High School Assessments – Alternative Maryland School Assessments

Setting Performance Standards Content Standards Voluntary State Curriculum The Tests Expertise: Student Knowledge/ Subject Matter Setting Performance Standards Cut Scores that match students to their appropriate Performance Categories

The Threshold Student A borderline or minimally qualified student – In terms of performance The least qualified of the Proficient or Advanced students

Mod-HSA Test Performance Standards Basic Recommended Standards threshold students ProficientAdvanced

Standard Setting Work Flow Content Groups Psychometric Council Review & Articulation Committee State Superintendent State Board of Education Set Standards

Step 1: Content Groups Participants all have appropriate content and grade-level expertise Include representatives from local Central Offices as well as school level staff Participate in separate content and grade level discussions

Responsibilities of Content Group Review Assessment Discuss what barely proficient and advanced students know and can do Propose cut scores and discuss Make recommendations to Review and Articulation group

Content Group Process: Round 1 Break into small groups to encourage independent decision-making Read, discuss ordered-item booklets Individuals vote (privately) for a cut score Small group discussion of votes

The Process: Round 2 Individuals vote (privately) for a cut score Small group discussion of votes Examine, discuss impact data

The Process: Round 3 Small groups converge Individuals vote (privately) for a cut score Discussion Individuals vote (privately) for a cut score The median cut score is the committee’s recommendation.

Bookmark Standard-Setting Procedure Test items students must know and be able to do to be considered Proficient Test items students must know and be able to do to be considered Advanced A P Ordered Item Booklet

Standard-Setting Process Individual Analysis Items most students get correct Items fewer students get correct Content every student should know

Review test; vote Round 1 Discuss data and distribution Median Standard-Setting Process Round 2 View impact data and discuss Vote Median

Standard-Setting Process Small groups join; vote Round 3 Median cut score is committee’s recommendation Median

Step 2: Psychometric Council Nationally-recognized experts Trained in measurement and psychometrics Independent Familiar with Maryland tests Similar to Technical Advisory Panels in other states

Psychometric Council Responsibilities Reviews work of all groups Ensures quality controls were followed Ensures standard-setting process technically sound Does not make judgments on whether standard is “good” or “bad” Forwards comments and recommendations to Review & Articulation Committee

Step 3: Review and Articulation Superintendents Assistant Superintendents Local Accountability Coordinators Advocates for students with special needs (special education and ELL) Parents Local Boards of Education Teachers Union representatives Business Community members MSDE

Review & Articulation Committee Responsibilities include: – Reviewing work of all groups and Psychometric Council – Ensuring rigor is equivalent among grades – Reviewing articulation among grades – Forwarding recommendations to the State Superintendent

Step 4: State Superintendent of Schools Reviews work of all groups, Psychometric Council, and Review & Articulation Committee Makes recommendation to the State Board of Education

Step 5: State Board Approval Reviews proposed standards Approves standards for implementation

Next Steps 1.Review and approve proposed standards to be used to determine proficient and advanced performance on the modified HSAs. The scores will be used to determine AYP for Maryland high schools next year. 2.Review and approve proposed standards for the Alt-MSA. Science will be reported this fall for the first time.

Overview of Standard Setting Questions and Discussion