1. Overview of Standard Setting Leslie Wilson Assistant State Superintendent Accountability and Assessment August 26, 2008

2. Purpose of Presentation To present the standard setting process To provide background information to the Board in preparation for setting standards for – Modified High School Assessments – Alternative Maryland School Assessments

3. Setting Performance Standards Content Standards Voluntary State Curriculum The Tests Expertise: Student Knowledge/ Subject Matter Setting Performance Standards Cut Scores that match students to their appropriate Performance Categories

4. The Threshold Student A borderline or minimally qualified student – In terms of performance The least qualified of the Proficient or Advanced students

5. Mod-HSA Test Performance Standards Basic Recommended Standards threshold students ProficientAdvanced

6. Standard Setting Work Flow Content Groups Psychometric Council Review & Articulation Committee State Superintendent State Board of Education Set Standards

7. Step 1: Content Groups Participants all have appropriate content and grade-level expertise Include representatives from local Central Offices as well as school level staff Participate in separate content and grade level discussions

8. Responsibilities of Content Group Review Assessment Discuss what barely proficient and advanced students know and can do Propose cut scores and discuss Make recommendations to Review and Articulation group

9. Content Group Process: Round 1 Break into small groups to encourage independent decision-making Read, discuss ordered-item booklets Individuals vote (privately) for a cut score Small group discussion of votes

10. The Process: Round 2 Individuals vote (privately) for a cut score Small group discussion of votes Examine, discuss impact data

11. The Process: Round 3 Small groups converge Individuals vote (privately) for a cut score Discussion Individuals vote (privately) for a cut score The median cut score is the committee’s recommendation.

12. Bookmark Standard-Setting Procedure Test items students must know and be able to do to be considered Proficient Test items students must know and be able to do to be considered Advanced A P Ordered Item Booklet 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

13. Standard-Setting Process Individual Analysis Items most students get correct Items fewer students get correct Content every student should know

14. Review test; vote Round 1 Discuss data and distribution Median Standard-Setting Process Round 2 View impact data and discuss Vote Median

15. Standard-Setting Process Small groups join; vote Round 3 Median cut score is committee’s recommendation Median

16. Step 2: Psychometric Council Nationally-recognized experts Trained in measurement and psychometrics Independent Familiar with Maryland tests Similar to Technical Advisory Panels in other states

17. Psychometric Council Responsibilities Reviews work of all groups Ensures quality controls were followed Ensures standard-setting process technically sound Does not make judgments on whether standard is “good” or “bad” Forwards comments and recommendations to Review & Articulation Committee

18. Step 3: Review and Articulation Superintendents Assistant Superintendents Local Accountability Coordinators Advocates for students with special needs (special education and ELL) Parents Local Boards of Education Teachers Union representatives Business Community members MSDE

19. Review & Articulation Committee Responsibilities include: – Reviewing work of all groups and Psychometric Council – Ensuring rigor is equivalent among grades – Reviewing articulation among grades – Forwarding recommendations to the State Superintendent

20. Step 4: State Superintendent of Schools Reviews work of all groups, Psychometric Council, and Review & Articulation Committee Makes recommendation to the State Board of Education

21. Step 5: State Board Approval Reviews proposed standards Approves standards for implementation

22. Next Steps 1.Review and approve proposed standards to be used to determine proficient and advanced performance on the modified HSAs. The scores will be used to determine AYP for Maryland high schools next year. 2.Review and approve proposed standards for the Alt-MSA. Science will be reported this fall for the first time.

23. Overview of Standard Setting Questions and Discussion