June 23, 2003 Council of Chief State School Officers What Does “Proficiency” Mean for Students with Cognitive Disabilities Dr. Ron Cammaert Riverside Publishing Company

June 23, 2003 Council of Chief State School Officers Introduction  Goal - recommend performance thresholds  Variation on a holistic method - consider whole of individual’s work  Loosely based on Jaeger and Mills’ method (Cizek, 2001)

June 23, 2003 Council of Chief State School Officers Introduction  Classify student work into one of four categories  Category decided from a minimum subject rating and a minimum rating on generalization skills

June 23, 2003 Council of Chief State School Officers Description of Scores Part I Content Area Part II Generalized Skill Total score = progress in content area (Part I score) + mod + set + soc+ self

Decision Rule for Determining Level of Performance on WAAS portfolio LevelTotal ScorePart I Score Required 416 to 203 or to 152 or 3 or 4 28 to 111 or 2 or 3 or 4 15 to 71 or 2 or 3 *

Relationship Among Total Score, Level and Score Patterns Score LevelConM/AS/CS RS D

June 23, 2003 Council of Chief State School Officers Initial Procedure  A sample of portfolios from 2001 and 2002 with adequate evidence selected  14 portfolios selected - 32 different scoring patterns

June 23, 2003 Council of Chief State School Officers Step 1 - Training  Information about standard setting and how results will be used

June 23, 2003 Council of Chief State School Officers Step 2 - Review of the Assessment Material  Participation criteria for WAAS  Content validity evidence for the assessment and the scoring processes  Exemplars of student work for each score point  The Portfolio Scoring Summary  EALR Extensions

June 23, 2003 Council of Chief State School Officers Step 3 - Understanding the Definitions of the Standards  Brainstorm - students and student work that typifies each category  Academic Achievement Standard Descriptions

June 23, 2003 Council of Chief State School Officers Step 4 - Round 1  Using the table of frequencies of scoring patterns and the descriptors for each standard panelists decided on the category (“meets” or “not meets” proficiency) for each score pattern

June 23, 2003 Council of Chief State School Officers Step 5 - Round 2  Feedback on the panel cut scores  Given the set of 14 scored portfolios  Working individually reviewed each portfolio  Decisions for each portfolio and reset, if needed, the minimum score for each level  In groups discuss ratings  Individually completed the rating form for Round 2

June 23, 2003 Council of Chief State School Officers Step 6 - Review of Impact Data  Statewide frequencies for each score pattern from 2002 portfolio scores  Panelists discussed the impact of standards  Gave final rating  The final cut scores the mean of the minimum total scores for each category

June 23, 2003 Council of Chief State School Officers Step 7 - Evaluation of the Standard-Setting Process  Questionnaire

June 23, 2003 Council of Chief State School Officers Documentation  Minutes from National TAC  Script developed  Observer during session – follow script  Technical Report