What (the Heck) Does “Proficient” Mean? Standard Setting on the MCAS Alternate Assessment Daniel J. Wiener Massachusetts Dept. of Education CCSSO - San Antonio, TX June 2003

2 Standard Setting: Important Decisions Naming and defining “Performance Levels” Which raw scores count for PL? How will raw scores be combined to get a PL? What combination of scores yields which PL?

3 MCAS-Alt Performance Levels: Describe how well student has learned the General Curriculum? Awareness: Very little, if at all Emerging: Just getting the hang of it Progressing : Moving right along on challenging standards, but below grade level Needs Improvement: Doing okay at grade level Proficient: Doing well at grade level Advanced: Excelling at grade level

4 Which Scores Count Toward the Performance Level? Based on stakeholders’ advice, “Parallel the General Assessment” Therefore, Rubric scores in: Complexity, Accuracy, and Independence are combined to determine an overall Performance Level; Not Self-Evaluation, Generalized Performance

5 Two Possible Ways to Combine Rubric Scores EITHER: 1)Calculate a total numerical score, based on individual rubric scores; then, set cut score for each Performance Level Example: Complexity X (Accuracy + Independence) = Total score

6 Two Possible Ways to Combine Rubric Scores (continued...) OR: 2)Describe the characteristics of student performance for all score combinations (no total numerical score) Example: What does a score of LC=3, DSC=4, Ind=3 indicate about a student’s performance?

7 Student Performance: A Balancing Act Complexity Accuracy Independence LOW HIGH

8 Performance Level: Awareness LOW HIGH Complexity Accuracy Independence

9 Performance Level: Awareness LOW HIGH Complexity Accuracy Independence

10 Performance Level: Emerging LOW HIGH Complexity Accuracy Independence

11 Performance Level: Emerging LOW HIGH Complexity Accuracy Independence

12 Performance Level: Progressing Complexity Accuracy Independence LOW HIGH

13 Analytical Rubric: Determine Performance Level from Sub-scores If Level of Complexity = 3 (‘entry points’), and Demonstration of Skills & Concepts (Accuracy) = Independence = Aw 2 Em 3 AwEmPg 4 AwEmPg

14 Calculating an Overall Performance Level Average the Performance Level subscores in each content area Student StrandPerformance Level #1 #2 #3 AAw (1) - Aw (1) -Em (2) = Awareness (ave: 1.33) BAw (1) -Em (2) - Em (2) = Emerging (ave: 1.67) CEm (2) - Pg (3) - NI (4) = Progressing (ave: 3.0)

15 So, What Is Proficient, Anyway? When the student: Addresses challenging, achievable outcomes, based on academic standards Demonstrates knowledge and skills with >50% accuracy and independence

16 Changes Since 2001 Fine-tune the Analytical Rubric We know more than we did in 2001 Report “Incomplete” portfolios separately Previously reported as Awareness (lowest score)