1. 1 Measure Up! Benchmark Assessment Quality Assurance Process RCAN September 10, 2010

2. 2 Measure Up! Objective To monitor and improve Benchmark Assessments in order to attain the most accurate possible measurement of student achievement with respect to California Content Standards Tests.

3. Measure Up! Components Content—Structure & Course Guides Predictability – Correlation of Benchmark Exams to CST scores – Association of Benchmark Exams with CST Performance Levels Item Analysis – Difficulty – Discrimination – Representative of CST items Evolving

4. 4 PSUSD Benchmark Structure CST ~153 ID 60 to 75 Items Blueprint Aligned Benchmark #1 ~45 ID ~20-35 Items 1 st 45 ID Paced Standards Partial Match to CST Benchmark #2 ~90 ID ~20-35 Items 2 nd 45 ID Paced Standards Partial Match to CST Benchmark #3 ~135 ID ~20-35 Items 3 rd 45 ID Paced Standards Partial Match to CST

5. 5 2008-2009 Algebra I (8 th Grade) Aggregation of 3 Benchmarks 2008-209 Algebra I (8 th Grade) CST Predictability

6. 6 Prof = 38 38 Algebra I BM 08-09 (8 th Grade) Algebra I CST 2009 (8 th Grade)

7. 7 N = 23 N = 119N = 125 N = 51 N = 5

8. 8

9. 9 2008-2009 ELA 10 th Grade Aggregation of 3 Benchmarks 2008-209 ELA 10 th Grade CST Predictability

10. 10 ELA 10 th Gd BM 08-09 ELA 10 th Gd CST 2009 Prof = 53 53

11. 11 N = 26 N = 151 N = 326 N = 249 N = 36

12. 12

13. 13 2008-2009 US History 11 th Grade Aggregation of 3 Benchmarks 2008-209 US History 11 th Grade CST Predictability

14. 14 r =.46 US His 11 th Gd BM 08-09 US His 11 th Gd CST 2009 Prof = 40 63

15. 15 N = 68 N = 116 N = 142 N = 72 N = 7

16. 16

17. 17 2008-2009 Science 8 th Grade Aggregation of 3 Benchmarks 2008-209 Science 8 th Grade CST Predictability

18. 18 r =.77 Science 8 th Gd BM 08-09 Science 8 th Gd CST 2009 Prof = 39 44

19. 19 N = 84 N = 307 N = 494 N = 275 N = 24

20. 20

21. 21 2008-2009 Math 6 th Grade Aggregation of 3 Benchmarks 2008-209 Math 6 th Grade CST Predictability

22. 22 r =.84 Math 6 th Gd BM 08-09 Math 6 th Gd CST 2009 Prof = 43 48

23. 23 N = 164 N = 393 N = 460 N = 224 N = 31

24. 24

25. 25 2008-2009 ELA 4 th Grade Aggregation of 3 Benchmarks 2008-209 ELA 4 th Grade CST Predictability

26. 26 r =.80 Prof = 44 102

27. 27 N = 88 N = 176 N = 334 N = 388 N = 93

28. 28

29. 29 Item Level Analysis

30. 30 Item Difficulty The p value for any item – percentage of correct answers – usually in decimal form Ideally p value range is.30 to.80 for most items For example – p value of.28 = 28% of the test takers got the item right – p value of.75 = 75% of the test takers got the item right – P value of.95 = 95% of the test takers got the item right

31. 31 Item Difficulty Monitoring Item NumberN StudentsStandardp Value 1853NS1.2.37 2853NS1.2.58 3853NS1.2.77 4853MG2.1.25 5853MG2.1.35 6853MG2.1.40 7853PS3.3.43 8853PS3.3.59 9853PS3.3.75 10853PS3.3.95

32. 32 Item Discrimination Is item “discriminating” appropriately between higher & lower scoring students Discrimination Index (DI) = difference between how upper half and lower half of students score on an item DI ranges between -1 and +1 We want items to discriminate positively

33. 33 Item Discrimination Monitoring Item Number N Students Standardp ValueDIMax DI 1853NS1.2.37.45.74 2853NS1.2.58.53.84 3853NS1.2.77.30.46 4853MG2.1.25 -.12.50 5853MG2.1.35.10.70 6853MG2.1.40.62.80 7853PS3.3.43.40.86 8853PS3.3.59.58.82 9853PS3.3.75.41.50 10853PS3.3.95.09.10

34. 34 2008-2009 7 th Grade Math 3 rd Benchmark 2009-2010 7 th Grade Math 3 rd Benchmark Representative of CST Items and our continual Revision Process

35. 35 7NS1.6 2008-2009 Item p =.33 DI =.34

36. 36 RTQ for 7NS1.6 CST = 1 RTQ = 1

37. 37 7NS1.6 2009-2010 Item p =.42 DI =.31

38. 38 Additionally… Item #3 replaced (as #3) with item modeled after RTQ # 16 (7NS1.7*)—CST = 5, RTQ = 7 Item #15 replaced (as #13) with item modeled after RTQ #47 (7AF1.5) Item #s 23 & 24 replaced (as #s 21 & 22) with items modeled after RTQ #s 89 & 88 (7MG3.4*)

39. 39 Measure Up! Next Steps Benchmark Exam Structure Institutionalize System CAHSEE Writing Prompt Discrimination Distractor Shaping

40. 40 Questions?