1. 2010 STAR Interpreting and Using Results August 11, 2010 Webcast Webcast starts at 9:00 a.m.

2. August 2010 Post-Test Workshop 2 Objectives Workshop participants will be able to: Describe the purposes of STAR reports Interpret STAR results Explain key statistics Compare and contrast types of reports Identify proper uses of reports

3. August 2010 Post-Test Workshop 3 Agenda What’s New? Results and Statistical Analysis Using Results Summary and Internet Reports Data CDs Individual Student Reports Early Assessment Program

4. August 2010 Post-Test Workshop 4 What’s New in 2010 California Modified Assessment (CMA) reports –Scale score, performance levels for Grades 6–8 English−Language Arts (ELA) Grades 6–7 mathematics Grade 8 science –Percent correct for ELA grade 9 Life Science, grade 10 Algebra I Manual (M) 2

5. More What’s New Standards-based Test in Spanish (STS) reports –Scale score, performance levels for Grades 5–7 Reading/Language Arts (RLA) Grades 5–7 mathematics –Percent correct for Grades 8–11 RLA Grades 7–11 Algebra I Grades 8–11 Geometry Subgroup Summary: Ethnicity Grade 11 – Early Assessment Program (EAP) – California Community Colleges added August 2010 Post-Test Workshop 5 M 2

6. Quiz Question 1 Which of these tests had scale scores reported for the first time in 2010? A.CST for World History B.CAPA for Science C.CMA for math grade 7 and science grade 8 D.STS for math grade 7 and Algebra I August 2010 Post-Test Workshop 6

7. Quiz Question 1 Which of these tests had scale scores reported for the first time in 2010? C. CMA for math grade 7 and science grade 8 August 2010 Post-Test Workshop 7

8. August 2010 Post-Test Workshop 8 Results: Purposes of STAR Reports Report progress of students toward proficiency on the state’s academic content standards Notify where improvement needed –To help students’ achievement –To improve educational programs Provide data for state and federal accountability programs M 4

9. August 2010 Post-Test Workshop 9 Results: Performance Levels State Goal: All students score at proficient or higher CAPA Proficient: 35 or higher scale score 350 or higher scale score –All CST –STS, grades 2–7, excluding Algebra I –CMA, grades 3–8, excluding Algebra I M 8−10

10. August 2010 Post-Test Workshop 10 Results: Other Performance Levels Advanced Basic cut score –CAPA: 30 –CST: 300 –CMA, grades 3–8: 300 (excluding Algebra I) –STS, grades 2–7: 300 (excluding Algebra I) Below basic Far below basic Cut points vary for advanced and below basic by –Subject –Grade M 8−10; Appendix B

11. August 2010 Post-Test Workshop 11 Results: Scale Scores Scale scores allow same score to mean same thing across test versions within grade and content area Accounts for differences in difficulty Scale score ranges by program: –CST, CMA (grades 3–8, excluding Algebra I), STS (grades 2–7, excluding Algebra I): 150–600 for each grade and subject –CAPA: 15–60 for each level and subject M 8−10

12. August 2010 Post-Test Workshop 12 Results: Equating Psychometric procedure Adjusts for test difficulty Additional information in the CST Technical Report on the CDE Web site M 8

13. August 2010 Post-Test Workshop 13 Results: Reporting Clusters (Content Areas) Three to six clusters for each subject May be useful as indicators of individual or group strengths and weaknesses But... Reporting clusters should be interpreted with caution M 8−12; Appendix A

14. August 2010 Post-Test Workshop 14 Results: Cluster Percent Correct Available for 2010 CST: all CMA: –Grades 3–8, excluding Algebra I STS: all M 8−12; Appendix A

15. August 2010 Post-Test Workshop 15 Results: Reporting Cluster Cautions Based on small numbers of items; therefore, may not be reliable or generalizable NOT equated from year to year Should not compare reporting cluster percent correct from year to year M 8−12; Appendix A

16. August 2010 Post-Test Workshop 16 Interpreting Reporting Clusters or Content Areas in the Same Year Compare to percent-correct range of proficient students statewide M 8−12; Appendix A

17. August 2010 Post-Test Workshop 17 2010 CST Reporting Clusters: Number of Questions and Average Percent Correct To be finalized with complete data in 2010 Post-Test Guide, after August 16. M Appendix A

18. August 2010 Post-Test Workshop 18 Examples—Interpreting Reporting Clusters for the CST for Geometry M 11

19. Quiz Question 2 What is a scale score? A.Percent correct of all questions B.Mean percent correct of all questions C.An adjustment of this year’s and last year’s raw scores to show changes D.An adjustment of the raw score to account for differences in difficulty August 2010 Post-Test Workshop 19

20. Quiz Question 2 What is a scale score? D. An adjustment of the raw score to account for differences in difficulty August 2010 Post-Test Workshop 20

21. August 2010 Post-Test Workshop 21 Using Results For instructional decisions with other data Used in Academic Performance Index (API) calculations: CSTs, CAPA, CMA (only grades 3–8, excluding Algebra I) Used in adequate yearly progress (AYP) calculations, ELA and mathematics: –Grades 2–8, CSTs –Grades 3–8, CMA (excluding Algebra I) –Grades 2–8 and 10, CAPA M 4

22. August 2010 Post-Test Workshop 22 Year-to-Year Comparisons Do Compare CSTs: Same Grade and Same Content Area Mean scale score –Same content and grade, varying years Percent in each performance level –Same content by grade across years e.g., 2009 ELA grade 10 with 2010 ELA grade 10 M 13−16

23. August 2010 Post-Test Workshop 23 Year-to-Year Comparisons Do Compare CSTs: Percent Proficient and Advanced Percentage of students scoring at PROFICIENT and above –For a given grade and subject, e.g., Percent proficient and above for grade 3 math in 2009 and 2010 –For a given subject and aggregated grades, e.g., Percent proficient and above for grades 2– 6 mathematics in 2009 and 2010 –Across grades and a subject, e.g., Percent proficient and above in all courses and all grades M 13−16

24. August 2010 Post-Test Workshop 24 Year-to-Year Comparisons DON’T Compare Individual scale scores or statistics based on scale scores for different grades or content areas –Subjects by grade are independently scaled –Different content standards are measured in different grades Cohorts across grades Across tests CMA grades 6−11, STS grades 5−11 to previous years CAPA, CMA grades 3−5, STS grades 2−4 to years before 2009 M 13−16

25. August 2010 Post-Test Workshop 25 Example—Using CST Results to Compare Grade Results from Year to Year 2009 CST for ELA 2010 CST for ELA Grade% Prof or Above Difference Grade 231%35%4% Grade 333% 0% Grade 429%31%2% Grade 534%32%–2% Grade 631%32%1% All Grades32%33%1% M 14

26. Quiz Question 3 What is the best comparison for CST scores of students within a middle school? A.2009 mean scale scores for ELA of a cohort of grade 7 students with 2010 scale scores for ELA of the same students in grade 8 B.2009 mean scale scores for ELA for grade 8 students with 2010 mean scale scores for ELA for grade 8 students C.2009 mean percent correct scores for ELA with 2010 percent correct scores for ELA for the same students in grade 8 D.2009 mean percent correct scores for ELA for grade 8 students with 2010 mean percent correct for ELA for grade 8 students August 2010 Post-Test Workshop 26

27. Quiz Question 3 What is the best comparison for CST scores of students within a middle school? B. 2009 mean scale scores for ELA for grade 8 students with 2010 mean scale scores for ELA for grade 8 students August 2010 Post-Test Workshop 27

28. Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare... A.To proficient students statewide B.One cluster to another, same year C.The same cluster to the same cluster, different years D.To the average percent correct of all students in a class August 2010 Post-Test Workshop 28

29. Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare… A.To proficient students statewide August 2010 Post-Test Workshop 29

30. August 2010 Post-Test Workshop 30 Aggregate (Summary) Reports What are they? –Student Master List Summary –Student Master List Summary End-of- Course (EOC) –Subgroup Summary Report Emphasis: CSTs –Criterion-referenced tests –Progress is measured in percent of students scoring proficient and advanced M 18−22

31. August 2010 Post-Test Workshop 31 Student Master List Summary By grade CSTs, CMA, CAPA, and STS Lists subjects % and # at each performance level Mean scale score (CST, CAPA, CMA grades 3–8 [ not Algebra I], STS grades 2–7 [not Algebra I]) Reporting cluster: mean percent correct (CST, CMA grades 3–8 [not Algebra I], STS) M 19–20, M 28−34

32. August 2010 Post-Test Workshop 32 Student Master List Summary Grade 7 Example M 34

33. August 2010 Post-Test Workshop 33 Student Master List Summary Basic Statistics M 28−34

34. August 2010 Post-Test Workshop 34 Total CST/CMA and CAPA multiple- choice answer documents submitted as scorable Minus -Documents marked as “Student enrolled after the first day of testing and was given this test” Who Counts? Number Enrolled M 28−34

35. August 2010 Post-Test Workshop 35 Who Counts? Number Tested All CST, CMA, CAPA, STS answer documents with one or more answers Plus –Z = Tested but marked no answers Not included –A = Students absent –E = Not tested due to significant medical emergency –P = Parent/guardian exemptions –T = Enrolled first day, not tested, tested at previous school –Students with inconsistent grades –Non–English learners who took the STS M 30−32

36. August 2010 Post-Test Workshop 36 Who Counts? Number and Percent Valid Scores Number Valid Scores –For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct). Percent Valid Scores –For the subject, number of valid scores divided by the number of students tested. –Not included:  Incomplete tests  Modified tests  Non–English learners who took the STS  Unknown EOC mathematics (except grade 7 mathematics) or science tests  Inconsistent grades M 30−32

37. August 2010 Post-Test Workshop 37 Who Counts? Number Tested with Scores All tests taken, including those taken with modifications, that receive a score Not included: –Incomplete tests –Non–English learners who took the STS –Unknown EOC mathematics (except grade 7 mathematics) or science tests –Inconsistent grades M 30−32

38. August 2010 Post-Test Workshop 38 Who Counts? Performance Levels All CSTs, CAPA CMA Grades 3–8, excluding Algebra I STS Grades 2–7, excluding Algebra I Advanced, proficient, basic, below basic –All valid scores falling in the performance level Far below basic –All valid scores falling in the performance level –CSTs taken with modifications (in aggregate reporting and accountability only) M 30−32

39. August 2010 Post-Test Workshop 39 Student Master List Summary Performance Levels M 31

40. August 2010 Post-Test Workshop 40 Student Master List Summary Reporting Clusters Compare to: Average percent correct range for students statewide who scored proficient on the total test M 31−32

41. August 2010 Post-Test Workshop 41 Student Master List Summary: Writing B=Blank C=Copied prompt I=Illegible L=Language other than English R=Refusal T=Off Topic W=Wrong prompt (prompt from an earlier administration) M 32

42. August 2010 Post-Test Workshop 42 Student Master List Summary: End-of-Course (EOC) CST, CMA, and STS By subject Lists each grade eligible to take test Mathematics (Grades 7–11) –CST General Math, Algebra I, Geometry, etc. –CMA Algebra I –STS Algebra I and Geometry CST Science (Grades 9–11) –Earth Science, Biology, Chemistry, etc. CST History–Social Science (Grades 9–11) –World History Same statistics as grade-level Student Master List Summary M 35−40

43. August 2010 Post-Test Workshop 43 Student Master List Summary End-of-Course CST Biology Example M 40

44. August 2010 Post-Test Workshop 44 Subgroup Summary: CSTs, CMA, CAPA, and STS Disability status –Based on disability status for CST, CMA, STS –CAPA: each disability type If missing, correct with demographic data corrections Economic status –Based on NSLP eligibility or parent education level Gender English proficiency Ethnicity Ethnicity for Economic Status (only for CSTs, CMA grades 3–8, and CAPA) M 41−59

45. Subgroup Summary: Gender August 2010 Post-Test Workshop 45 Grade 5 Example

46. Subgroup Summary: Ethnicity From answer document sections 8 and 9: –Hispanic or Latino? –Race: grouped and worded differently –May cause counts to differ from previous year –Federal Reporting Requirement Two or more races (instead of “unknown”) August 2010 Post-Test Workshop 46

47. August 2010 Post-Test Workshop 47 Subgroup Summary: Ethnicity for Economic Status Ethnicity for Economically Disadvantaged CST Grade 6 Example M 64

48. August 2010 Post-Test Workshop 48 Subgroup Summary: Ethnicity for Economic Status Example: Economically disadvantaged for each ethnicity M 60−64

49. August 2010 Post-Test Workshop 49 Subgroup Summary: Ethnicity for Economically Disadvantaged M 60−64

50. Break — 10 minutes

51. August 2010 Post-Test Workshop 51 Internet Reports Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS Available to the public online for school, district, county, and state More subgroups than paper reports –Parent education –Special program participation Access from http://star.cde.ca.gov/ M 104−117

52. August 2010 Post-Test Workshop 52 Internet Reports: Available Reports CSTs –Mean scale scores –Percents by performance levels CST Summary –Percent proficient and advanced CMA –Grades 3–8 (not Algebra I): mean scale scores, percents by performance levels –Grade 9 ELA, Grade 10 Life Science, Algebra I: average percent correct CAPA –Mean scale scores –Percents by performance levels STS –Grades 2–7 (not Algebra I): mean scale scores, percents by performance levels –Grades 8–11: average percent correct M 104−117

53. Internet Demonstration August 2010 Post-Test Workshop 53

54. August 2010 Post-Test Workshop 54 Internet Reports: CST Sample M 108−109

55. August 2010 Post-Test Workshop 55 Internet Reports: CST Summary Sample M 109−110

56. Other Internet Reports CMA (M 111−112) –Same as CST—CMA grades 3−8 –Percent correct—CMA grades 9–11, Algebra I CAPA (M 112−115) –State level: same as CST; separate Level I –County, district, school Mean scale score Percent proficient or above STS (M 116−117) –Same as CST—STS grades 2−7 –Percent correct—STS grades 8−11 August 2010 Post-Test Workshop 56

57. Quiz Question 5 Which subgroup can only be accessed from the Internet? A.Parent Education Level B.CAPA by individual disability status C.Ethnicity for Economic Status D.English Proficiency August 2010 Post-Test Workshop 57

58. Quiz Question 5 Which subgroup can only be accessed from the Internet? A.Parent Education Level August 2010 Post-Test Workshop 58

59. August 2010 Post-Test Workshop 59 Data CDs What are they? –Lists of information from answer documents and scores of every student in district –In.txt format: wraps What are they used for? –Searching for specific data –Creating unique reports –Verifying paper reports What else is needed? –Text editor –or Desktop application –or Student Information System

60. August 2010 Post-Test Workshop 60 View of Data As.txt, word wrap on With text editor, word wrap off

61. August 2010 Post-Test Workshop 61 Organization of Data Two files: –Demographics, special conditions, and test scores –Accommodations, modifications, English Learners, and irregularities Data Layout = guide to location of data on files –Position –Number of characters –Whether numeric or alpha

62. August 2010 Post-Test Workshop 62 Data Layout Sample

63. August 2010 Post-Test Workshop 63 Individual Reports STAR Student Record Label –Adhesive label to affix to student’s permanent school record STAR Student Master List –Alphabetical list of students and their scores –Tests listed in order within grade CSTs CMA CAPA STS STAR Student Report: individual’s scores –2 two-sided color copies for each test –For parents/guardians, school M 23−26

64. August 2010 Post-Test Workshop 64 Student Record Label Grade 10 Sample: Student Name and Identification M 65

65. August 2010 Post-Test Workshop 65 Student Record Label CST/CMA Grade 10 Example M 65−66

66. August 2010 Post-Test Workshop 66 Student Master List CST/CMA Grade 3 Example M 69−71

67. August 2010 Post-Test Workshop 67 Student Report CST Grade 11 Example M 72; 73−82

68. August 2010 Post-Test Workshop 68 Student Report CST Grade 11 Example M 74−75

69. August 2010 Post-Test Workshop 69 Student Report ♦= Percent correct obtained by the student on the reporting cluster/content area ▬ = Range of percent correct scores on the reporting cluster for students statewide who scored proficient on that test Student name on back CST Grade 10 Example M 76−78

70. August 2010 Post-Test Workshop 70 Student Report CST Grade 11 Example, EAP Results M 78; 82

71. August 2010 Post-Test Workshop 71 Explain to Parents Scale Score  Average % correct cluster score x 600 Reporting Clusters not comparable –Different difficulty –Varying number of questions –Average % correct of clusters  % correct of total scale score Scale scores –Use conversion tables and other statistical techniques –Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient) M 13

72. August 2010 Post-Test Workshop 72 Other Student Reports CMA (M 82−90) –Performance levels and scale scores reported for grades 3–8  Back, cluster reporting for grades 3–8 –Percent correct for entire test for grades 9–11  Back, “About the CMA” CAPA (M 91−95) – Back, “About the CAPA” STS (M 96−103) –In Spanish –Performance levels and scale scores, grades 2–7  Back, cluster percent correct –Percent correct for entire test, grades 8–11  Back, cluster percent correct  Back, how to use report

73. August 2010 Post-Test Workshop 73 Unmatched Reports Grade 7 Writing Unmatched Multiple-Choice Report  CST multiple-choice score but no writing score  CMA multiple-choice score but no writing score Unmatched Writing Report  Writing score but no CST multiple-choice score  Writing score but no CMA multiple-choice score Students receive 2 reports if writing score not matched to multiple-choice score

74. Unmatched Report CMA Grades 8 −11 August 2010 Post-Test Workshop 74 Student took at least 1 CMA test, but no CST report matches All grades except grade 9 require a CST GradeTests That Can be Taken as CST or CMA CST Required (No CMA option) 8ELA, Algebra I, ScienceHistory-Social Science 9ELA, Algebra I 10Algebra I, Life ScienceELA 11Algebra IELA, History-Social Science Table does not include end-of-course CSTs for grades 8−11

75. Unmatched Report CMA Grades 8−11 Report for each grade Lists student data for those with only CMA or demographics do not match CST August 2010 Post-Test Workshop 75

76. August 2010 Post-Test Workshop 76 Summary Do’s and Don’ts –Do compare mean scale score, percent at performance levels within same grade, same content area –Don’t compare mean scale scores across grades, content area Summary and Internet reports Data CDs Individual reports

77. August 2010 Post-Test Workshop 77 For more information see: http://www.cde.ca.gov/ta/tg/sr/ http://www.star.cde.ca.gov/ http://www.startest.org/ STAR Technical Assistance Center –800-955-2954 CDE Accountability –aau@cde.ca.gov –916-319-0863

78. To Contact CDE CDE STAR office: 916-445-8765 –English–Language Arts tests, including writing: Jamie Contreras, jcontrer@cde.ca.gov, 916-319-0353 –Mathematics tests: Jane Liang, jliang@cde.ca.gov, 916-322-1854 –Science tests: Blessing Mupanduki, bmupanduki@cde.ca.gov, 916-319-0969 –History–Social Science tests: Nicole Jespersen, njespersen@cde.ca.gov, 916-319- 0364 –CAPA, CMA: Don Killmer, dkillmer@cde.ca.gov, 916-319-0350 August 2010 Post-Test Workshop 78