1. 2012 STAR Interpreting and Using Results August 8, 2012 Webcast Webcast starts at 9 a.m.

2. August 2012 Post-Test Workshop 2 Objectives Workshop participants will be able to: Describe the purposes of STAR reports Interpret STAR results Explain key statistics Compare and contrast types of reports Identify proper uses of reports

3. August 2012 Post-Test Workshop 3 Agenda What’s New? Results and Statistical Analysis Using Results Summary and Internet Reports Data CDs Individual Student Reports Early Assessment Program

4. August 2012 Post-Test Workshop 4 What’s New in 2012 California Modified Assessment (CMA) reports –Scale score, performance levels for  ELA, grades 10 and 11  Geometry Manual (M) 2

5. Quiz Question 1 Which of these tests had scale scores reported for the first time in 2012? A.CST for World History B.CAPA for Science C.CMA for Algebra I D.CMA for Geometry August 2012 Post-Test Workshop 5

6. Quiz Question 1 Which of these tests had scale scores reported for the first time in 2012? D. CMA for Geometry August 2012 Post-Test Workshop 6

7. August 2012 Post-Test Workshop 7 Results: Purposes of STAR Reports Report progress toward proficiency on the state’s academic content standards Notify where improvement needed –To help students’ achievement –To improve educational programs Provide data for state and federal accountability programs M 3

8. August 2012 Post-Test Workshop 8 Results: Performance Levels State goal: All students score at proficient or higher 350 or higher scale score –All CST –All CMA –STS for RLA and math in grades 2–7, excluding Algebra I CAPA proficient: 35 or higher scale score M 8−9

9. August 2012 Post-Test Workshop 9 Results: Other Performance Levels Advanced Basic cut score –CST: 300 –CMA: 300 –STS: 300 for RLA and math in grades 2–7 (excluding Algebra I) –CAPA: 30 Below basic Far below basic For each testing program, cut points vary for advanced and below basic by –Subject –Grade M 8−9; Appendix B

10. August 2012 Post-Test Workshop 10 Results: Scale Scores Scale scores allow the same score to mean the same thing across test versions within grade and content area Accounts for differences in difficulty Scale score ranges by program: –CST, CMA, STS (grades 2–7, excluding Algebra I): 150–600 for each grade and subject –CAPA: 15–60 for each level and subject M 7−10

11. August 2012 Post-Test Workshop 11 Results: Equating Psychometric procedure Adjusts for test difficulty Additional information in the technical report on the CDE Web site M 7

12. August 2012 Post-Test Workshop 12 Results: Reporting Clusters (Content Areas) Three to six clusters for each subject May be useful as indicators of individual or group strengths and weaknesses But... Reporting clusters should be interpreted with caution M 8−10; Appendix A

13. August 2012 Post-Test Workshop 13 Results: Reporting Cluster Cautions Cluster percent correct available for all CST, CMA, STS Based on small numbers of items; therefore, may not be reliable or generalizable NOT equated from year to year Should not compare reporting cluster percent correct from year to year M 8−10; Appendix A

14. August 2012 Post-Test Workshop 14 Interpreting Reporting Clusters or Content Areas in the Same Year Compare to percent-correct range of proficient students statewide M 8−10; Appendix A

15. August 2012 Post-Test Workshop 15 2012 CST Reporting Clusters: Number of Questions and Average Percent Correct 2012 Post-Test Guide, Appendixes A and C as posted on startest.org will be finalized with complete data on August 31. M Appendix A

16. August 2012 Post-Test Workshop 16 Examples—Interpreting Reporting Clusters for the CST for Geometry M 9

17. Quiz Question 2 What is a scale score? A.Percent correct of all questions B.Mean percent correct of all questions C.An adjustment of this year’s and last year’s raw scores to show changes D.An adjustment of the raw score to account for differences in difficulty August 2012 Post-Test Workshop 17

18. Quiz Question 2 What is a scale score? D. An adjustment of the raw score to account for differences in difficulty August 2012 Post-Test Workshop 18

19. August 2012 Post-Test Workshop 19 Using Results For instructional decisions with other data Used in Academic Performance Index (API) calculations: all grades and subjects; CST, CMA, CAPA Used in adequate yearly progress (AYP) calculations, ELA and mathematics: –CST – grades 2–8 –CMA – grades 3–8 –CAPA – grades 2–8, 10 M 3

20. August 2012 Post-Test Workshop 20 Year-to-Year Comparisons Do Compare CSTs: Same Grade and Same Content Area Mean scale score –Same content and grade, varying years Percent in each performance level –Same content by grade across years  e.g., 2011 grade 10 ELA with 2012 grade 10 ELA M 11−14

21. August 2012 Post-Test Workshop 21 Year-to-Year Comparisons Do Compare CSTs: Percent Proficient and Advanced Percentage of students scoring at PROFICIENT and above –For a given grade and subject, e.g., Percent proficient and above for grade 3 math in 2011 and 2012 –For a given subject and aggregated grades, e.g., Percent proficient and above for grades 2– 6 mathematics in 2011 and 2012 –Across grades and a subject, e.g., Percent proficient and above in all courses and all grades M 11−14

22. August 2012 Post-Test Workshop 22 Year-to-Year Comparisons DON’T Compare Individual scale scores or statistics based on scale scores for different grades or content areas –Subjects by grade are independently scaled –Different content standards are measured in different grades Cohorts across grades Across tests Scale scores to percent correct scores CAPA to years prior to 2009, due to new standard setting then M 11−14

23. August 2012 Post-Test Workshop 23 Example—Using CST Results to Compare Grade Results from Year to Year 2011 CST for ELA 2012 CST for ELA Grade% Prof or Above Difference Grade 231%35%4% Grade 333% 0% Grade 429%31%2% Grade 534%32%–2% Grade 631%32%1% All Grades32%33%1% M 13

24. Quiz Question 3 Which is the best comparison for CST scores of students within a middle school? A.2011 mean scale scores for ELA of a cohort of grade 7 students with 2012 scale scores for ELA of the same students in grade 8 B.2011 mean scale scores for ELA for grade 8 students with 2012 mean scale scores for ELA for grade 8 students C.2011 mean percent correct scores for ELA with 2012 mean percent correct scores for ELA for the same students D.2011 mean percent correct scores for ELA for grade 8 students with 2012 mean percent correct for ELA for grade 8 students August 2012 Post-Test Workshop 24

25. Quiz Question 3 Which is the best comparison for CST scores of students within a middle school? B. 2011 mean scale scores for ELA for grade 8 students with 2012 mean scale scores for ELA for grade 8 students August 2012 Post-Test Workshop 25

26. Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare... A.To proficient students statewide B.One cluster to another, same year C.The same cluster to the same cluster, different years D.To the average percent correct of all students in a class August 2012 Post-Test Workshop 26

27. Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare… A.To proficient students statewide August 2012 Post-Test Workshop 27

28. August 2012 Post-Test Workshop 28 Aggregate (Summary) Reports What are they? –Student Master List Summary –Student Master List Summary End-of- Course (EOC) –Subgroup Summary Report emphasis: CSTs –Criterion-referenced tests –Progress is measured in percent of students scoring proficient and advanced Note: Back of reports provides guide to abbreviations, score codes M 16−19

29. August 2012 Post-Test Workshop 29 Student Master List Summary By grade Results by program (CSTs, CMA, CAPA, and STS) and subject # and % at each performance level (except STS for: RLA grades 8−11, Algebra I, and Geometry) Mean scale score (except STS for: RLA grades 8−11, Algebra I and Geometry) Reporting cluster: mean percent correct (except CAPA ) M 16–17, M 26−32

30. August 2012 Post-Test Workshop 30 Student Master List Summary End-of-Course By program (CSTs, CMA, and STS) and subject Results for each grade, and for all grades combined # and % at each performance level (except STS for Algebra I and Geometry) Mean scale score (except STS for Algebra I and Geometry) Reporting cluster: mean percent correct M 17, M 33−38

31. August 2012 Post-Test Workshop 31 Student Master List Summary Grade 7 Example M 32

32. August 2012 Post-Test Workshop 32 Student Master List Summary Basic Statistics M 27−30

33. August 2012 Post-Test Workshop 33 Total CST/CMA and CAPA multiple- choice answer documents submitted as scorable Minus –Documents marked as “Student enrolled after the first day of testing and was given this test” Who Counts? Number Enrolled M 28

34. August 2012 Post-Test Workshop 34 Who Counts? Number Tested All CST, CMA, CAPA, STS answer documents with one or more answers Plus –Z = Tested but marked no answers Not included –A = Students absent –E = Not tested due to significant medical emergency –P = Parent/guardian exemptions –T = Enrolled first day, not tested, tested at previous school –Students with inconsistent grades –Non–English learners who took the STS M 28

35. August 2012 Post-Test Workshop 35 Who Counts? Number and Percent Valid Scores Number Valid Scores –For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct) –Not included:  Incomplete tests  Modified tests  Non–English learners who took the STS  Unknown EOC mathematics (except grade 7 mathematics) or science tests –Inconsistent grades Percent Valid Scores –For the subject, number of valid scores divided by the number of students tested M 28

36. August 2012 Post-Test Workshop 36 Who Counts? Number Tested with Scores All tests taken, including those taken with modifications, that result in a score Not included: –Incomplete tests –Non–English learners who took the STS –Unknown EOC mathematics or science tests –Inconsistent grades M 28

37. August 2012 Post-Test Workshop 37 Student Master List Summary Performance Levels M 29

38. August 2012 Post-Test Workshop 38 Who Counts? Performance Levels All CSTs, CAPA, CMA STS grades 2–7 RLA and math (excluding Algebra I) Advanced, proficient, basic, below basic –All valid scores falling in the performance level Far below basic –All valid scores falling in the performance level –CSTs taken with modifications (in aggregate reporting and accountability only) M 29

39. Who Counts? Mean Scale Scores Average of valid scale scores Can be used to compare results for the same content/grade across years August 2012 Post-Test Workshop 39 M 29

40. August 2012 Post-Test Workshop 40 Student Master List Summary: Reporting Clusters Compare to: Average percent correct range for students statewide who scored proficient on the total test (See the Post-Test Guide, Appendix A.) M 29

41. August 2012 Post-Test Workshop 41 Student Master List Summary: Writing B=Blank C=Copied prompt I=Illegible L=Language other than English R=Refusal T=Off Topic W=Wrong prompt (prompt from an earlier administration) M 29

42. August 2012 Post-Test Workshop 42 Subgroup Summary: CSTs, CMA, CAPA, and STS Disability status –Based on disability status for CST, CMA, STS –CAPA: each disability type Economic status –Based on NSLP eligibility or parent education level Gender English proficiency Ethnicity Ethnicity for Economic Status (only for CSTs, CMA, and CAPA) M 39−53

43. August 2012 Post-Test Workshop 43 Subgroup Summary: Ethnicity for Economic Status M 50−53

44. August 2012 Post-Test Workshop 44 Example: Economically disadvantaged for each ethnicity Subgroup Summary: Ethnicity for Economic Status M 50−53

45. August 2012 Post-Test Workshop 45 Subgroup Summary: Ethnicity for Economic Status M 50−53

46. Break — 10 minutes

47. August 2012 Post-Test Workshop 47 Internet Reports Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS Available to the public online for school, district, county, and state “Students with Scores” = number tested with scores CST Summaries of % Advanced and Proficient More subgroups than paper reports –Parent education –Special program participation Access from http://star.cde.ca.gov/ M 93−106

48. Internet Demonstration August 2011 Post-Test Workshop 48

49. August 2012 Post-Test Workshop 49 Internet Reports: CST Sample M 97−106

50. August 2012 Post-Test Workshop 50 Internet Reports: CST Summary Sample M 98−99 Only available for CSTs

51. Other Internet Reports CMA (M 99−100) Same as CST CAPA (M 101−104) –State level: same as CST; separate Level I –County, district, school  Mean scale score  Percent proficient or above STS (M 105−106) –Same as CST—STS grades 2−7 RLA and math (except Algebra I) –Average percent (%) correct—STS grades 8−11 RLA and Algebra I and Geometry August 2012 Post-Test Workshop 51

52. Quiz Question 5 Which subgroup can only be accessed from the Internet? A.Parent Education Level B.CAPA by individual disability status C.Ethnicity for Economic Status D.English Proficiency August 2012 Post-Test Workshop 52

53. Quiz Question 5 Which subgroup can only be accessed from the Internet? A.Parent Education Level August 2012 Post-Test Workshop 53

54. August 2012 Post-Test Workshop 54 Data CDs What are they? –Lists of information from answer documents and scores of every student in the district –In.txt format What are they used for? –Searching for specific data –Creating unique reports –Verifying paper reports What else is needed? –Text editor –or Desktop application –or Student Information System

55. August 2012 Post-Test Workshop 55 View of Data As.txt, word wrap on With text editor, word wrap off

56. August 2012 Post-Test Workshop 56 Organization of Data Two files: –Demographics, special conditions, and test scores –Accommodations, modifications, English learners, and irregularities Data Layout = guide to location of data on files –Position –Number of characters –Whether numeric or alpha

57. August 2012 Post-Test Workshop 57 Data Layout Sample

58. August 2012 Post-Test Workshop 58 Individual Reports STAR Student Record Label –Adhesive label to affix to student’s permanent school record STAR Student Master List –Alphabetical list of students and their scores –Tests listed in order within grade  CSTs  CMA  CAPA  STS STAR Student Report: individual’s scores –2 two-sided color copies for each test –For parents/guardians, school –Per regulations, district must forward within 20 business days M 20−24

59. August 2012 Post-Test Workshop 59 Student Record Label Grade 10 Sample: Student Name and Identification M 55−56

60. August 2012 Post-Test Workshop 60 Student Record Label CST/CMA Grade 10 Example M 55−56

61. August 2012 Post-Test Workshop 61 Student Master List CST/CMA Grade 3 Example M 57−60

62. August 2012 Post-Test Workshop 62 Student Report CST Grade 11 Example M 61−92; 70 −71

63. August 2012 Post-Test Workshop 63 Student Report CST Grade 11 Example M 70

64. August 2012 Post-Test Workshop 64 Student Report ♦= Percent correct obtained by the student on the reporting cluster/content area ▬ = Average percent-correct range on the reporting cluster for students statewide who scored proficient on the total test scores CST Grade 11 Example Student’s name on back M 71

65. August 2012 Post-Test Workshop 65 Explain to Parents Scale Score  Average % correct cluster score x 600 Reporting clusters not comparable –Different difficulty –Varying number of questions –Average % correct of clusters  % correct on total test (or total number of test items) Scale scores –Use conversion tables resulting from statistical procedures –Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient) M 8 −10

66. August 2012 Post-Test Workshop 66 Other Student Reports CMA ( M 72−79 ) –Performance levels and scale scores  Back, cluster percent correct CAPA (M 80−84) – Back, “About the CAPA” STS (M 85−92) –In Spanish –Performance levels and scale scores, grades 2–7 RLA and math (except for Algebra I)  Back, cluster percent correct –Percent correct for entire test, grades 8–11 RLA and Algebra I and Geometry  Back, cluster percent correct –Back, how to use report

67. August 2012 Post-Test Workshop 67 Unmatched Reports Writing CST, CMA—Grs 4, 7 Only multiple-choice Only writing Students receive 2 reports if writing score not matched to multiple-choice score CST, CMA—Grs 8, 11 Only CMA and no CST Grades 8 and 11 require CST for History–Social Science

68. August 2012 Post-Test Workshop 68 Summary Do’s –Do compare mean scale score, percent at performance levels within same grade, same content area –Do compare cluster scores to the corresponding proficient range provided for this year Don'ts –Do not compare cluster scores to each other within test, nor across years, grades, content areas –Do not compare mean scale scores across grades, content areas Summary and Internet reports Data CDs Individual reports

69. Upcoming Dates EventDate Post-Test Guide posted on startest.orgAugust 10 State Superintendent’s public announcement August 31 Appendixes A & C Post-Test Guide posted on startest.org August 31 Web Reports availableAugust 31 Research files availableAugust 31 Post-Test Guide distributedFirst week of September Data Corrections windowOctober 1– November 9 August 2012 Post-Test Workshop 69

70. 2013 Setting Up Administration 1.Determine if November election changes number of instruction days. 2.Finalize instruction schedule. 3.Set up admin in STAR management system prior to December 1. 4.Enter and approve orders prior to December 1. After order approval, test administration dates NOT changeable. August 2012 Post-Test Workshop 70

71. August 2012 Post-Test Workshop 71 For more information see: http://www.cde.ca.gov/ta/tg/sr/ http://star.cde.ca.gov/ http://www.startest.org/ STAR Technical Assistance Center –800-955-2954 CDE Accountability –aau@cde.ca.gov –916-319-0863

72. To Contact the CDE CDE STAR office: 916-445-8765 –CST English–Language Arts, including writing: Jamie Contreras, jcontrer@cde.ca.gov, 916- 319-0353 –CMA English–Language Arts, including writing: Shobhana Rishi, srishi@cde.ca.gov, 916-391-0201 –CST and CMA Mathematics: Jane Liang, jliang@cde.ca.gov, 916-322-1854 –CST and CMA Science: Blessing Mupanduki, bmupanduki@cde.ca.gov, 916-319-0349 –CST History–Social Science, CAPA: Don Killmer, dkillmer@cde.ca.gov, 916-319-0350 August 2012 Post-Test Workshop 72

73. To Contact the CDE –STS Reading/Language Arts: Noelia Ramirez, nramirez@cde.ca.gov, 916- 319-0337 –STS Mathematics: Amalia Chamorro, achamorr@cde.ca.gov, 916-319-0192 –Directions for Administration, Coordinator Manual: Mark McLean, mmclean@cde.ca.gov, 916-319-0356 –Reports, Post-Test Guide, Sherry Shi, sshi@cde.ca.gov, 916-319-0285 August 2012 Post-Test Workshop 73

74. August 2012 Post-Test Workshop 74 Next Webcast 2013 STAR for Students with Disabilities Webcast September 19, 2012

75. Early Assessment Program Results August 8, 2012 Connie Grueter

76. Early Assessment Program (EAP) EAP results are being accepted at participating California Community Colleges (CCC). –List of participating colleges is available on the CCC website at www.CollegeEAP.org. Check the site often for an updated list of campuses. August 2012 Post-Test Workshop 76 PowerPoint slides - www.CollegeEap.org, Educators, About EAP.

77. Early Assessment Program (EAP) An EAP informational web site has been developed. www. CollegeEAP.org. –The site provides EAP information for both CSU and CCC –It is a resource for Students, Parents, and Educators August 2012 Post-Test Workshop 77 PowerPoint slides - www.CollegeEap.org, Educators, About EAP.

78. www.collegeEAP.org August 2012 Post-Test Workshop 78

79. August 2012 Post-Test Workshop 79 Early Assessment Program (EAP) Reporting Results –New this year – Conditional Exemptions for EAP English –Ready for CSU and participating CCC college-level English/mathematics courses – Conditional This status in conjunction with a grade of “C” or better in an approved English/math course or a supervised e- learning program can be exempt from the EPT or ELM for CSU or other English or math placement tests at participating California Community Colleges (CCC) A list of courses and e-learning programs can be found at www.csumathsuccess.org PowerPoint slides - www.CollegeEap.org, Educators, About EAP.

80. Early Assessment Program (EAP) August 2012 Post-Test Workshop 80 –Sample EAP Box on the STAR Student Report The EAP box can be found on page 70 of the Post-Test Guide PowerPoint slides - www.CollegeEap.org, Educators, About EAP.

81. Early Assessment Program (EAP) Web Sites –www.CollegeEAP.org  Links to CSU and CCC informational sites for Students, Parents, and Educators –www.CalState.edu/EAP  Provides a list of EAP Coordinators, EAP test blueprints, and informational materials –www.CCCCO.edu/EAP  Provides a list of participating Community Colleges August 2012 Post-Test Workshop 81 PowerPoint slides - www.CollegeEap.org, Educators, About EAP.