2008 STAR Interpreting and Using Results August/September 2008

Post-Test Workshop2 Objectives Workshop participants will be able to: Describe the purposes of STAR reports Interpret STAR results Explain key statistics Compare and contrast types of reports Identify proper uses of reports

August/September 2008Post-Test Workshop3 Agenda Quiz Post Test Guide What’s New? Results and Statistical Analysis Using Results Summary and Internet Reports Individual Student Reports Teacher Reports

August/September 2008Post-Test Workshop4 Post-Test Guide Layout Part I: General Info –New in 2008 –Interpreting Reports –Comparing Results Part II: Report Descriptions –Summary –Individual –Teacher –Internet Part III: Appendices –Reporting clusters –Scale score ranges –Statistical tables –Test Score Verification procedures –Demographic Data Correction procedures –California reading list –Glossary Post-Test Guide Pages (Pp) i–iii

August/September 2008Post-Test Workshop5 What’s New in 2008 CMA reports –Percent correct –Grades 3–5 STS reports –Percent correct –Grades 5–7 added CAPA Science percent correct Ethnicity for Economic Status subgroup summary (CSTs only) STAR Student Reports sorted for district –All without addresses –All with addresses Post-Test Guide P 2

August/September 2008Post-Test Workshop6 Results: Purposes of STAR Reports Report progress of students toward proficiency on the state’s academic content standards Inform where improvement needed –To help students’ achievement –To improve educational programs Provide data for state and federal accountability programs Post-Test Guide P 3

August/September 2008Post-Test Workshop7 Results: Performance Levels State goal: PROFICIENT or higher –Scale score of 350 or higher for CSTs –Scale score of 35 or higher for CAPA Other performance levels: –Advanced –Basic (300 CSTs, 30 CAPA) –Below Basic –Far Below Basic –CST and CAPA cut points vary for Advanced, Below Basic, and Far Below Basic by Subject Grade Post-Test Guide P 7

August/September 2008Post-Test Workshop8 Results: Scale Scores Scale scores allow same score to mean same thing across test forms within grade and content area and across time Scale score ranges by program: –CSTs: 150–600 for each grade and subject –CAT/6 Survey: 1–999 for each grade and subject –CAPA: 15–60 for each grade and subject –CMA, STS, CAPA Science: Not available until 2009 Post-Test Guide P 7–8

August/September 2008Post-Test Workshop9 Results: Equating Psychometric procedure Adjusts for test difficulty Additional information in CST Technical Report on CDE Web site For 2008 –CSTs and CAPA (except science) are equated –CMA (3–5), CAPA (science), and STS (2–4) and will be equated starting in 2009 CAT/6 nationally norm-referenced; not equated for STAR Post-Test Guide Page 7

August/September 2008Post-Test Workshop10 Results: Reporting Clusters (Content Areas) Four to six clusters for each content area (subject) May be useful as indicators of individual or group strengths and weaknesses But… Reporting clusters should be interpreted with caution Post-Test Guide P 36 Appendices A & B

August/September 2008Post-Test Workshop11 Results: Reporting Cluster Cautions Based on small numbers of items; therefore may not be reliable or generalizable NOT equated from year to year Should not compare reporting cluster percent correct from year to year Post-Test Guide P 8–10 Appendices A & B

August/September 2008Post-Test Workshop12 Interpreting Reporting Clusters or Content Areas in the Same Year CSTs—Performance on the clusters for students statewide scoring from –Lowest score for proficient to –Lowest score for advanced Post-Test Guide P 8–10; 84 Appendices A & B

August/September 2008Post-Test Workshop13 CSTs 2008 Reporting Clusters — Number of Questions and Average Percent Correct Post-Test Guide Page 138 (Appendix A)

August/September 2008Post-Test Workshop14 Examples—Interpreting Reporting Clusters for CST for Geometry Post-Test Guide P 9

August/September 2008Post-Test Workshop15 Using Results For instructional decisions with other data: CSTs, CMA, CAPA, STS Used in Academic Performance Index (API) calculations: CSTs, CAT/6 Survey, CAPA Used in adequate yearly progress (AYP) calculations for grades 2–8 in ELA and math for CSTs and 2–8 and 10 in ELA and math for CAPA Post-Test Guide P 3

August/September 2008Post-Test Workshop16 Year-to-Year Comparisons Do Compare Same Grade and Same Content Area Mean scale score –Same content and grade, varying years Percent in each performance level –Same content by grade across years e.g., 2007 ELA grade 10 with 2008 ELA grade 10 Post-Test Guide Pp 11 – 14

August/September 2008Post-Test Workshop17 Year-to-Year Comparisons Do Compare Same Grade or Same Content Area Percentage of students scoring at PROFICIENT and above –For a given grade and subject, e.g., Percent proficient and above for grade 3 math in 2007 & 2008 –For a given subject and aggregated grades, e.g., % proficient and above for grades 2 to 6 math in 2007 & 2008 –Across grades and a subject, e.g., % proficient and above in all courses and all grades Post-Test Guide Pp 11 – 14

August/September 2008Post-Test Workshop18 Year-to-Year Comparisons DON’T Compare Individual scale scores or statistics based on scale scores for different grades or content areas –Subjects by grade are independently scaled –Different content standards are measured in different grades Cohorts across grades CMA, STS scores because not scaled Post-Test Guide Pp 11 – 14

August/September 2008Post-Test Workshop19 Example—Using CST Results to Compare Grade Results from Year to Year Post-Test Guide P CST for ELA 2008 CST for ELA Grade% Prof or Above Difference Grade 231%35%4% Grade 333% 0% Grade 429%31%2% Grade 534%32%-2% Grade 631%32%1% All Grades32%33%1%

August/September 2008Post-Test Workshop20 Aggregate (Summary) Reports What are they? –Student Master List Summary –Student Master List Summary EOC –Subgroup Summary –Group Summary: CAT/6 Survey Report Emphasis: California Standards Tests (CSTs) –Criterion-referenced tests –Progress is measured in percent of students scoring proficient and advanced Post-Test Guide Pp 17–21

August/September 2008Post-Test Workshop21 Student Master List Summary By grade CSTs, CMA, CAPA, and STS Lists subjects % and # at each Performance Level –Not for CMA, CAPA Science, STS Mean Scale Score –Not for CMA, CAPA Science, STS Reporting Cluster: Mean Percent Correct –Not for CMA, CAPA, and STS grades 5–7 Post-Test Guide Pp 17; 26 – 37

August/September 2008Post-Test Workshop22 Student Master List Summary Post-Test Guide P 27

August/September 2008Post-Test Workshop23 Student Master List Summary Post-Test Guide P 27

August/September 2008Post-Test Workshop24 Who Counts? Number Enrolled –All CST, CMA, CAPA, and STS multiple- choice answer documents submitted as scorable –Minus Documents marked as “Student enrolled after the first day of testing and was given this test” Post-Test Guide P 28

August/September 2008Post-Test Workshop25 Who Counts? Number, Percent Tested –All CST, CMA, CAPA, STS answer documents with one or more answers –Plus Z = Tested but marked no answers –Not included P = Parent/guardian exemptions A = Students absent T = Enrolled first day, not tested, tested at previous school E = Not tested due to significant medical emergency Students with inconsistent grades Post-Test Guide P 28

August/September 2008Post-Test Workshop26 Who Counts? Number Valid Scores –For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct). –Not included: Incomplete tests Modified tests Non-English learners who took the STS Unknown EOC math or science tests Inconsistent grades Why does STS not have 100% valid scores? Post-Test Guide P 27

August/September 2008Post-Test Workshop27 Who Counts? Number Tested with Scores –All tests taken, including those taken with modifications, that receive a score –All STS and CMA tests taken –Not included Incomplete tests Non-English learners who took the STS Unknown EOC math or science tests Inconsistent grades Post-Test Guide P 28

August/September 2008Post-Test Workshop28 Who Counts? Performance Level Counts for CSTs – Advanced, Proficient, Basic, Below Basic All valid scores falling in the performance level –Far Below Basic All valid scores falling in the performance level Modified tests (in aggregate reporting and accountability only) Post-Test Guide P 29

August/September 2008Post-Test Workshop29 Student Master List Summary Performance Levels Post-Test Guide P 30

August/September 2008Post-Test Workshop30 Student Master List Summary Reporting Clusters Compare to: Average percent correct range for students statewide who scored proficient on the total test Post-Test Guide P 30

August/September 2008Post-Test Workshop31 Student Master List Summary: Writing B=Blank C=Copied Prompt I=Illegible L=Language other than English R=Refusal T=Off Topic W=Wrong prompt (Prompt from an earlier administration) Post-Test Guide P 30

August/September 2008Post-Test Workshop32 Student Master List Summary: End-of-Course By Subject Lists each grade eligible to take test Math (Grades 7–11) –General Math, Algebra I, Geometry, etc. Science (Grades 9–11) –Earth Science, Biology, Chemistry, etc. History–Social Science (Grades 9–11) –World History Same statistics as grade-level Student Master List Summary Post-Test Guide Pp 32 – 37

August/September 2008Post-Test Workshop33 Subgroup Summary: CSTs, CMA, CAPA, and STS Disability Status –Based on “Disability Code” Economic Status –Based on “NSLP” & “Parent Ed Level” Gender English Proficiency Primary Ethnicity Ethnicity for Economic Status: CSTs and CAPA (ELA and Math) only Post-Test Guide Pp 38 – 64

August/September 2008Post-Test Workshop34 Subgroup Summary: Ethnicity for Economic Status Post-Test Guide Pp 60 – 64

August/September 2008Post-Test Workshop35 Subgroup Summary: Ethnicity for Economic Status Post-Test Guide Pp 60 – 64

August/September 2008Post-Test Workshop36 Subgroup Summary: Ethnicity for Economically Disadvantaged Post-Test Guide Pages 60 – 64

August/September 2008Post-Test Workshop37 Internet Reports Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS Available to public online More subgroups than paper reports –English-language fluency –Parent education –Special Program Participation Access from Post-Test Guide Pp 123 – 134

August/September 2008Post-Test Workshop38 Internet Reports: Available Reports CSTs –Mean scale scores –Percents by performance levels CST Summary –Percent proficient and advanced CMA –Average percent correct –Grades 3–5 only STS –Average percent correct –Grades 2–7 only CAPA –Mean scale scores –Percents by performance levels –Average percent correct for science CAT/6 Survey –Mean scale scores –NPR for “Avg. Student” –Percent above percentile cuts –Grades 3 and 7 only Post-Test Guide Pp

August/September 2008Post-Test Workshop39 Internet Reports Access Reports Post-Test Guide Pp 124

August/September 2008Post-Test Workshop40 Internet Reports: CST Sample Post-Test Guide P 127

August/September 2008Post-Test Workshop41 Internet Reports: CST Summary Sample Post-Test Guide P 128

August/September 2008Post-Test Workshop42 Internet Reports: CMA % Correct Post-Test Guide P 129

August/September 2008Post-Test Workshop43 Internet Reports: STS % Correct Post-Test Guide Pp 133

August/September 2008Post-Test Workshop44 Exercise Work with a partner or in small groups Plan what data you’ll present to leaders in your district Remember the Do’s and Don’ts and Cautions

Break — 15 minutes

August/September 2008Post-Test Workshop46 Individual Reports STAR Student Record Label –Adhesive label to affix to student’s permanent school record STAR Student Master List –Alphabetical list of students and their scores –Tests listed in order within grade CSTs-CAT/6 Survey CMA-CAT/6 Survey CAPA STS STAR Student Report: individual’s scores –2 two-sided color copies –For parents/guardians –For each test Post-Test Guide Pp 70 – 112

August/September 2008Post-Test Workshop47 Student Record Label: Student Name and Identification Post-Test Guide Pp 70 – 73

August/September 2008Post-Test Workshop48 Student Record Label: CST for Grade 10 Results Post-Test Guide Pp 70 – 73

August/September 2008Post-Test Workshop49 Student Master List: CSTs/CMA-CAT/6 for Grade 3 Sample Post-Test Guide Pp 74 – 79

August/September 2008Post-Test Workshop50 Student Master List: CSTs/CMA-CAT/6 for Grade 3 Sample Post-Test Guide Pp 74 –7 9

August/September 2008Post-Test Workshop51 Student Report: CST for Grade 11 Sample Post-Test Guide Pp 81 – 90

August/September 2008Post-Test Workshop52 Student Report: CST for Grade 11 Sample Post-Test Guide Pp 81–90

August/September 2008Post-Test Workshop53 Student Report: CSTs for Grade 10 Sample Post-Test Guide Pp 81–90 ♦=Percent correct obtained by the student on the reporting cluster/content area ▬▬▬=Range of percent correct scores on the reporting cluster for students who scored proficient on the total test Student Name On Back

August/September 2008Post-Test Workshop54 Reporting Cluster Range for Each Content Area by Grade Reporting cluster average percent correct from all minimally proficient scores (350) Reporting cluster average percent correct from all minimally advanced scores minus one (e.g., 401, grade 4 math)

August/September 2008Post-Test Workshop55 Explain to Parents Scale Score  Average % correct cluster score x 600 Reporting Clusters not comparable –Different difficulty –Varying number of questions –Average % correct of clusters  % correct of total scale score Scale scores –Use conversion tables and other statistical techniques –Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient)

August/September 2008Post-Test Workshop56 Student Report: CST for Grade 7 with CAT/6 Sample Post-Test Guide P 85

August/September 2008Post-Test Workshop57 Student Report: CST for Grade 11 with EAP Sample Post-Test Guide Pp 81–90

August/September 2008Post-Test Workshop58 Other Student Reports CMA (Pp 91–98) –Percent correct, no performance levels –Back, “About the CMA,” P 93 –No cluster reporting CAPA (Pp 99–104) – Back, performance level descriptors for ELA, math P 104 STS (Pp 105–112) –In Spanish –Percent correct, no performance levels –Reporting clusters percent correct, grades 2–4 –Back, how to use report P 108

August/September 2008Post-Test Workshop59 Exercise Work with a partner or in small groups Use a Student Report sample and explain content reporting diamonds and bars on back

August/September 2008Post-Test Workshop60 Unmatched Report Grade 3 –CST but no CAT/6 Survey –CMA but no CAT/6 Survey –CAT/6 Survey but no CSTs or CMA Grades 4 and 7 –Unmatched Multiple-Choice Report CST Multiple-choice but no writing test –Unmatched Writing Report Writing but no CSTs multiple-choice test

August/September 2008Post-Test Workshop61 Unmatched Report:

August/September 2008Post-Test Workshop62 Reports for Teachers Summary of results for previous year’s class(es) Two types of summary reports –By grade level (e.g., grade 11 ELA) or course (e.g., Algebra I) –By teacher name or group name (e.g., GATE) Separate reports for: –CST ELA –CST Math Not available for: –CST science or history–social science –CMA –CAPA –STS Post-Test Guide Pp 113 – 122

August/September 2008Post-Test Workshop63 Reports for Teachers Individualized reports (teacher name or group name) –if name on SGID or Pre-ID –SGID priority over Pre-ID Group reports (grade level or course): all ELA and math teachers should receive EOC Math –Grades 8–11 by subject only –Not grade: e.g., All Algebra I in a school Post-Test Guide Pp 113 – 122

August/September 2008Post-Test Workshop64 Report for Teachers: Sample Post-Test Guide Pp 114 – 122

August/September 2008Post-Test Workshop65 Report for Teachers: 2008 Performance By Reporting Clusters Reporting clusters defined under this section. Post-Test Guide Pp 114

August/September 2008Post-Test Workshop66 Report for Teachers: Cluster Reporting For each reporting cluster, the average percent correct achieved by: –Statewide students –District students –Your students –Proficient range Definition of symbols  = average % correct for teacher’s students ▬▬▬ = average % correct range for students statewide who scored proficient on total test

August/September 2008Post-Test Workshop67 Report for Teachers Back School % at each performance level by subgroups % Proficient and above –2007 vs –School; district; state –By subgroup Resources Post-Test Guide P 117

August/September 2008Post-Test Workshop68 Early Assessment Program (EAP) Reporting Results –Student Statuses are on the STAR Report –Ready for CSU college-level English/mathematics courses –Ready for CSU college-level mathematics courses – Conditional This status in conjunction with a grade of “C” or better in an approved math course or a supervised e-learning program can be exempt from the ELM A list of courses and e-learning programs can be found at –Not yet demonstrating readiness for CSU college-level English/mathematics courses

August/September 2008Post-Test Workshop69 Summary Do’s and Don’ts –Do compare mean scale score, percent at performance levels within same grade, same content area –Don’t compare across grades, content area Summary reports Individual reports Teacher reports Quiz answers Evaluations

August/September 2008Post-Test Workshop70 Quiz Answers 1. B 2. D 3. A 4. C 5. D