Guide to Test Interpretation Using DC CAS Score Reports to Guide Decisions and Planning District of Columbia Office of the State Superintendent of Education CTB/McGraw-Hill September 26–27, 2013

Introductions OSSE  Jessica Enos Data Analysis Manager  Jeff Noel Director of Data Management  Tommy Shen Senior Data Analyst CTB  Amy Dement-Dorey Senior Program Manager  Glenn Gage Program Associate  Gabriel Martinez Content Development Lead  Rick Mercado Standard Setting Specialist 2 Copyright © 2013 CTB/McGraw-Hill LLC.

Objectives  Participants will be prepared to –describe the information found in DC CAS score reports accurately; –share this information with their colleagues and stakeholders; and –use the information on those reports to make well- informed, practical decisions. 3 Copyright © 2013 CTB/McGraw-Hill LLC.

About This Presentation  Overview the Guide to Test Interpretation (GTI)  Describe the reports available in the GTI  Discuss best practices for interpreting and using the reports  Provide hands-on experience and discussion with colleagues 4 Copyright © 2013 CTB/McGraw-Hill LLC.

Agenda  About the DC CAS and the GTI  Discuss the tools and information needed to make good decisions  Discuss the types of score reports available  Provide hands-on experience using score reports  Discuss how to use the information to effect positive changes in the classroom  Closing discussion 5 Copyright © 2013 CTB/McGraw-Hill LLC.

About the DC CAS  Statewide standards-based tests for: –Reading (Grades 2–10) –Mathematics (Grades 2–8 & 10) –Science (Grades 5 & 8), Biology (High School) –Composition (Grades 4, 7, & 10) –Health (Grades 5, 8, & High School) 6 Copyright © 2013 CTB/McGraw-Hill LLC.

What does the DC CAS measure?  DC CAS: –is aligned to the DC content standards –assesses students’ proficiency in the knowledge and skills specified in the DC content standards –communicates student proficiency using performance levels and performance level descriptors (PLDs) 7 Copyright © 2013 CTB/McGraw-Hill LLC.

Performance Level Descriptors  Abbreviated PLDs are printed on the backs of score reports.  Full PLDs can be found: –in Appendix A of the GTI –on the OSSE website at osse.dc.gov/service/dc-cas-performance-level-descriptors osse.dc.gov/service/dc-cas-performance-level-descriptors 8 Copyright © 2013 CTB/McGraw-Hill LLC.

Guide to Test Interpretation  The GTI provides information about how results are reported and how those results can be interpreted and acted upon.  It arrives in schools in the fall, following the spring test administration. 9 Copyright © 2013 CTB/McGraw-Hill LLC.

Inside the GTI  Sample DC CAS score reports  PLDs for each tested grade and content area  Accommodations that are allowable for Students with Disabilities (SWDs) and English Language Learners (ELLs) 10 Copyright © 2013 CTB/McGraw-Hill LLC.

Important Companion Document  The DC CAS Resource Guide includes: –Descriptions of the reporting categories –Content standards that are targeted in DC CAS –Scoring rubrics and scoring guides 11 Copyright © 2013 CTB/McGraw-Hill LLC.

DC CAS Resource Guide  The 2013 DC CAS Resource Guide is now available on the CTB website: –Then click on “Training and Ancillary Materials”  The 2014 DC CAS Resource Guide is scheduled to be posted later in the year. 12 Copyright © 2013 CTB/McGraw-Hill LLC.

Three Types of Score Reports Student Reading Mathematics Science Biology Composition Classroom Roster Report Content Strands Roster Report Scoring Categories Roster Report School Content Strands Summary Report Item Analysis Summary Copyright © 2013 CTB/McGraw-Hill LLC. 13

Score Report Terminology  Raw scores –Number (or percentage) of score points achieved  Scale scores (SS) –Derived from raw scores –Enable comparison across school years, in the same grade –Do not enable comparison across grades or content areas 14 Copyright © 2013 CTB/McGraw-Hill LLC.

Score Report Terminology (cont.)  Percent index (PI) scores –PI scores can be interpreted very much like percent correct scores and range 0–100. –They have been shown to be more reliable than percent correct scores, representing an expected percent correct if students had been given all possible items measuring the content area. –PI scores are computed using the scale scores derived from item response theory (IRT). Copyright © 2013 CTB/McGraw-Hill LLC. 15

Score Report Terminology (cont.)  Content strands –Subcategory of related content standards in a content area –Usually synonymous with “reporting category”  Performance levels –Levels of performance on DC CAS, with a focus on “proficiency”  Performance level descriptors (PLDs) –Summarize the knowledge, skills, and abilities expected of students in each performance level 16 Copyright © 2013 CTB/McGraw-Hill LLC.

Score Report Terminology (cont.)  Roster –List of student scores organized by class or grade  Disaggregated results –Test results reported for separate student subgroups  Item analysis summary –Information on student performance for each test item 17 Copyright © 2013 CTB/McGraw-Hill LLC.

Performance Level Descriptors Example of the back of a Grade 2 student report. Also shown on page 11 of the 2013 GTI. 18

Student Report, Reading/Math Example of a Grade 2 Student Report for Reading and Mathematics. As shown on page 10 of the 2013 GTI. 19

Using the Student Report  Parents and teachers can use this report to: –Assess a student’s proficiency (i.e., knowledge and skills) in the DC CAS content standards using the PLD for the student’s performance level –Compare the student’s content strand scores to state average content strand scores 20 Copyright © 2013 CTB/McGraw-Hill LLC.

Student Report Information  Scale scores and performance levels –Scale scores and performance levels are comparable across years for that content area and grade level. –Scale scores and performance levels are not comparable across grades.  Content strand percent correct scores –Students’ content strand performance can be compared with the average State performance. 21 Copyright © 2013 CTB/McGraw-Hill LLC.

Student Report, Composition Example of a Grade 4 Student Report for Composition. As shown on page 13 of the 2013 GTI. 22

Content Strands Summary Report Example of a Grade 2 Mathematics Content Strands Summary Report. As shown on page 17 of the 2013 GTI. 23

Using the Content Strands Summary Report  Teachers and principals can use this report to: –Evaluate grade-level performance on the content strands by comparing the school’s performance to state performance on each content strand. –Identify content strands where instructional approaches, instructional materials, or emphasis (e.g., number of instructional hours, homework) may need to be reconsidered. Copyright © 2013 CTB/McGraw-Hill LLC. 24

Content Strands Summary Report Information  PI score for the school and state, for each content strand –You cannot compare these PI scores across content strands or across years. –PI scores are not adjusted for the varying difficulty levels of the items in each strand. Copyright © 2013 CTB/McGraw-Hill LLC. 25

Student Roster Report 26 Example of a Grade 6 Roster Report. Also shown on page 15 of the 2013 GTI.

Using Student Roster Reports  Roster reports list and summarize the information found in other reports.  The student roster report provides: –Alphabetical list of students, by class and grade. –For multiple grades in one class, a separate report is produced for each grade. 27 Copyright © 2013 CTB/McGraw-Hill LLC.

Content Strands Roster Report Example of a Grade 8 Mathematics Content Strands Roster Report. Also shown on page 19 of the 2013 GTI. 28

Content Strands Roster Reports Information  Part 1: Roster –Alphabetical list of students, by class and grade. –Scores for the total test: scale score, raw score, percent correct, scale score error range, performance level, PI. –Scores for the content strands: number of items (and # points possible), minimum PI for proficiency, and performance indicator/PI score. Performance indicators allow for comparison of student proficiency across content strands  Part 2: Summary –Summary by class and grade (i.e., mean PI, mean scale score, mean raw score). Copyright © 2013 CTB/McGraw-Hill LLC. 29

Content Strands Roster Report Summary Example of a Grade 8 Mathematics Content Strands Roster Report Summary. Also shown on page 21of the 2013 GTI. 30

Item Analysis Summary Example of a Grade 8 Mathematics Item Analysis Summary. Also shown on page 25 of the 2013 GTI. 31

Using the Item Analysis Summary  Teachers and principals can use this report to: –Evaluate grade-level performance on the objectives within each content strand –Identify objectives where instructional approaches, instructional materials, or emphasis (e.g., number of instructional hours, homework) may need to be reconsidered –Compare the school’s performance to state performance on each objective –Compare performance on multiple choice and constructed response items 32 Copyright © 2013 CTB/McGraw-Hill LLC.

Item Analysis Summary Information  Percent correct score for the school and state, for items in each tested standard –Do not compare percent correct scores across items or across years; they are not adjusted for difficulty. 33 Copyright © 2013 CTB/McGraw-Hill LLC.

Transition to the Common Core 2011 Transition to the Common Core State Standards began 2012 DC CAS Reading and Writing were aligned to Common Core DC CAS Mathematics focused on essential knowledge and skills students needed to be successful with Common Core 2013 Mathematics was aligned to Common Core 2014 All DC CAS assessments are aligned to Common Core Copyright © 2013 CTB/McGraw-Hill LLC. 34

Practice with Reports  In your own or in pairs, use the reports in the GTI to answer the three questions on the Practice with Reports worksheet. –We’ll review the answers in several minutes. 35 Copyright © 2013 CTB/McGraw-Hill LLC.

Practice with Reports 1.How many students in my class scored Proficient or above in Math overall? a.Which report(s) can I use to answer the question? Roster Report (p. 15); or Content Strands Roster Report, Part 2 (p. 21). b.What is the answer using the report in the GTI? Roster Report, 4 students; or CSRR2, 26 students. 36 Copyright © 2013 CTB/McGraw-Hill LLC.

Practice with Reports 2.For which content strand(s) did my class have the most difficulty reaching proficiency? a.Which report(s) can I use to answer the question? Content Strands Roster Report, Part 1 (p. 19) b.What is the answer using the report in the GTI? Geometry (5 students did not score proficient) 37 Copyright © 2013 CTB/McGraw-Hill LLC.

Practice with Reports (cont.) 3.For which content strand(s) is my school’s performance higher than the state (DC public schools, including charters), for the grade level shown? a.Which report(s) can I use to answer the question? Content Strands Summary Report (p. 17) b.What is the answer using the report in the GTI? Measurement & Data shows the school ahead of State. 38 Copyright © 2013 CTB/McGraw-Hill LLC.

Small Group Discussion 1.On the Data Exploration Worksheet, identify one or two instructional challenges you face this school year with a class or grade level of students. 2.Identify specific information in the reports that might help you examine these challenges. 3.Use the information to sketch out a plan for addressing the challenges this year, and what information would be expected to change if there is improvement. 39 Note: Some types of questions (e.g., growth) are computed separately from the reports, and are not able to be answered using these reports.

Large Group Discussion  Share your insights, comments, and unanswered questions: –What challenges did you identify? –What information in the score reports was useful? –What thoughts and questions do you have about the score reports? –What actions, interventions, or other next steps are you considering? –What additional information or support do you need? 40 Copyright © 2013 CTB/McGraw-Hill LLC.

Closing Discussion  Planning: –What are your next steps? –Will DC CAS reports help with your planning? –Will additional information be needed to plan for the 2013–2014 school year?  Please give feedback on today’s presentation on the workshop evaluation. 41 Copyright © 2013 CTB/McGraw-Hill LLC.

For Further Information  For further assistance regarding score reports or the Guide to Test Interpretation, contact:  For more DC CAS information and resources, visit the CTB website at: Copyright © 2013 CTB/McGraw-Hill LLC.

Thank you!  Rick Mercado   Gabe Martinez   Glenn Gage   Amy Dement-Dorey  43 Copyright © 2013 CTB/McGraw-Hill LLC.