2017 CAASPP Post-Test Workshop: Using Assessment Results to Inform Teaching and Learning Assessment Services

Learning Goal Participants will be equipped with the information, and tools necessary to: Understand the CAASPP scores Access the various reports Analyze the results Use the results and available resources to improve teaching and learning Assessment Services

Moving Beyond the Scores Agenda Principles of Scoring Accessing the Reports Analyzing the Results Moving Beyond the Scores Assessment Services

Principles of Scoring Computer Adaptive Testing Review Contributions in the Overall Score Scores Students will Receive Principles of Scoring

A Computer Adaptive Test (CAT) is Based on: Large item bank - covering all areas assessed and varying levels of difficulty (statistically calibrated on a common scale with ability estimates) Recommended blueprint - focuses the selection of questions from the test bank to appropriate content so the structure of the test is similar for every student See handout Assessment Services

A Computer Adaptive Test (CAT) is Based on (Continued): Programming language (or algorithm) - a step-by-step approach that tells the CAT what to do next based on students’ answers Algorithm rules – set of rules to ensure each student’s test contains the proper types of questions and content covered Balance – enough of the concept Question type – selected response, constructed response Reading length – short, medium, long Difficulty – appropriate for grade level Assessment Services

How Does a CAT Work? Example: A Student of Average Ability Expanded Very High Ability High Med-High Medium PATTERN Med-Low Low Expanded Very Low 1 2 3 4 5 6 7 8 9 10 Test Questions Assessment Services Answers (R/W) R R W R W W W W R R

Scoring the CAT Final scale scores are based on item PATTERN SCORING: As a student progresses through the test, his or her pattern of responses is tracked and revised estimates of the student’s ability are calculated Successive test questions are selected to increase the precision of the student’s level-of-achievement estimate Scores from the CAT portion of the test are based on the difficulty of the items that were right or wrong, NOT on the total number of correct answers The test question bank for a particular grade level are designed to include an enhanced pool of test questions that are more or less difficult for that grade, but still match the grade’s test blueprint Assessment Services

Key Principle of Scoring Assessment Services

Performance Tasks (PTs) Performance task (PT) = set of complex questions challenging students to apply their knowledge and skills to respond to real-world problems PTs are not adaptive and, therefore, are not targeted to a student’s specific ability level The items associated with the PTs may be scored by a machine or by human raters Assessment Services

Final Scoring: Contribution of CAT and PT Sections For each student, the responses from the PT and CAT portions are merged for final scoring Number of Items Defined by Test Blueprints ELA/Literacy Mathematics Grade CAT PT 3–5 38–41 4–6 31–34 2–6 6–8 37–42 30–34 11 39–41 33–36 See handout Assessment Services

Final Scoring: Contribution of CAT and PT Sections (cont.) Based on the test blueprint, the CAT section is emphasized because there are more CAT items/points than PT items/points Claims with more items/points are emphasized Because scores are based on pattern scoring, groups of items that are more difficult will have a larger contribution on final scores There is no specific weight associated with either the PT or CAT Assessment Services

Final Scoring: Contribution of the CAT and PT Relative contribution of the PT and CAT sections on the total score will vary across students since the overall score Is based on pattern scoring (for the CAT) Depends on performance on the CAT section Depends on performance on the PT section Depends on the average difficulty of the PT section Example: A PT might have a significant negative contribution on a students overall score if they did extremely well on the CAT but performed poorly on a very easy PT section Upshot: All parts of the Smarter Balanced Assessments are important Assessment Services

Reporting Scale After estimating the student’s overall ability, it is mapped onto the reporting scale Scores are on a vertical scale Expressed on a single continuum for a content area Measures student growth over time across grade levels For each grade level and content area, there is a separate scale score range Assessment Services

Overall Achievement Levels Achievement level classifications are based on overall scores for English language arts and mathematics 4 Exceeded the Standard 3 Met the Standard 2 Students will receive one of four score levels. Students scoring in the two top levels are on a path to be college and career ready at high school graduation. Students scoring below the “Met the Standard” level will need further development to get on path. Elementary & Jr. High: We are in a transition year – More students are likely to need further development to meet the new standards . But its ok, as students spend additional time in class rooms employing the new curriculum and teaching, results will improve. High School: We are in a transition periods – Results may show more students are likely to need further development to meet the new standards. But it’s ok – students still have another year to continue to progress. They are now on a focused path to gaining these skills. Public colleges and universities in several states have decided to use the results of the Grade 11 ___________exams to help determine whether admitted students are ready to proceed directly to entry-level, credit-bearing courses and can skip the placement tests that are typically administered.  In all of these cases colleges will continue to use their existing admissions criteria. <insert state specific policy> Nearly Met the Standard 1 Has Not Met the Standard

Smarter Balanced Scale Score Ranges for ELA/Literacy Grade Level 1 Level 2 Level 3 Level 4 3 2114–2366 2367–2431 2432–2489 2490–2623 4 2131–2415 2416–2472 2473–2532 2533–2663 5 2201–2441 2442–2501 2502–2581 2582–2701 6 2210–2456 2457–2530 2531–2617 2618–2724 7 2258–2478 2479–2551 2552–2648 2649–2745 8 2288–2486 2487–2566 2567–2667 2668–2769 11 2299–2492 2493–2582 2583–2681 2682–2795 See handout Assessment Services

Smarter Balanced Scale Score Ranges for Mathematics Grade Level 1 Level 2 Level 3 Level 4 3 2189–2380 2381–2435 2436–2500 2501–2621 4 2204–2410 2411–2484 2485–2548 2549–2659 5 2219–2454 2455–2527 2528–2578 2579–2700 6 2235–2472 2473–2551 2552–2609 2610–2748 7 2250–2483 2484–2566 2567–2634 2635–2778 8 2265–2503 2504–2585 2586–2652 2653–2802 11 2280–2542 2543–2627 2628–2717 2718–2862 See handout Assessment Services

Measurement Precision: Error Bands Tests are imprecise! For each scale score estimated for a student, there is measurement error associated with it Measurement error occurs due to factors unrelated to learning (e.g. mood, health, testing conditions) Error bands are used to construct an interval estimate corresponding to a student’s true ability/proficiency for a particular content area with a certain level of confidence The error bands used to construct interval estimates were based on one standard error of measurement (If the same test is given to a student multiple times, the student will score within this band about 68 percent of the time) Measurement error refers to the degree of imprecision or uncertainty in the assessment Assessment Services

Achievement Levels for Claims Achievement levels for claims are very similar to sub-scores; they provide supplemental information regarding a student’s strengths or weaknesses Only three achievement levels for claims were developed since there are fewer items within each claim A student must complete most items within a claim to receive an estimate of his or her performance on a claim Achievement levels for claims are based on the distance a student’s performance on the claim is from the Level 3 “standard met” criterion Assessment Services Copyright © 2009 Educational Testing Service.

Achievement Levels for Claims Below Standard Above Standard Near Standard English Language Arts Reading Writing Speaking & Listening Research/Inquiry Mathematics Concepts & Procedures Problem Solving Communicating Reasoning Modeling & Data Analysis

Questions about the scores that students receive?

Accessing the various reports ORS Reports Illuminate Reports CDE Reports Individual Student Reports Accessing the various reports

Access & Timeline for 2017 CAASPP Reports Preliminary CAASPP Results in ORS Illuminate CAASPP Reports CDE CAASPP Reporting Site Individual CAASPP Student Reports ACCESS Available to site coordinators, principals, and instructional cabinet Available to SDUSD staff who have access to Illuminate (e.g. teachers, principals) Available to the public via the CDE website Mailed home to parents by Assessment Services; school copies sent to sites via truck AVAILABLE Approximately three weeks after a student tests July (preliminary) Late August (final) Late August or Early September September Assessment Services

Test Results in the Online Reporting System (ORS) Online Reporting System (ORS) = a Web-based system that displays score reports and completion data for each student who has taken the assessments Results are available approximately three weeks after a student completes both parts—CAT and PT—of a content area Test results are added nightly See handout Assessment Services

Online Reporting System (ORS) After logging into ORS via TOMS, select [Score Reports] 1 ORS Dashboard 2 Assessment Services

ORS Home Page Dashboard: Select Test, Administration, and Enrollment Status 1 Select Test 1 Enrollment Status View Select Administration 2014–15 data are final 2015–16 data are final 2016–17 are partial Assessment Services

ORS Dashboard: Select Grade and Subject 2 Test results are available for students who have completed both parts—CAT and PT. Students are in the process of completing tests—results are not yet available. Assessment Services

ORS Subject Detail Report Legend Demographic Subgroups Exploration Menu Assessment Services

ORS Exploration Menu Assessment Services

ORS Claim-Level Report Detail Assessment Services

ORS Student Listing Report Sort Assessment Services

Reports in Illuminate Assessment Services

Reports in Illuminate SBA Performance Summary = this report summarizes SBA ELA or Math performance levels and claim performance for past or current students SBA Multi-Year Report = this report summarizes SBA ELA or Math assessments overall and claim performance levels over two years SBA Site or Grade Level Comparison = this report summarizes SBA ELA or Math assessments overall and claim performance levels either by grade level or school site SBA Student Roster = this report identifies students that are at each level overall and by claim performance SBA Student Summary = this report identifies ELA or Math performance levels and claim performance for a single student https://illuminate.zendesk.com/hc/en-us/sections/201815807-CA-Smarter-Balanced-Assessment-Reports Assessment Services

Tips for Accessing Reports in Illuminate Check the enrollment date in the report filter or control panel (upper right) to ensure you are pulling the correct set of students (Do you want to see results for students who tested with you or for current students?) Always clear the [Search] box Do not use too many filters Navigation for Pre-Built Reports: Reports > List Reports > Select “Prebuilts” in filters > Type “SBA” in Search box (type “Preliminary” for preliminary reports) Navigation for School’s Data File: Assessments > List Assessments > Select “State & National Publisher Assessment” in filters > Type “SBA” in Search box (type “Preliminary” for preliminary reports) Assessment Services

Reports on CDE Website Compare test results across counties, districts, schools, or the state on a single screen View results over time in addition to a single year Assessment Services See handout

Reports on CDE Website The reports on the CDE website display the mean scale score by grade level and overall for each content area The mean scale score is an important data point to consider when reviewing the results Assessment Services

Comparing Median Scale Scores from Year to Year 2800 Threshold Scale Score Range Standard Exceeded ● 2639 ● 2620 ● 2622 ● 2610 ● 2607 ● 2595 ● 2570 Standard Met ● 2558 ● 2535 ● 2555 ● 2540 ● 2541 ● 2527 ● 2530 ● 2505 ● 2494 Standard Nearly Met ● 2485 ● 2475 ● 2457 ● 2433 ● 2419 Standard Not Met Spring 2015 ● Spring 2016 ● Spring 2017 ● 2300 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 11 Assessment Services

Test Results Reported on the Individual Student Reports (IRS) Student Score Reports Test Results Reported on the Individual Student Reports (IRS) Student Score Report Versions Assessment Grades Smarter Balanced for ELA and mathematics 3, 4, 6, 7, and 11 Smarter Balanced for ELA, mathematics, and science (California Science Test [CAST] pilot) 5, 8, and 11 CAAs for ELA and mathematics CAAs for ELA, mathematics, and science (CAA for Science pilot) CAST or CAA for science (CAST pilot and CAA for Science pilot) 10 and 12 Assessment Services

Student Score Reports See handout Assessment Services

Test Results from Multiple Years Assessment Services

Low, Medium and High Bands (Divide the difference by three = 21 pts) (Divide the difference by three = 21 pts) 2367 to 2387 2388 to 2408 2409 to 2431 2432 to 2450 2451 to 2469 2470 to 2489 (Divide the difference by three = 21 pts) (Divide the difference by three = 21 pts) 2416 to 2433 2434 to 2452 2453 to 2472 2473 to 2491 2492 to 2511 2512 to 2532 See handout Assessment Services

Low, Medium and High Bands (cont.) 2015 2432 2016 2470 Assessment Services

Questions about the various reports available?

Research Recall Reflect Respond Analyzing the results

Framing the Conversation Student Performance - Data Review and Discovery What is the current state of performance? Informing Teaching and Learning Optimizing Results – Possible Actions Moving Forward What should/will change? Learning Conditions -Recalling Program, Practice, Policies, etc. What were the conditions of learning? Practices, Programs, and Policies Impacting Performance – Making Connections What may have contributed to the observed performance? Assessment Services

From Frame to Process: The Four R’s Data Review and Discovery Possible Actions Moving Forward Program, Practice, Policies Possible Connections RESOURCES See handout Assessment Services

Research Engage with the data to find facts Describe what you observe Look for data trends Focus on facts, not conjecture RESEARCH Using the data sources available, report the facts. Look for trends or areas of concern. Look for areas of success. This is an objective data discovery. Assessment Services

Research Guiding Questions: What Is the Current State of Performance? Are there any trends or patterns that emerge by grade level, by student group, by content area, and/or by claim area? What scores look noticeably different from last year’s scores? Where do the scaled scores fall within the levels (i.e, high, medium, or low band)? See handout Assessment Services

Recall Consider school/classroom programs, practices, and policies. Think about last year: What happened? What curriculum was in place? What professional learning occurred? RECALL Focusing on the prior school year(s), discuss the realities of the classroom, school, and LEA program, practice, and policies. What happened? Focus on facts only—no conjecture. Assessment Services

Recall Guiding Questions: What Were the Conditions for Learning? What curriculum/instructional materials did we initiate/continue last year? What opportunities did we provide our students and staff to interact with the assessment tools (e.g., practice test, SBAC interims) prior to the summative assessment administration? What evidence did we collect during the year that is consistent with the evidence required on the assessment? Assessment Services

Reflect Determine possible reasons for the performance Consider existing programs, practices and policies that might help to explain observed patterns, trends, or gaps in achievement Keep the conversation honest REFLECT Connect performance with prior year practices, programs, and policies. Consider possible reasons. Be honest about what occurred last year and how that may have impacted the performance that you observe. Assessment Services

Reflect Guiding Questions: What May Have Contributed to the Observed Performance? What would you consider the key factors contributing to the apparent successes/needs as indicated by the results? How do these results affirm areas where instruction was focused and increased learning was expected? Assessment Services

Respond Use guiding questions Think about what you can control What other data are available to you? How will you know progress is being made? RESPOND What might be some possible ways to move forward? These responses should tie directly to what emerged from the reflection process. Assessment Services

Respond Guiding Questions: What Should/Will Change? How might we re-structure our professional learning communities and collegial work to support the needs illustrated by the data? What evidence do we need during classroom instruction to know that our students are making progress? Assessment Services

Reminders: Analyzing Results Examining assessment data: Helps promote the effective and appropriate use of data when done correctly Helps to build a common understanding of expectations Assessment Services

Questions about Analyzing the Results?

Moving beyond the results Rightful Place and Rightful Purpose Evidence Centered Design Moving beyond the results

Evidence-Centered Design Knowing how the assessment items were developed, helps in understanding the results and making connections to the classroom ITEMS Evidence Statements Content Claims Targets Overall Claims The Smarter Balanced Hierarchy of Item Development and Reporting of Scores Assessment Services

Relationships Between CCSS, CA Frameworks, and the Assessment   English Language Arts/Literacy Content Specifications (Claims) English Language Arts/English Language Development Framework English Language Arts/Literacy Item Specifications (Claims and targets per claim) English Language Arts/Literacy Assessment Blueprint English Language Arts/Literacy Summative Assessment English Language Arts   Mathematics Content Specifications (Claims) Mathematics Framework for California Public Schools Mathematics Item Specifications (Claims and targets per claim) Mathematics Assessment Blueprint Mathematics Summative Assessment Mathematics Assessment Services

Example of Item Specifications See handout Evidence Required for Target 1 The student will identify text evidence (explicit details and/or implicit information) to support a GIVEN inference or conclusion based on the text. Item Specifications delineate the types of evidence that should be collected regarding the knowledge, skills, and /or abilities that are articulated in the standards Assessment Services

Test Blueprint Test Blueprint describes the content of the assessments, how that content will be assessed (CAT/PT), including number of items and DOK levels Key Data Systems (KDS) modified the SBAC test blueprint and aligned it to the standards Assessment Services See handout

Rightful Place/Purpose: Assessment Frequency and Impact on Instruction Statewide Summative Classroom Formative Assessment Services

….but they don’t tell the entire story Statewide summative assessments are like the tip of an iceberg—it pays to pay attention ….but they don’t tell the entire story Assessment Services

Rightful Place, Rightful Purpose for Statewide Summative Assessment Provide a general direction—we must dig deeper to determine cause Focus on groups, programs, and disaggregation Rarely provide definitive answers, but raise many questions, allowing reflection on context and practice Provide an entry point into a collaborative, honest conversation Assessment Services

Call to Action – Pay Attention but Move Beyond the Scores Focus on improving learning Not solely about increasing scores Reflect on what you can control to move beyond the scores Use with other sources of evidence Test scores should be used in conjunction with multiple pieces of evidence to arrive at a more complete understanding of student learning and progress Collaborative process that requires: Honesty Ability to handle ambiguity Patience Assessment Services

Final Thoughts Assessment Services

Resources Content Specifications, Item Specifications and Blueprints http://www.smarterbalanced.org/assessments/development/ Smarter Balanced Scale Score Ranges http://www.cde.ca.gov/ta/tg/ca/sbscalerange.asp Sample 2016-17 Student Score Report http://www.cde.ca.gov/ta/tg/ca/documents/caaspp1617ssr.pdf Claim Descriptions for ELA and Mathematics http://www.cde.ca.gov/ta/tg/ca/elaclaimdescript.asp Target Score Report FAQs http://www.caaspp.org/rsc/pdfs/CAASPP.target-report-FAQs.2016.pdf Achievement Level Descriptors http://www.smarterbalanced.org/assessments/scores/ Teacher Guides to Smarter Balanced http://www.cde.ca.gov/ta/tg/ca/sbteacherguides.asp Understanding CAASPP Results for Parents http://www.testscoreguide.org/ca Assessment Services

Assessment Services Webpage Staff Portal > Departments > Assessment Services > CAASPP> Results Assessment Services

Contact Information Erin Gordon: egordon@sandi.net 619-725-5687 Amal Morcos: amorcos@sandi.net 619-725-7061 Assessment Services main phone: 619-725-7065 Assessment Services CAASPP Web page: https://www.sandi.net/staff/assessment-services/caaspp Assessment Services