2019 MCAS Reporting Call for Principals and Test Coordinators September 20, 2019 Embargoed until 12:01 a.m., September 24

Logistics for This Session Questions may be asked at any time using the Questions feature on the GoToWebinar screen. Questions about a specific student should be sent by email to mcas@doe.mass.edu. Some questions may be covered during the course of the presentation. We will answer some questions at the end of this session. After the session, we will email questions and answers to participants. This session is being recorded and will be available online in about one week at the MCAS Resource Center.

Michol Stapel, Associate Commissioner for Student Assessment Presenters Michol Stapel, Associate Commissioner for Student Assessment Bob Lee, MCAS Chief Analyst Scott Kelley, Data and Reporting Specialist David Ragsdale, Test Security Specialist Massachusetts Department of Elementary and Secondary Education

Today’s Call MCAS reporting highlights Statewide results for all grades and subjects Transition to next-generation grade 10 tests Release schedule and additional resources

2019 MCAS Reporting Highlights Overall gains across grades 3–8 between 2017 and 2019 Grade 3 ELA: Meeting Expectations rose from 52% in 2018 to 56% in 2019 Essay scores rose by almost a point per essay Grade 6 Mathematics: Meeting Expectations rose from 47% in 2018 to 51% in 2019 Open response scores up 7% First math test in grades 3–8 to surpass an average scaled score of 500 and 50% Meeting Expectations

2019 MCAS Reporting Highlights (continued) Grade 10 ELA and Mathematics First year of next-generation tests with new scaled scores & achievement levels Graduation and scholarship eligibility standards and rules remain the same as the legacy standards for at least the classes of 2021 and 2022 87% of students passed the MCAS testing requirement in all three subjects on the first attempt (same as last year) Transition to computer-based testing nearly complete Only high school STE and retests were paper-based in 2019

2019 MCAS Participation Rates by Subject and Mode Grade Participation Rates ELA Math STE Grade 3 99.4% 99.7% - Grade 4 99.5% 99.6% Grade 5 Grade 6 99.3% Grade 7 99.2% Grade 8 99.0% 99.1% 98.8% Grades 3-8 Grade 10 98.4% 98.2% Grade Computer Based Testing Grade 3 97.3% Grade 4 Grade 5 97.6% Grade 6 96.7% Grade 7 97.9% Grade 8 97.8% Grades 3-8 97.4% Grade 10 (ELA & math only)

English Language Arts Achievement Results 2019 ELA by Grade 2017–2019 Change Grade Average Scaled Scores Scaled Score Change, 2017 to 2019 2017 2018 2019 Grade 3 498.8 502.2 504.1 +5.3 Grade 4 499.2 501.8 +2.6 Grade 5 498.9 501.9 501.2 +2.3 Grade 6 499.4 501.0 501.3 +1.9 Grade 7 499.1 497.0 499.0 -0.1 Grade 8 499.8 +1.0 Grades 3-8 500.5 +2.2 Grade 10 506.2

Mathematics Achievement Results 2019 Mathematics by Grade 2017–2019 Change Grade Average Scaled Scores Scaled Score Change, 2017 to 2019 2017 2018 2019 Grade 3 498.8 499.9 499.4 +0.6 Grade 4 498.0 497.9 499.2 +1.2 Grade 5 498.6 497.5 498.5 -0.1 Grade 6 500.8 +1.6 Grade 7 498.2 -0.4 Grade 8 499.6 499.0 -0.6 Grades 3-8 498.4 +0.4 Grade 10 505.1

Science and Technology/Engineering May 2018 Science and Technology/Engineering 2019 Next-Generation Gr. 5 & 8 Legacy Grade 10 Grade 5 Grade 8 Exceeding Expectations 8 48% 46% Meeting Expectations 40 38 Partially Meeting 39 41 Not Meeting Expectations 12 13 Avg. Scaled Score 498.9 498.2

Tips for reporting next-generation results at the group level Use average scaled scores to compare groups It is still appropriate to use percentages (such as percent Meeting or Exceeding Expectations), but average scaled scores are more precise when comparing groups or change over time. Results can be averaged across grades 3-8, as shown on slides 7 and 8 Grade 10 results should not be averaged with lower grades. For the state release, ESE defined improving as gaining 2.0 or more scaled score points, or more than 2% gains. Statistical significance varies depending on school size. Student growth percentiles can be averaged across all grades

2017 to 2019 School Level Scaled Score Changes in Grades 3-8 English Language Arts Mathematics

Grades 3-8 Results by Race, 2017-2019

Changes in Scaled Scores by Race, 2017–2019

Grade 3 ELA Results – Essay Scores Results by district District results by poverty rate 99% of districts saw a reduction in zero-scored essays The number of students receiving zeroes on one of their two essays dropped from 40,400 (58%) to 16,800 (25%) Similar gains were seen in high poverty schools and low poverty schools

Transition to Next-Generation Grade 10 ELA and Math Tests First administration of next-generation tests in 2019 Based on Massachusetts 2017 revised curriculum frameworks Computer-based Aligned to NAEP and evidence-based definitions of college & career readiness Consistent with results established in 2017 for Grades 3-8 Legacy graduation and scholarship standards remain the same while students become familiar with the new tests The Department used statistical techniques validated by expert educators to ensure that the knowledge, skills and abilities needed to pass and graduate with a Competency Determination remained similar for at least the next two classes.

Legacy Achievement Level Results (2015)

Grade 10 Transition: Legacy Compared to Grades 3–8 (2018) May 2018 Grade 10 Transition: Legacy Compared to Grades 3–8 (2018) ELA Mathematics Next-Generation Legacy Next-Generation Legacy

% to College by ELA Grade 10 % to College by Math Grade 10 As most Proficient and many Needs Improvement students have attended college… % to College by ELA Grade 10 % to College by Math Grade 10

% Graduating College by ELA % Graduating College by Math . . . most of the Proficient and very few Needs Improvement students graduated from college within eight years. % Graduating College by ELA % Graduating College by Math Meeting Expectations Meeting Expectations Proficient on Legacy Proficient on Legacy

Standard Setting Needed anytime you introduce a new test Establish standards for each achievement level; without standard setting, can’t say whether a student met expectations, only whether they answered an individual question right or wrong Context for standard setting Legacy MCAS needed to be updated to reflect expectations in revised learning standards, as well as for alignment with grades 3-8 tests Legacy MCAS high school tests were designed to measure readiness to graduate, not readiness for college and career Approximately 25% of MA public high school graduates who enrolled in public colleges or universities placed into remedial courses for Mathematics, 10% for reading and writing In Mathematics 90% of students at 220 required remediation and 50% of students at 250

Standard Setting for Grades 5 & 8 STE and Grade 10 ELA & Math August 5-7, 2019 75 educators (with an average of 18 years experience) Four panels Grade 10 ELA Grade 10 Mathematics Grade 5 Science and technology/engineering (STE) Grade 8 STE Massachusetts Department of Elementary and Secondary Education

Key Standard Setting Activities Massachusetts educators worked in grade- and subject-specific groups, lead by facilitators, to make judgments that produced cut score recommendations by identifying the knowledge, skills, and abilities that students need to demonstrate to reach each achievement level (Grade 10 only) Educators examined student results on the legacy and next-generation tests to verify the interim standard

Next-Generation MCAS Achievement Levels Exceeding Expectations A student who performed at this level exceeded grade-level expectations by demonstrating mastery of the subject matter. Meeting Expectations A student who performed at this level met grade-level expectations and is academically on track to succeed in the current grade in this subject. Partially Meeting Expectations A student who performed at this level partially met grade-level expectations in this subject. The school, in consultation with the student's parent/guardian, should consider whether the student needs additional academic assistance to succeed in this subject. Not Meeting Expectations A student who performed at this level did not meet grade-level expectations in this subject. The school, in consultation with the student's parent/guardian, should determine the coordinated academic assistance and/or additional instruction the student needs to succeed in this subject.

Achievement Levels and Cut Scores Not Meeting Expectations Partially Meeting Expectations Meeting Expectations Exceeding Expectations Achievement Partially Meeting Expectations Cut Score Meeting Expectations Cut Score Exceeding Expectations Cut Score Less More

Educator Validation of Equipercentile Linking For each cut score, educators compared: Educators verified that the knowledge, skills, and abilities defined by each of these sets of results were equivalent for each cut score. Student results on a set of items from the 2018 legacy test that is associated with that legacy cut score (located just below, at, or just above the cut score) Student results on a set of items from the 2019 next-generation test that is associated with the statistically defined interim cut score (located just below, at, or just above the cut score) to

Linking Model for Locating Cuts (ELA example) 13% Exceeding Expectations 48% Meeting Expectations % of students 31% Partially Meeting Expectations 8% Not Meeting Expectations

Statistical Linking Model (ELA) 13% Exceeding Expectations 48% Meeting Expectations % of students 31% Partially Meeting Expectations 8% Not Meeting Expectations

Zooming In on the Pass/Fail Line (ELA) 472 Partially Meeting Expectations 240 470 EPP required % of students 5% Passing & Not Meeting Expectations 455 220 3% Not Passing & Not Meeting Expectations 200 440

Interim CD Standard for the Classes of 2021 and 2022 Legacy PASSING but requires an Educational Proficiency Plan (EPP) in ELA/math only Legacy PASSING and met the MCAS graduation requirement Next Gen PASSING EPP Next Gen PASSING and met the MCAS ELA 220–238 240+ 455–471 472+ Mathematics 469–485 486+ STE 220 220+ TBD - Summer 2020

Interim Adams and Koplik Cut Scores for the Classes of 2021 and 2022 Legacy Proficient Legacy Advanced Next-Gen Proficient Equivalent Next-Gen Advanced Equivalent ELA 240-258 260-280 472-500 501-560 Math 486-503 504-560 STE N/A

MCAS Passing Rates Remain Virtually Identical to Previous Years   Subgroup Class of 2021 Class of 2020a Class of 2019a ELA Math ELA and Math STE All Three Tests All Students 95 90 89 93 87 88 Gender    Female 96 92 91 94    Male 85 86    Non-Binary N/A Race/Ethnicity    African American/Black 82 80 77 78    Asian 97    Native Hawaiian or Pacific Islander 84    Hispanic or Latino 83 73 72 71    Multi-Race, Non-Hispanic or Latino    Amer. Ind. or Alaskan Nat.    White Student Status    High Needs 74    Non-Disabled    Students with Disabilities 69 67 63 64    English Learner (EL) 68 61 49 62 43 44 40    Former EL 98    EL and Former EL 79 66 59 54    Economically Disadvantaged 75 aTo provide comparable data, results for the Classes of 2020 and 2019 are based on MCAS tests through the spring 2018 and spring 2017 administrations, respectively. Number and percentage of students earning a full CD after their first grade 10 attempt CD Requirement Number Class of 2021 after grade 10 Class of 2020 after grade 10 Earned CD 54,533 74% ELA and Mathematics Proficient or Higher 54,781 75% ELA Proficient or Higher 65,179 89% 88% Mathematics Proficient or Higher 55,664 76% STE Needs Improvement or Higher 67,787 93%

Available Resources Student Work Samples Samples of student work for each score point for ELA, math and STE; ELA essays include annotations for each score point. MCAS Sample Student Work & Scoring www.doe.mass.edu/mcas/student/2019/ Student Essay Responses One essay at each grade in grades 4–8; one constructed response item in grade 3; both essays released at grade 10 PearsonAccess Next mcas.pearsonaccessnext.com Released Items Approximately 50% of items released at grades 3–8; 100% released at grade 10 ELA and math MCAS Resource Center mcas.pearsonsupport.com/released-items/ Edwin Analytics Official embargoed MCAS interactive reports available on Sept. 19; student Competency Determination status available in PE618 Student CD Roster DESE Security Portal gateway.edu.state.ma.us District Analysis and Report Tools (DART) Success after high school report Analysis of school and district postsecondary outcomes. See what proportion of the students from your school attended college, required remediation and graduated. DESE Data Tools http://www.doe.mass.edu/dart/dart-success-after-high-school.xlsx

Upcoming Release Dates (Tentative) September 18 Communication about special considerations due to the voided essay prompt and the John and Abigail Adams Scholarship released September 19 Official embargoed school and district results available in Edwin Analytics September 23 Embargoed data given to members of the press and superintendents. Messaging bullet points released via On the Desktop communication. September 24 Public release of official MCAS results on Profiles September 30 District offices receive printed Parent/Guardian Reports Week of Sept. 30 Parent/Guardian 5-minute video about Grade 10 reports to be released alongside FAQs, templates and translations available http://www.doe.mass.edu/mcas/parents/

(Use the “Questions” feature to ask questions.) Questions & Answers (Use the “Questions” feature to ask questions.)

Today: Complete the evaluation form. Next Steps Today: Complete the evaluation form. Responses associated with the name and email address used to log in Email your input to mcas@doe.mass.edu if you have problems accessing or completing the form. Later this week/early next week: Receive an email with the Q&A from this session Recording will be available Massachusetts Department of Elementary and Secondary Education

The Office of Student Assessment Services THANK YOU The Office of Student Assessment Services 781-338-3625 mcas@doe.mass.edu www.doe.mass.edu/mcas 75 Pleasant Street, Malden, MA 02148