1. Accountability & Assistance Advisory Council Meeting
October 26, 2016
Best Western Royal Plaza Hotel
Marlborough, MA

2. Agenda Welcome & introductions Review of discussion norms & protocols
Summary of 2016 accountability reporting
Update & discussion on accountability & assistance under ESSA
Discussion of proposed technical amendments to MA accountability regulations
Update on MCAS 2.0
Next steps & closing
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

3. Welcome & introductions
New Council members
Sam DePina
David Krane
Jacob Waah
Massachusetts Department of Elementary and Secondary Education

4. Role of AAAC
Review & advise ESE & the BESE on the policies and practices of the office of school & district accountability & ESE’s targeted assistance & intervention efforts
Massachusetts Department of Elementary and Secondary Education

5. Norms & protocols
Promote risk-taking by not assigning specific comments to individual members in meeting summaries & minutes
Maintain confidentiality before, during, & after meetings
Seek clarification & check understanding to avoid mistakenly attributing ideas to an individual or organization represented on the council
Keep improvements in student learning at the core of the discussions. Students should drive the conversation
Stay engaged in the issues
Actively address implicit biases & instill cultural proficiency in discussions
Encourage discussion from all voices of members. Be additive, not repetitive
Let members know if pre-reading or other pre-meeting assignments require more than 30 minutes of preparation time
Make it fun!
Massachusetts Department of Elementary and Secondary Education

6. Summary of 2016 accountability reporting

7. Modifications to 2016 reporting
Accountability reporting for schools & districts administering PARCC in ELA & mathematics in grades 3-8 in 2016 only
2016 PPI calculated using standard rules
2016 accountability & assistance level held harmless: the level can stay the same or improve from 2015, but cannot decline due to performance
Exceptions: Hold harmless does not apply to high schools, MCAS-only schools or districts, schools with persistently low graduation rates, or schools or districts with low or very low assessment participation
Accountability reporting for schools & districts that administered PARCC in ELA & mathematics in grades 3-8 in 2015 & 2016
2016 cumulative PPI run twice (including & excluding 2015 annual PPIs), with the higher value used in 2016 accountability determinations
School percentiles for all non-high schools calculated twice
Including & excluding 2015 ELA & math assessment data, with the higher value used in 2016 accountability determinations
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

8. Modifications to 2016 reporting
Minimum group size for accountability determinations
Minimum group size for reporting: 20 students (aggregate & subgroups)
In 2016, this change only applies to subgroups that were large enough to receive accountability determinations in 2015
Participation rates calculated two ways
Actual 2016 rates for each group in each subject
For any group with an actual 2016 participation rate below 95 percent in any subject, a two-year participation average is calculated
The higher of the two resulting rates will be used in considering placement into an accountability & assistance level
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

9. Impact of 2016 hold harmless
117 schools were held harmless from a lower level in 2016
89 schools were held in Level 1 (17% of all Level 1 schools)
24 schools were held in Level 2 (3% of all Level 2 schools)
4 schools were held in “Insufficient Data”
30 schools would have fallen into Level 3 without the hold harmless
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

10. 2016 Commendation schools 49 Commendation schools
Subset of Level 1 schools
31 took PARCC in 2016 & 18 took MCAS
Three categories (some commended in more than one category)
High progress (27 schools)
Narrowing proficiency gaps (19 schools)
High achievement (14 schools)
Includes 3 National Blue Ribbon Schools
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

11. School levels – 2015 & 2016
1 Schools with insufficient data to be eligible for a level are schools ending in grade PK, K, 1, or 2, very small schools, & schools without four full years of data
Massachusetts Department of Elementary and Secondary Education

12. District levels – 2015 & 2016
1 Schools & single-school districts with insufficient data to be eligible for a level are schools ending in grade PK, K, 1, or 2, very small schools, & schools without four full years of data
Massachusetts Department of Elementary and Secondary Education

13. Accountability & assistance under ESSA

14. Accountability framework under ESSA
Maintains NCLB’s annual testing requirements
Requires states to incorporate a 95% assessment participation requirement into the accountability system
Requires system of “annual meaningful differentiation” for all public schools
Requires states to establish “ambitious state-designed long-term goals” & measures of interim progress, for all students & subgroups including a focus on gap closing
Massachusetts Department of Elementary and Secondary Education

15. Accountability framework under ESSA
Requires states to include the following indicators in an accountability system
Academic achievement based on annual assessments in ELA, math & science
A measure of student growth or progress for elementary & middle schools
Graduation rates for high schools
Progress in achieving English proficiency for English learners
At least one measure of school quality or student success
Massachusetts Department of Elementary and Secondary Education

16. Accountability framework under ESSA
“Substantial weight” must be given to the achievement, progress, EL proficiency & graduation rate indicators
Taken together, these indicators must be given “much greater weight” in the differentiation process than any measures of school quality or student success
Requires the identification of the lowest performing 5 percent of schools & high schools with graduation rates below 67%
Requires the identification of schools with low performing subgroups
Massachusetts Department of Elementary and Secondary Education

17. ESSA accountability transition timeline
Year
Federal context
ESEA flexibility waiver
Current accountability requirements
Transition year
Support low performing schools while reconsidering design of system
ESSA
New accountability requirements
Massachusetts Department of Elementary and Secondary Education

18. Planning for transition to 2017-18
April - July 2016
July - October 2016
October - December 2016
December March 2017
Listening
Modeling
Revising
External stakeholders
ESE staff
BESE
Massachusetts Department of Elementary and Secondary Education

19. Guiding principles in design
System focus on proficiency gap closing & preparing students for success at the next level
Measures included in the system must be actionable at the district or school level
System must provide a clear designation or signal to our highest & lowest performing schools
System must provide clear signal to schools about where improvement is needed to focus efforts
Measures are supported by research showing evidence of improved student outcomes
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

20. Framing questions
To what extent does an indicator align with ESE goals & mission?
To what extent does an indicator align with the values of internal & external stakeholders?
To what extent can the state validly & reliably measure the indicator for all schools or groups of schools?
To what extent is an indicator actionable at the district or school level?
To what extent should an indicator be included in the formal accountability system versus being publicly reported in a broader school report card?
Massachusetts Department of Elementary and Secondary Education

21. Recommendations to discuss
Make better use of levels to differentiate among schools
Currently 80% of schools are in Levels 1 & 2
About 2.5% of schools are in Levels 4 & 5
Use descriptors for Underperforming & Chronically Underperforming schools
Eliminate the use of the Composite Performance Index (CPI)
Data is lost in calculation (100 points from 240 to 280)
Move to average scaled score to show all achievement & improvement
Massachusetts Department of Elementary and Secondary Education

22. Recommendations to discuss
Move to percentile based leveling system
Measure status & improvement on all measures
Schools would be leveled based on their overall percentile or attainment of a certain level of performance
Maintain use of multiple of years of data in leveling
Build up to four years of data with new assessment
Do not mix use of MCAS, PARCC & Next-Gen MCAS in grades 3-8
Maintain consequences for low participation
Massachusetts Department of Elementary and Secondary Education

23. Recommendations to discuss
Incorporate measures to align with internal & external stakeholder values
Criteria for success
Proposed measure(s)
Grade span
All students perform well against challenging grade-level expectations
Next-generation MCAS results for English language arts, math, science
Student growth in ELA & math
All
All ELLs make progress towards English proficiency
ACCESS results
All students graduate from high school
4-year cohort graduation rate
5-year graduation plus still enrolled rate
Annual dropout rate
High school
All students attend school regularly
Chronic absenteeism
All students experience a broad & challenging curriculum
Access to the arts
Access to a well-rounded curriculum
Passing all grade 9 courses
Access to advanced coursework
Schools support all students in attaining these goals
School climate survey (under development & testing)
Massachusetts Department of Elementary and Secondary Education

24. Public feedback forums
Monday, November 14th – Boston
Monday, November 21st – Shrewsbury
Tuesday, November 29th – Brockton
Thursday, December 1st – Holyoke
Tuesday, December 6th – Salem
Any recommendations on specific feedback we should be looking for related to accountability?
Massachusetts Department of Elementary and Secondary Education

25. Input & review timeline
Month
Activity
September
BESE meeting to set context & review what we have heard from stakeholders
October &
November
Finish modeling & gather feedback from external stakeholders through multiple modes
December
BESE meeting to present proposed model incorporating feedback received from the field
January
Revise model based on feedback from BESE
February
Final review of proposed model with BESE
March
Submit ESSA state plan to U.S. Department of Education
Massachusetts Department of Elementary and Secondary Education

26. Discussion
Massachusetts Department of Elementary and Secondary Education

27. Proposed changes to accountability regulations

28. 603 CMR 2.00: Accountability & Assistance for School Districts & Schools
Governs the review of the educational programs & services provided by MA public schools & the assistance to be provided by districts & ESE to improve them
Identifies the circumstances under which a school may be declared underperforming (Level 4) & those under which a school or school district may be declared chronically underperforming (Level 5)
Describes the process for developing & implementing turnaround plans for Level 4 schools & Level 5 schools & districts
Massachusetts Department of Elementary and Secondary Education

29. Proposed changes Proposed technical amendments:
Provide clarity to underperforming schools & chronically underperforming schools & districts, & conform to existing practice with respect to Level 4 schools
Make explicit that an existing turnaround plan remains in effect until the statutory process to develop a new, modified or renewed turnaround plan is completed, or the school or district is no longer designated as underperforming or chronically underperforming
Massachusetts Department of Elementary and Secondary Education

30. Process
With BESE approval, ESE will solicit public comment on the proposed amendments
AAAC has the opportunity to review & comment on all regulations relative to accountability & assistance before board approval
After reviewing all comments & determining whether further changes are needed, the Commissioner will bring the amendments to the BESE in January 2017 for final adoption
Massachusetts Department of Elementary and Secondary Education

31. Discussion
Massachusetts Department of Elementary and Secondary Education

32. Next-Generation MCAS update

33. What does “next-generation” mean?
Computer-based
Stronger alignment to the MA Curriculum Frameworks
New item types that more deeply assess students’ level of knowledge & ability
Developing a consistent, coherent set of achievement standards (e.g., Advanced, Proficient) across grades & subjects
Sending clear signals about readiness to students, parents, & educators (readiness for the next level/grade, readiness for college/career, etc.)
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

34. Transition timeline Transition plan for computer-based testing
2017: Grades 4 & 8 ELA & math
2018: Grades 4, 5, 6, & 8 ELA, math & STE
2019: Grades 3-8 & 10 ELA & math, plus 5 & 8 STE
Next-generation tests will be a combination of legacy MCAS items & newly developed MCAS items, as well as PARCC items for ELA & math
Massachusetts Department of Elementary and Secondary Education

35. English language arts (ELA)
Read & respond to passages
Literary texts
Informational texts
Pairings of literary & informational texts
Reading comprehension items
Grades 3-8: multiple choice & technology-enhanced
Grades 3 & 4: short written response
Writing assessed at all grades
Essays of 1-2 pages in length, depending on student grade
Massachusetts Department of Elementary and Secondary Education

36. Mathematics Traditional item types New item types Two sessions
Multiple choice, short answer, & open-response questions
New item types
Multiple select, 3-point open response, fill in the blank, technology enhanced & multi-type items
Two sessions
Grades 3-6: Both sessions are non-calculator
Grades 7-8:
Session 1 = non-calculator
Session 2 = calculator allowed
Massachusetts Department of Elementary and Secondary Education

37. Science & technology/engineering (STE)
Current MCAS STE tests based on 2001/2006 standards
Grades 5 & 8: life, earth, physical, & tech/eng
High school: Biology, chemistry, intro physics & tech/eng
New STE standards adopted in 2016
Transition timeline:
Grade 5
Assess 2001/2006 standards
Assess overlapping standards (2001/2006/2016)
Assess 2016 standards 8 HS
TBD
Massachusetts Department of Elementary and Secondary Education

38. Accessibility features & accommodations
Accessibility features available to all students on both computer- & paper-based tests
Contrast & magnification tools, tracking devices, audio aids, etc.
Certain accommodations available only to students with disabilities
Paper tests, screen-readers/Braille, read-aloud, etc.
Accommodations for English language learners
Approved word-to-word dictionaries, read-aloud, scribe, test directions in student's native language (when available), etc.
Massachusetts Department of Elementary and Secondary Education

39. Still to come Standard setting for the new tests
New names for the levels of achievement
New meanings for the levels
Redesigned reports for parents & educators
Decisions about high school testing
External & internal workgroups
History & social science testing
CD vote
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

40. Next steps
ESE staff to develop a potential accountability system model for further discussion at the January 20 AAAC meeting
ESE staff to share Council comments regarding proposed technical amendments to 603 CMR 2.00 with Commissioner Chester
Massachusetts Department of Elementary and Secondary Education