1. Accountability & Assistance Advisory Council Meeting
May 8, 2018
Best Western Royal Plaza Hotel
Marlborough, MA

2. AGENDA Welcome & introductions Review of discussion norms
Weighting of accountability indicators
Proposed amendments to state accountability regulations
Update on assistance model
Revising DESE’s District Standards & Indicators
Next Steps
AGENDA

3. Welcome & introductions01

4. Welcome & introductions
Council co-chairs: Meg Mayo-Brown & Valerie Annear

5. Review of discussion norms02

6. Norms & protocols
Promote risk-taking by not assigning specific comments to individual members in meeting summaries & minutes
Maintain respect before, during, & after meetings
Seek clarification & check understanding to avoid mistakenly attributing ideas to an individual or organization represented on the council
Keep improvements in student learning at the core of the discussions. Students should drive the conversation
Stay engaged in the issues
Actively address implicit biases & instill cultural proficiency in discussions
Encourage discussion from all voices of members. Be additive, not repetitive
Let members know if pre-reading or other pre-meeting assignments require more than 30 minutes of preparation time
Make it fun!

7. Weighting of accountability indicators03

8. System highlights Additional accountability indicators
Provide information about school performance & student opportunities beyond test scores
Normative & criterion-referenced components
Accountability percentiles & progress toward targets
Focus on raising the performance of each school's lowest performing students
In addition to the performance of the school as a whole
Discontinuation of accountability & assistance levels 1-5
Replaced with accountability categories that define the progress that schools are making & the type of support they may receive from the Department
Districts classified based on district-level data
No longer based on the performance of a districts lowest performing school

9. Massachusetts’ accountability indicators – non-high schools
Measure
Achievement
English language arts (ELA) average scaled score
Mathematics average scaled score
Science achievement (Composite Performance Index (CPI))
Student Growth
ELA mean student growth percentile (SGP)
Mathematics mean SGP
English Language
Proficiency
Progress made by students towards attaining English language proficiency (percentage of students meeting annual targets required in order to attain English proficiency in six years)
Additional Indicator(s)
Chronic absenteeism (percentage of students missing 10 percent or more of their days in membership)

10. Massachusetts’ accountability indicators – high schools
Measure
Achievement
English language arts (ELA) achievement (Composite Performance Index (CPI))
Mathematics achievement (CPI)
Science achievement (CPI)
Student Growth
ELA mean student growth percentile (SGP)
Mathematics mean SGP
High School Completion
Four-year cohort graduation rate
Extended engagement rate (five-year cohort graduation rate plus the percentage of students still enrolled)
Annual dropout rate
English Language
Proficiency
Progress made by students towards attaining English language proficiency (percentage of students meeting annual targets required in order to attain English proficiency in six years)
Additional Indicator(s)
Chronic absenteeism (percentage of students missing 10 percent or more of their days in membership)
Percentage of 11th & 12th graders completing advanced coursework (Advanced Placement, International Baccalaureate, dual enrollment courses, &/or other selected rigorous courses)

11. Weighting decisions to be made
Three weighting decisions to be made
Weighting of indicator categories in the school percentile calculation
Weighting of “meeting target” points
Weighting of “all students” vs. “lowest performing” students

12. Considerations for weighting achievement & growth
The current ratio of achievement & growth is 3 (achievement) to 1 (growth)
Impact of increasing weight of growth in system:
Could increase differentiation between similarly-achieving schools
Increases the value of a normative measure (there will always be a 1st percentile & a 99th percentile)
Decreases the value of grade 3 assessment results (no SGP for students in grade 3)
Decreases value of science assessment in system (no SGP for science)

13. Considerations for weighting achievement & growth
All indicators need to be included in the weighting
Progress towards English language proficiency only applies to a subset of schools, & weighting needs to be flexible
Ratio between achievement & growth can be held constant between non-high schools & high schools but actual weightings will differ
ESE intends to apply the same weighting rules to both the normative & criterion-referenced components of the system
Recommendation is to maintain current ratio (3:1)
Board to make final decision this spring

14. Proposed weighting of indicators in non-high schools
Measures
Current Weighting 3:1
With ELL
No ELL
Achievement
ELA, math, & science achievement values (based on scaled score)
60%
67%
Student Growth
ELA & Math Student Growth Percentile (SGP)
20%
23%
English Language
Proficiency
Progress made by students towards attaining English language proficiency
10%
Additional Indicators
Chronic absenteeism

15. Proposed weighting of indicators in high schools & middle/high/K-12 schools
Measures
Current Weighting 3:1
With ELL
No ELL
Achievement
ELA, math, & science achievement
40%
48%
Student Growth
ELA & Math Student Growth Percentile (SGP)
20%
22%
High School Completion
Four-year cohort graduation rate
Extended engagement rate
Annual dropout rate
English Language
Proficiency
Progress made by students towards attaining English language proficiency
10%
Additional Indicators
Chronic absenteeism
Percentage of students completing advanced coursework

16. Compares schools administering similar statewide assessments
Normative component
Accountability percentile 1-99, calculated using all available indicators for a school
Compares schools administering similar statewide assessments
Used to identify the lowest performing schools in the state
Same calculation used at the subgroup level to identify low- performing subgroups

17. Criterion-referenced component
Focus on closing the achievement gap by raising the “achievement floor”
Gap-closing can occur as a result of a decline in performance by the high- performing group
In addition to meeting targets for the school as a whole, the performance of the lowest performing students in each school will be measured
Every school has a group of lowest performers
Identified from cohort of students who were enrolled in the school for more than one year

18. Criterion-referenced component
Targets set for each accountability indicator, for the school as a whole & for the lowest performing students in each school
Indicator
Non-high schools
High schools & middle/high/K-12 schools
All students
Lowest performing students
ELA scaled score ✔
Math scaled score
Science achievement
ELA SGP
Math SGP
Four-year cohort graduation rate
N/A
Extended engagement rate
Annual dropout rate
EL progress
Chronic absenteeism
Advanced coursework completion
*Minimum group size for each indicator = 20 students

19. Criterion-referenced component
Declined
No change
Improved
Met target
Exceeded target 1 2 3 4
Points assigned based on progress toward target for each indicator, for both the aggregate & the lowest performing students
Proposed model has equal intervals between categories of performance against meeting targets
Recommendation is to maintain these intervals until multiple years of data allow for impact analysis of unequal weighting of points

20. Criterion-referenced component calculation – non-high school
Indicator
All students (50%)
Lowest performing students (50%)
Points
earned
Total possible points
Weight
ELA scaled score 3 4 - 2
Math scaled score
Science achievement
Achievement total 7 12
60% 8
67%
ELA SGP
Math SGP
Growth total
20%
23%
EL progress
10%
Chronic absenteeism
Weighted total
6.1
9.6 - 
4.9
7.6  -
Percentage of possible points
63.5%
64.7%
Criterion-referenced target percentage
64%

21. Criterion-referenced component calculation – high school
Indicator
All students (50%)
Lowest performing students (50%)
Points
earned
Total possible points
Weight
ELA achievement 3 4 - 2
Math achievement
Science achievement 1
Achievement total 7 12
40% 5
67%
ELA SGP
Math SGP
Growth total 8
20%
23%
Four-year cohort graduation rate
Extended engagement rate
Annual dropout rate
High school completion total 10
EL progress
10%
Chronic absenteeism
Advanced coursework completion
Additional indicators total 6
Weighted total
7.0
10.0 - 
5.6
10.3
Percentage of possible points
70.0%
54.4%
Criterion-referenced target percentage
62%

22. Categorization of schools
Schools will no longer be placed in a vertical hierarchy of levels 1-5
Number of schools that will be placed into a category based upon a relative standing will be cut in half from previous system
Approximately 90 percent of schools could be categorized based on their own performance against targets
Most schools will have 50 percent of its categorization based on students that have been in the school for at least two years
Category labels are primarily tied to the level of required assistance or intervention
Stronger emphasis on schools commended for success

23. Categorization of schools
Schools without required assistance or intervention
(approx. 85%)
Schools requiring assistance or intervention (approx. 15%)
Schools of recognition
Schools demonstrating high achievement, significant improvement, or high growth
Meeting
targets
Criterion-referenced target percentage
75-100
Partially meeting targets
50-74
Not meeting targets
0-49
Focused/targeted support
Non-comprehensive support schools with percentiles 1-10
Schools with low graduation rate
Schools with low performing subgroups
Schools with low participation
Broad/
comprehensive support
Underperforming schools
Chronically underperforming schools
Notes:
Category names not finalized
School percentiles & performance against targets will be reported for all schools

24. Categorization of districts
Districts will be classified based on the performance of the district as a whole
No longer categorized based on performance of lowest performing school
District accountability percentiles will not be calculated
Classified based on criterion-referenced component
Adjustments made for low graduation rates & low assessment participation
Board may designate a district as underperforming or chronically underperforming

25. Categorization of districts
Districts without required assistance or intervention
Districts requiring assistance or intervention
Meeting
targets
Criterion-referenced target percentage
75-100
Partially meeting targets
50-74
Not meeting
0-49
Focused/targeted support
Districts with low graduation rate
Districts with low participation
Broad/
comprehensive support
Underperforming districts
Chronically underperforming districts
Notes:
Category names not finalized
Performance against targets will be reported for all districts

26. Accountability reports
Accountability reports published for each district & school (fall 2018)
Reports will include:
Overall classification
Including reason(s) for classification (e.g., low graduation rate, low-performing subgroup)
Criterion-referenced target percentage
Accountability percentile (schools only)
Data related to performance on each accountability indicator for each subgroup meeting the minimum group size (20 students)
All students
Lowest-performing students
High needs students
English learners
Students with disabilities
Economically disadvantaged students
Major racial/ethnic subgroups

27. Proposed amendments to state accountability regulations04

28. State accountability regulations
603 CMR 2.00 – Accountability & Assistance for School Districts & Schools
Current regulations describe the process & measures used to place schools & districts into accountability & assistance levels 1-5
Do not align with Massachusetts’ federally approved Every Students Succeeds Act (ESSA) state plan
Do no reflect changes in how we report MCAS data
Next-Generation MCAS achievement results reported using a different measure than the legacy MCAS tests

29. State accountability regulations
Proposed amendments align the regulations to our approved ESSA state plan
Describe the general framework of the accountability system & the assistance provided to schools & districts identified as needing support from the Department
Clarify the Board of Elementary & Secondary Education’s role in reviewing & approving the system

30. ESE Turnaround Assistance Redesign: New Opportunities 2018 & beyond05

31. Focusing on L3, L4, & L5 districts with 42% of the state’s students
Current Model
Focusing on L3, L4, & L5 districts with 42% of the state’s students

32. Redesigned Regional Support Structure Features
Vision: All students in the Commonwealth, especially the historically marginalized, learn in dynamic and responsive educational environments, graduate with the tools and agency to achieve their visions for success, and contribute as responsible members of their communities.
West/Central Region
Co-Directors of Regional Assistance
Joan Tuttle
Donna Harlan
Worcester Fitchburg
Chicopee Pittsfield
North Adams Springfield
Gill Montague Winchendon
Webster Adams Cheshire
Monson Athol Royalston
Gardner
Coastal Region
Co-Directors of Regional Assistance
Mary Ann Jackman
Susan Berglund
Boston Taunton
Chelsea New Bedford
Lynn Brockton
Lowell Fall River
Haverhill Framingham
Randolph
Strategic Transformation Region
Director of Strategic Transformation
Lauren Woo
Lawrence
Holyoke
Southbridge
Level 5 schools
Springfield Empowerment Zone Partnership (SEZP)

33. Redesigned Regional Support Structure Staffing
Support, facilitate, and manage execution of work plans developed by Assistance Leads and their districts
Manage, coach, and support Assistance Leads, Data Specialist, Content Specialists, and other regional staff
Coordinate teaming structures for each region, including Networks
Identify and advocate for additional resources needed to meet the needs of their districts and schools
Co-Directors of Regional Assistance
Possess diverse skills and experience
Support schools and districts in developing, monitoring, and revising turnaround plans
With Co-Directors of Regional Assistance, identify the TAP implementation assistance priorities (work plan)
Collaborate with other Assistance Leads to provide support to schools and districts
Assistance Leads
Support Co-Directors of Regional Assistance and Assistance Leads in using data to make decisions about assistance priorities and plans for each region.
Work with Assistance Leads to help districts assess their data needs and develop ongoing capacity to create effective turnaround plans, monitor implementation, and support a culture of data use.
Data Specialists
Content Specialists
Additional Capacity for Regional Assistance (driven by school and district needs)
- Support Contracts: Additional support, as needed, around math, literacy, ELL, SWD, SEL, and other areas.
Provide content support, in partnership with assistance leads, to schools and districts around math, literacy, ELL, SWD, SEL, and educator effectiveness.
Support regional teams in vetting and selecting content-focused partners to support schools and districts.
- Content Specialists: Math, Literacy, STE, ELL, Ed Effectiveness, SWD
- Not necessarily “support” content at schools, but more higher level decision making.

34. Discussion Question
How should the redesign Statewide System of Success measure its impact? What outcomes should we be paying attention to as we measure our effectiveness? 

35. Revising DESE’s District Standards & Indicators06

36. Overview of DESE’s District Standards & Indicators Revision process
Discussion
CONTENTS

37. Overview: District Standards & Indicators

38. What they are and how they are used
DESE’s definition of effective district policy and practice
Incorporate DESE’s expectations for schools
Framework for district reviews
Guidance for district reflection and planning

39. Revision process

40. Same 6 standards with revised/updated indicators Goals for revision:
Guiding principles
Same 6 standards with revised/updated indicators
Goals for revision:
High-leverage
Explicit focus on equity
Aligned with current ESE priorities/expectations
Realistic/practical
Succinct
Coherent/consistent
Flexibility as to extent of revisions

41. To date: Going forward: Gathering feedback
Initial revisions & iterative feedback
Input from across & beyond DESE (standard-specific)
Going forward:
Broader review of complete document
Research focus

42. Discussion

43. For your consideration…
What are your recommendations for improving/clarifying the Standards & Indicators?
What are some ways that the Standards & Indicators (and/or supporting documents) might be useful for districts?
How can we better promote districts’ use of the Standards & Indicators?

44. Next steps 07