1. MA Educator Preparation Mid-Atlantic Comprehensive Center Webinar April 29, 2015

2. Massachusetts Department of Elementary & Secondary Education 2 Ed Prep: MA Context  80 Sponsoring Organizations (EPPs)  IHE & Alternatives  All expectations the same  13 SOs currently seek NCATE/TEAC accreditation  Largest producers – complete 50% of candidates annually  1,800 + programs  E.g. Math 5-8 Initial, Post-Baccalaureate ; Math 8-12 Initial Baccalaureate  Approximately 6,500 program completers annually  65% employed in MA Public schools

3. Massachusetts Department of Elementary & Secondary Education 3  Regulatory Authority:  Approval determined by the Commissioner of Elementary & Secondary Education (not the Board) – tied to licensing authority  Single line of Statute granting authority  Set of High-level Regulations  Details sits in Guidelines  Separate from Dept. of Higher Education & Dept. of Early Childhood & Care  Three types of review: Informal, Formal, Interim  Formal review = every 7 years  Prep Team (5 people) sits within the Center for Educator Effectiveness (Evaluation, Recognition Programs, Licensure, Induction, Professional Development). Ed Prep: MA Context

4. Massachusetts Department of Elementary & Secondary Education 4 Moral Imperative & Driving Belief  Moral Imperative: All children in Massachusetts, especially students who need the most, must have access to effective teachers and leaders.  Driving Belief: Preparation CAN and SHOULD prepare educators to be ready on day one.

5. Massachusetts Department of Elementary & Secondary Education 5 MA Ed Prep Public Transparency Accountability Continuous Improvement

6. Massachusetts Department of Elementary & Secondary Education Reform Timeline Program Approval Regulations Revised June 2012 Two sets of pilot reviews (2009 & 2011) to aide in the development of new approval standards Draft regulations out for a period of public comment BeforeSince Moratorium on reviews for 2012-13 & 2013-14 Built out review process (rubrics, tools, guidance, etc.) Micro-pilots of new process during the 2013-14 year Full implementation of process & standards 2014-15 Increased Practicum Hours “Effective” requirement for supervising practitioners New Approval Standards New Accountability Levers

7. Massachusetts Department of Elementary & Secondary Education 7 New Program Approval Standards  Communicate a shift/Raised bar  Emphasis at organization (unit) level  Looking closely at K-12 partnerships and systems of continuous improvement  Focus on outcomes and evidence of impact  State Available Data Linkages (n=6)  Employment & Retention  Evaluation Ratings & Student Growth/Impact  Survey results (candidate, 1-year out completer, supervising practitioner & hiring employer).

8. Massachusetts Department of Elementary & Secondary Education 8 The Challenge  Building a review process that is:  Effective  Efficient  Consistently Rigorous

9. Massachusetts Department of Elementary & Secondary Education 9 Building a Strong Review Process Goal: implement a process that provides a solid evidence base for decision-making  Evidence-based decision making for:  ESE  Sponsoring Organization  Understanding Best Practices  “Informed Researcher” perspective

10. Massachusetts Department of Elementary & Secondary Education 10 Important Early Decisions  Value human judgment  Emphasize accountability at the Org Level  No longer Review Program Specific Syllabi  Summative Evaluation  ESE not the expert  Descriptive of expectations, not prescriptive of approach  Transparency/Communication key

11. Massachusetts Department of Elementary & Secondary Education 11 Review Process of Steel: Examples  Recruitment, selection & training of Reviewers  Needs Assessment for all dormant/low-enrollment programs  Eval Tool  Evaluates evidence, not criteria  Triangulates evidence – offsite, onsite and outputs  New Evidence Collection Methods  K-12 Partner survey  Use of live-polling technology during focus groups  Vetting Panel before release

12. Massachusetts Department of Elementary & Secondary Education 12 Promising Early Results  34% of programs up for review expired at point of needs assessments for 2014-2015 Formal Review (135/393)  Similar pattern observed in 2015-2016 cohort  Organizations have done things they have never done before (intentional conversations with PK12 partners, data-driven goal setting, advisory councils of recent completers)  More differentiation in terms of ratings (exemplary, proficient, needs improvement, unsatisfactory) across and within organizations than we have been able to establish previously  SOs and reviewers WANT to engage in the process

13. Questions

14. Massachusetts Department of Elementary & Secondary Education 14 Greatest Successes  Quality of relationship with our field  Recruiting PK12 educators as reviewers  Dialogue around evidence of impact  Calibration/reliability of ratings & Differentiation of results

15. Massachusetts Department of Elementary & Secondary Education 15 Ongoing Challenges/Considerations  Demand on reviewers  Evidence-driven narratives from SOs  Inclusion of data in the process  Establishing the right rewards and consequences  When/whether to provide examples or exemplars  Weighting criteria/domains  SEA Human Capital

16. Massachusetts Department of Elementary & Secondary Education 16 Lessons Learned  You do not need to set numeric benchmarks in order for outcome measures to be influential in the judgments being made  Reviewers need a mechanism and structure through which to make difficult, high-stakes decisions – otherwise they will avoid doing so.  People are going to be wary/reluctant of the unknown – no matter what you do. Creating “early allies” is the most effective way to mitigate anxiety from the field.

17. Massachusetts Department of Elementary & Secondary Education 17 Advice  Separate technical assistance from program evaluation  Embed accountability within a larger framework of reform initiatives – explicitly draw connections  Focus on continuous improvement  Build a system that your state can grow into  Walk the walk

18. Questions

20. Just IN Case Slides

21. Massachusetts Department of Elementary & Secondary Education Rubric  Eval Tool Criteria: Admission criteria and processes are rigorous such that those admitted demonstrate success in the program and during employment in licensure role. Criteria: Admission criteria for post-baccalaureate candidates verify content knowledge upon entrance to the program.

22. Massachusetts Department of Elementary & Secondary Education Eval Tool Criterion Indication of whether outputs demonstrates support for criteria (+), contrasts with (-) criteria, or is inconclusive (?) Domain List of pertinent evidence sources to be referenced by reviewers Overall decision indicating whether there is sufficient evidence in support of a criteria being met Space for review team to provide suggestions for improvement relative to the criterion Rating determination for evidence Box for reviewer to provide a rationale explaining the rating

23. Massachusetts Department of Elementary & Secondary Education 23 Review Criteria  Descriptive of expectations, not prescriptive of approach  Emphasize impact  Examples:  Admission Criteria: Admission criteria and processes are rigorous such that those admitted demonstrate success in the program and during employment in licensure role.  Diversity Criteria: Recruitment efforts yield a diverse candidate pool.  Not weighted (at this point)  Criteria not rated, evidence is

24. Massachusetts Department of Elementary & Secondary Education 24 Evaluating Evidence, Not Criteria RatingDescriptor of Evidence 4 Compelling Irrefutable evidence that criteria is being met consistently; OR, sufficient evidence that while criteria is being met throughout organization, one or more areas (i.e., programs) presents evidence above and beyond criteria 3 Sufficient Clear, convincing evidence demonstrating criteria is being met 2 Limited Evidence inconsistently supports criteria; gaps within evidence exist; evidence is weakly linked to criteria 1 Insufficient Inadequate evidence was found in support of the criteria

25. Massachusetts Department of Elementary & Secondary Education

26. 26 Worksheets  Offsite Submission  Linked to criteria  Streamlined – forcing clarity and best evidence choice  Manageable for reviewers  BIG shift

27. Massachusetts Department of Elementary & Secondary Education Required Documents Prompts Optional Context Optional Additional Documents Key Components on all Worksheets

28. Massachusetts Department of Elementary & Secondary Education 28 Prompts Linked to Criteria Criteria: Recruitment efforts yield a diverse candidate pool. Criteria: Waiver policy ensures that academic and professional standards of the licensure role are met.

29. Massachusetts Department of Elementary & Secondary Education 29 Reviewers  Conducted exploratory analysis – other states, past reviewers, current orgs  Raise prestige of the role:  Cohort Model  Intentional Selection  Market  Teacher Leadership  Build a robust training model  Full day training & ongoing calibration  Future  online modules/calibration assessments “I have worked with the state on many initiatives in my role as principal, and curriculum coordinator in a regional district. I have to say this was one of the more clear, focused and effective trainings I have been involved in.”

30. Massachusetts Department of Elementary & Secondary Education 30 Needs Assessments  Policy Context  Significant impact  12/30 new programs confirmed in 2014 Informal Cycle  34% of programs up for review expired at point of needs assessments for 2014-2015 Formal Review (135/393)  Set precedent for the review

31. Massachusetts Department of Elementary & Secondary Education 31

32. Massachusetts Department of Elementary & Secondary Education Evidence-Based Decision Making Evidence Base Findings & Commendations Exemplary Proficient Needs Improvement Unsatisfactory Approved with Conditions Approved Not Approved

33. Massachusetts Department of Elementary & Secondary Education 33 2012 Program Approval Standards  Standard A: Continuous Improvement  Standard B: Collaboration & Program Impact  Standard C: Capacity  Standard D: Subject Matter Knowledge  Standard E & F: Professional Standards  Standard G: Educator Effectiveness

34. Massachusetts Department of Elementary & Secondary Education 34 Standards & Indicators  Domains Program Approval Standards & Indicators Domain Strand  Criteria Strand  Criteria Strand  Criteria Domain Strand  Criteria Strand  Criteria