Adult and Community Learning Services (ACLS) Program Quality Reviews (PQR) FY2019 – FY2022 Community Adult Learning Centers and Correctional Institutions Adult and Community Learning Services (ACLS)

What Adult Education Program Directors Can Expect Before, During and After a Program Quality Review

1 2 CONTENTS 3 4 Overview of Program Quality Reviews How to Prepare for a Program Quality Review CONTENTS 3 What to Expect During a Program Quality Review 4 What to Expect After a Program Quality Review

Overview of Program Quality Reviews 01

Purpose High-quality programs Diagnosing areas for improvement Rating against a set of Indicators Identifying promising practices

PQR Cohorts All programs fully reviewed during a four-year funding cycle. Cohorts organized by performance and local areas: Year 1 (FY19)-Cohort 1 Year 2 (FY20)-Cohort 2 Year 3 (FY21)-Cohort 3 PQR Master Schedule released in the fall of each year.

Program Quality Review Schedule Examples Year 1: Program Quality Review Year 2: Site Visit Year 3: Site Visit Program A (if in cohort 1) Year 1: Site Visit Year 2: Program Quality Review Program B (if in cohort 2) Year 3: Program Quality Review Program C (if in cohort 3)

PQR Team ACLS Review Team Lead ACLS Program Quality Team Member (a Program Quality Team member) Local Workforce Development Board Representative ACLS Program Quality Team Member Adult Education Director (from another local workforce area) ACLS Curriculum Specialist ACLS Program Specialist (if not a member of the Program Quality Team) Program Quality Review Consultant

PQR Team-cont. Depending on the size of the program, a Review Team may be comprised of anywhere from two to seven members. Each Review Team will have two members of the Program Quality (PQ)Team The Local Workforce Development Board representative is always invited, but depending on schedules may or may not be part of the team.

Indicators of Program Quality The following Indicators of Program Quality (IPQ) will be the focus: IPQ 3-Career Pathways Collaboration IPQ 4-Curriculum and Instruction IPQ 7-Organizational Capacity IPQ 8-Program Leadership Indicator 5 (Student Performance) data is reviewed by the team before the visit and may be included in interviews depending on performance-related concerns. Indicators will be rated using IPQ-specific rubrics

Ratings Ratings will: Be assigned at the standard and indicator level; Be based on solid evidence; Be used as corroborative evidence in future funding decisions; Capture the program’s performance at the time of the visit; Inform the ACLS policy infraction protocol (PIP); Inform the site visit TA process; Inform the program’s continuous improvement process.

Ratings-cont. Limited evidence: The program demonstrates little to no evidence related to the Indicator and/or significant concerns are noted. Developing: The program demonstrates inconsistent evidence related to the Indicator and/or moderate concerns are noted. Proficient: The program demonstrates consistent evidence related to most standards of the Indicator and/or minor concerns are noted. Exemplary: The program demonstrates consistent evidence related to all standards of the Indicator and is a potential exemplar in this area.

Classroom Observations Data Collection Classroom Observations Classroom observations do not exceed 30 minutes each and are conducted in such a way as to not disrupt classroom activities. The Team will visit several classes and use a tool and rubric to gather evidence on activities and practices related to four essential elements of high-quality instruction: scaffolding, differentiated instruction, checking for understanding, and student engagement. Interview Questions Document Review For some of the Indicators, ACLS requires programs to have specific documents prepared for the day of the visit, prior to the team’s arrival. During the visit, the Team will interview program leadership, a sampling of teachers, advisors, and students. No written response to the interview questions is expected.

Interviews Each interview must be held in a secure, quiet location. More than one interview may be scheduled at the same time during the visit. Each participant must be present for the entirety of the interview and is given complete freedom to speak. Attendees should not attend multiple group interviews. In general, supervisors and supervisees should not be in the same group interview.

Interviews-cont. Program Leadership Teachers Advisors Students .5 hour Must represent all levels and content areas services by the program Depending on program size, the group may vary in size. Final number to be decided during preparation phone call. 1. 5 hour Must include staff who oversee the AE program directly and are responsible for its daily management (program director, assistant director, lead teacher). Must include program’s director’s supervisor. Advisors Students .5 hour Should include maximum 10 students Must represent all levels and content areas serviced by the program Program staff must not be in attendance. 1 hour Must represent advisors who work with students at all levels.

The team will ask questions related to NRS follow-up. Interview Questions Interview questions will relate to the 4 Indicators reviewed during the visit. The team may ask questions related to Indicator 5 (Student Performance) depending on concerns related to MSG performance. The team will ask questions related to NRS follow-up.

Indicator of Program Quality Document Review Indicator of Program Quality Documents Requested Page Limit When IPQ 4-Curriculum and Instruction Lesson plans Corresponding scope and sequences and curriculum units for the observed classes. Lesson plan-2 pages Corresponding scope and sequences and curriculum units-3 pages each Lesson plan-the morning of the visit Scope and Sequences and curriculum units-2 weeks before the visit IPQ 7-Organizational Support   Resumes and job descriptions for the program’s leadership Resume and job description program leaders and supervisor- 2 pages each 2 weeks before the visit IPQ 8-Educational Leadership Three copies of actual feedback director provided to teachers including but not limited to supervision, classroom observations, and annual evaluation reviews. N/A

Structure of the Day – Sample Schedule #1 Sample Schedule for a smaller program, 2-3 member team. (*Note: Start and end times of activities are subject to change.) Activity Activity Length Schedule Staff Involved Indicators of Program Quality Focus Team arrival and set-up 15 min 8:30-8:45 PQR Team N/A Facility Tour 8:45-9:00 PQR Team Members & Program Director 7 Interview with program leadership 90 min 9:00-10:30 PQR Team Member (s) & Program Leadership 3, 4, 7, 8 Classroom Observations 60 min 10:30-11:30 PQR Team Members & Teaching Staff 4 Student Focus Group Interview 30 min 11:30-12:00 PQR Team Members & Students 3, 4, 7 ,8 Teacher Focus Group Interview 12:00-1:00 Lunch & Team Check-in 1:00-1:30 Advisor Focus Group Interview 1:30-2:30 PQR Team Members & Advisors 3, 7 , 8 Team Work Time (aggregate classroom observations data, sort evidence, build preliminary finding, prepare report-out) 105 min 2:30-4:15 Report Out 4:15-4:30 PQR Team and Program Leadership Staff

Structure of the Day – Sample Schedule #2 Sample Schedule for a larger, multi-site program, 6-7 member team (*Note: Start and end times of activities are subject to change.) Schedule Team members 1, 2, 3 Team members 4 , 5, 6, 7 8:30-8:45 Team arrival and set-up 8:45-9:00 Facility Tour 9:00-10:30 Interview with program leadership 10:30-11:30 Classroom Observations Advisor Focus Group Interview 11:30-12:00 Student Focus Group Interview Classroom observations 12:00-12:30 Lunch and Team check-in 12:30-1:30 Teacher Focus Group Interview 1:30-4:15 Team Work Time (aggregate classroom observations data, sort evidence, build preliminary finding, prepare report-out) 4:15-4:30 Report-out

Logistics The program director should inform the team of: Parking logistics Food allergy restrictions in the building Meeting place in the morning Debrief room Interview room Team members cannot accept or solicit any meals. Program director needs to be available throughout the day for any clarifying questions.

End of Day Report-Out Program leadership and their supervisor are required to attend. The team will only share a set of preliminary findings, most likely on one of the four Indicators only. The team will not provide ratings or extensive feedback at this time.

How to Prepare for a Program Quality Review 02

Before the Program Quality Review Director’s Role Serve as the main point of contact for the PQR. Work with the Review Team leader to confirm the visit date. Call or email the Review Team leader if there are any sudden changes in the program schedule. Work with the Review Team Leader and program staff to determine the agenda for the day. Reserve a secure, quiet meeting room space for the Review Team to conduct interviews and debrief throughout the day. Ensures lesson plans and lesson materials are made available in the Review Team’s meeting room before the team arrives. Gather and organize all required documents for the Review Team’s document review. Email required documents to the Review Team leader two weeks in advance of the visit.

Before the Program Quality Review Prepare Staff Review PQR agenda and schedule with program staff participating in the review. Review Indicators of Program Quality 3, 4, 5, 7 and 8 with program staff participating in the review. Ensure that program staff understand the purpose of the program quality review and what will be expected of them. Ensure program staff know the Program Quality Review Protocol. Identify and prepare a group of 10 students representative of all program’s levels for the student interview.

Useful Documents Program Leadership and staff participating in the PQR should review the following documents in preparation for the visit: Program Quality Review Protocol Indicators of Program Quality ACLS Performance Standards

What to Expect During a Program Quality Review 03

During the Program Quality Review Director’s Role Act as the main point of communication between the program and the PQR team. Ensure that copies of any lesson plans and lesson materials are made in advance for the scheduled classroom observations. Be available to meet with the team during schedule times and throughout the day for any additional questions or follow-up. Bring any questions or concerns to the Review Team leader Attend the report-out at the end of the day.

What to Expect After a Program Quality Review 05

Written Report Director’s Role The Review Team Leader will send a report to the Program Director within three weeks of the visit. The Program Director should: Work with relevant program staff, agency/organization leadership to review the report. Share the report with program staff and other key stakeholders (e.g. Board of Directors, Superintendent, Dean, CEO). If applicable, respond to the report within 2 weeks after receipt. Develop an improvement plan if needed. Works with their assigned program specialist and/or SABES to receive further TA on the Indicators rated low.

Adult and Community Learning Services (ACLS) THANK YOU Adult and Community Learning Services (ACLS) 781.338.3850 http://www.doe.mass.edu/acls/ 75 Pleasant Street, Malden, MA 02148