1. Adult and Community Learning Services (ACLS)
Program Quality Reviews (PQR) FY2019 – FY2022 Community Adult Learning Centers and Correctional Institutions
Adult and Community Learning Services (ACLS)

2. What Adult Education Program Directors Can Expect Before, During and After a Program Quality Review

3. 1 2 CONTENTS 3 4 Overview of Program Quality Reviews
How to Prepare for a Program Quality Review
CONTENTS 3
What to Expect During a Program Quality Review 4
What to Expect After a Program Quality Review

4. Overview of Program Quality Reviews01

5. Purpose High-quality programs Diagnosing areas for improvement
Rating against a set of Indicators
Identifying promising practices

6. PQR Cohorts
All programs fully reviewed during a four-year funding cycle.
Cohorts organized by performance and local areas:
Year 1 (FY19)-Cohort 1
Year 2 (FY20)-Cohort 2
Year 3 (FY21)-Cohort 3
PQR Master Schedule released in the fall of each year.

7. Program Quality Review Schedule Examples
Year 1: Program Quality Review
Year 2: Site Visit
Year 3: Site Visit
Program A
(if in cohort 1)
Year 1: Site Visit
Year 2: Program Quality Review
Program B
(if in cohort 2)
Year 3: Program Quality Review
Program C
(if in cohort 3)

8. PQR Team ACLS Review Team Lead ACLS Program Quality Team Member
(a Program Quality Team member)
Local Workforce Development Board Representative
ACLS Program Quality Team Member
Adult Education Director
(from another local workforce area)
ACLS Curriculum Specialist
ACLS Program Specialist
(if not a member of the Program Quality Team)
Program Quality Review Consultant

9. PQR Team-cont.
Depending on the size of the program, a Review Team may be comprised of anywhere from two to seven members.
Each Review Team will have two members of the Program Quality (PQ)Team
The Local Workforce Development Board representative is always invited, but depending on schedules may or may not be part of the team.

10. Indicators of Program Quality
The following Indicators of Program Quality (IPQ) will be the focus:
IPQ 3-Career Pathways Collaboration
IPQ 4-Curriculum and Instruction
IPQ 7-Organizational Capacity
IPQ 8-Program Leadership
Indicator 5 (Student Performance) data is reviewed by the team before the visit and may be included in interviews depending on performance-related concerns.
Indicators will be rated using IPQ-specific rubrics

11. Ratings Ratings will: Be assigned at the standard and indicator level;
Be based on solid evidence;
Be used as corroborative evidence in future funding decisions;
Capture the program’s performance at the time of the visit;
Inform the ACLS policy infraction protocol (PIP);
Inform the site visit TA process;
Inform the program’s continuous improvement process.

12. Ratings-cont.
Limited evidence: The program demonstrates little to no evidence related to the Indicator and/or significant concerns are noted.
Developing: The program demonstrates inconsistent evidence related to the Indicator and/or moderate concerns are noted.
Proficient: The program demonstrates consistent evidence related to most standards of the Indicator and/or minor concerns are noted.
Exemplary: The program demonstrates consistent evidence related to all standards of the Indicator and is a potential exemplar in this area.

13. Classroom Observations
Data Collection
Classroom Observations
Classroom observations do not exceed 30 minutes each and are conducted in such a way as to not disrupt classroom activities. The Team will visit several classes and use a tool and rubric to gather evidence on activities and practices related to four essential elements of high-quality instruction: scaffolding, differentiated instruction, checking for understanding, and student engagement.
Interview Questions
Document Review
For some of the Indicators, ACLS requires programs to have specific documents prepared for the day of the visit, prior to the team’s arrival.
During the visit, the Team will interview program leadership, a sampling of teachers, advisors, and students. No written response to the interview questions is expected.

14. Interviews Each interview must be held in a secure, quiet location.
More than one interview may be scheduled at the same time during the visit.
Each participant must be present for the entirety of the interview and is given complete freedom to speak.
Attendees should not attend multiple group interviews.
In general, supervisors and supervisees should not be in the same group interview.

15. Interviews-cont. Program Leadership Teachers Advisors Students .5 hour
Must represent all levels and content areas services by the program
Depending on program size, the group may vary in size. Final number to be decided during preparation phone call.
1. 5 hour
Must include staff who oversee the AE program directly and are responsible for its daily management (program director, assistant director, lead teacher). Must include program’s director’s supervisor.
Advisors
Students
.5 hour
Should include maximum 10 students
Must represent all levels and content areas serviced by the program
Program staff must not be in attendance.
1 hour
Must represent advisors who work with students at all levels.

16. The team will ask questions related to NRS follow-up.
Interview Questions
Interview questions will relate to the 4 Indicators reviewed during the visit.
The team may ask questions related to Indicator 5 (Student Performance) depending on concerns related to MSG performance.
The team will ask questions related to NRS follow-up.

17. Indicator of Program Quality
Document Review
Indicator of Program Quality
Documents Requested
Page Limit
When
IPQ 4-Curriculum and Instruction
Lesson plans
Corresponding scope and sequences and curriculum units for the observed classes.
Lesson plan-2 pages
Corresponding scope and sequences and curriculum units-3 pages each
Lesson plan-the morning of the visit
Scope and Sequences and curriculum units-2 weeks before the visit
IPQ 7-Organizational Support
Resumes and job descriptions for the program’s leadership
Resume and job description program leaders and supervisor- 2 pages each
2 weeks before the visit
IPQ 8-Educational Leadership
Three copies of actual feedback director provided to teachers including but not limited to supervision, classroom observations, and annual evaluation reviews.
N/A

18. Structure of the Day – Sample Schedule #1
Sample Schedule for a smaller program, 2-3 member team. (*Note: Start and end times of activities are subject to change.)
Activity
Activity Length
Schedule
Staff Involved
Indicators of Program Quality Focus
Team arrival and set-up
15 min
8:30-8:45
PQR Team
N/A
Facility Tour
8:45-9:00
PQR Team Members & Program Director 7
Interview with program leadership
90 min
9:00-10:30
PQR Team Member (s) & Program Leadership
3, 4, 7, 8
Classroom Observations
60 min
10:30-11:30
PQR Team Members & Teaching Staff 4
Student Focus Group Interview
30 min
11:30-12:00
PQR Team Members & Students
3, 4, 7 ,8
Teacher Focus Group Interview
12:00-1:00
Lunch & Team Check-in
1:00-1:30
Advisor Focus Group Interview
1:30-2:30
PQR Team Members & Advisors
3, 7 , 8
Team Work Time (aggregate classroom observations data, sort evidence, build preliminary finding, prepare report-out)
105 min
2:30-4:15
Report Out
4:15-4:30
PQR Team and Program Leadership Staff

19. Structure of the Day – Sample Schedule #2
Sample Schedule for a larger, multi-site program, 6-7 member team (*Note: Start and end times of activities are subject to change.)
Schedule
Team members 1, 2, 3
Team members 4 , 5, 6, 7
8:30-8:45
Team arrival and set-up
8:45-9:00
Facility Tour
9:00-10:30
Interview with program leadership
10:30-11:30
Classroom Observations
Advisor Focus Group Interview
11:30-12:00
Student Focus Group Interview
Classroom observations
12:00-12:30
Lunch and Team check-in
12:30-1:30
Teacher Focus Group Interview
1:30-4:15
Team Work Time (aggregate classroom observations data, sort evidence, build preliminary finding, prepare report-out)
4:15-4:30
Report-out

20. Logistics The program director should inform the team of:
Parking logistics
Food allergy restrictions in the building
Meeting place in the morning
Debrief room
Interview room
Team members cannot accept or solicit any meals.
Program director needs to be available throughout the day for any clarifying questions.

21. End of Day Report-Out
Program leadership and their supervisor are required to attend.
The team will only share a set of preliminary findings, most likely on one of the four Indicators only.
The team will not provide ratings or extensive feedback at this time.

22. How to Prepare for a Program Quality Review02

23. Before the Program Quality Review
Director’s Role
Serve as the main point of contact for the PQR.
Work with the Review Team leader to confirm the visit date.
Call or the Review Team leader if there are any sudden changes in the program schedule.
Work with the Review Team Leader and program staff to determine the agenda for the day.
Reserve a secure, quiet meeting room space for the Review Team to conduct interviews and debrief throughout the day.
Ensures lesson plans and lesson materials are made available in the Review Team’s meeting room before the team arrives.
Gather and organize all required documents for the Review Team’s document review.
required documents to the Review Team leader two weeks in advance of the visit.

24. Before the Program Quality Review
Prepare Staff
Review PQR agenda and schedule with program staff participating in the review.
Review Indicators of Program Quality 3, 4, 5, 7 and 8 with program staff participating in the review.
Ensure that program staff understand the purpose of the program quality review and what will be expected of them.
Ensure program staff know the Program Quality Review Protocol.
Identify and prepare a group of 10 students representative of all program’s levels for the student interview.

25. Useful Documents
Program Leadership and staff participating in the PQR should review the following documents in preparation for the visit:
Program Quality Review Protocol
Indicators of Program Quality
ACLS Performance Standards

26. What to Expect During a Program Quality Review03

27. During the Program Quality Review
Director’s Role
Act as the main point of communication between the program and the PQR team.
Ensure that copies of any lesson plans and lesson materials are made in advance for the scheduled classroom observations.
Be available to meet with the team during schedule times and throughout the day for any additional questions or follow-up.
Bring any questions or concerns to the Review Team leader
Attend the report-out at the end of the day.

28. What to Expect After a Program Quality Review05

29. Written Report
Director’s Role
The Review Team Leader will send a report to the Program Director within three weeks of the visit. The Program Director should:
Work with relevant program staff, agency/organization leadership to review the report.
Share the report with program staff and other key stakeholders (e.g. Board of Directors, Superintendent, Dean, CEO).
If applicable, respond to the report within 2 weeks after receipt.
Develop an improvement plan if needed.
Works with their assigned program specialist and/or SABES to receive further TA on the Indicators rated low.

30. Adult and Community Learning Services (ACLS)
THANK YOU
Adult and Community Learning Services (ACLS)
75 Pleasant Street, Malden, MA 02148