1. Measurement Orientation
School Year
September 21, 2017

2. Agenda Time Agenda Item Objective 10:00 AM – 10:05 AM Introductions
Introduce each other, establish norms, get refreshments
10:05 AM – 10:45 AM
Review Membership Level: Enrollment & Attendance
Attendees will understand baseline requirements for measurement work
10:45 AM – 11:15 AM
Review Membership Level: Program Quality
Attendees will understand additional requirements for Program Quality level
11:15 AM – 11:35 AM
Review Membership Level: Youth Outcomes
Attendees will understand additional requirements for Youth Outcomes level
11:35 AM – 12:00 PM
Next Steps
Attendees will understand next steps involved to become a member of the Boston Beyond network

3. Guiding Principles of Measurement
Program staff:
1. Understand SE skills
2. Receive PD on how to develop SE skills
3. Integrate practices that develop SE skills into programming
Programming designed to develop SE skills is provided to students
Students participate in and complete all components of program
Students develop SE Skills
Students are better prepared for success in school, careers, and life
Inputs
Activities
Outputs
Initial
Long-term
Outcomes
Although the next few hours will focus on the logistics of implementing measurement at your programs, we wanted to begin with the guiding principles of our network’s measurement.
ACT Skills Framework: what skills are important for a child’s success (research, stakeholder input, etc.)
Cycle of Continuous Improvement: helps programs know where they’re at and how to get better/how to better meet the needs of their youth
Network, program, and participant security: ensures we use data responsibly to meet network, program, and participant goals
The main goal of Boston After School & Beyond is to ensure that all youth in the City of Boston are reaching their full potential through access to high quality programs. In these programs, they can learn and develop skills important for their success in life.
This logic model is our approach to meeting that goal.

4. Track Data in Common Database: Social Emotional Skills:
Continuous Improvement
The Cycle of Continuous Improvement helps programs know where they’re at and how to get better/how to better meet the needs of their youth.
Track Data in Common Database:
-over time
-city-wide comparisons
Student Enrollment & Program Attendance
Teacher & Student Perspectives
Social Emotional Skills:
Observer and Student Perspectives
Program Quality:
Peer-to-Peer Connections and Learning
&
Targeted Professional Learning
Cycle of Continuous Improvement: helps programs know where they’re at and how to get better/how to better meet the needs of their youth

5. Associated Outcomes from Research Review
The ACT Skills Framework guides the measurement and includes skills that key stakeholders (business, colleges, and program providers) deem important for a child’s success.
ACT Skills Framework
ACT Skills Framework
Associated Outcomes from Research Review
ACHIEVING
Critical Thinking
College success and workforce readiness; Employee selection by management
Creativity
Improved academic achievement
Perseverance
High education attainment; Higher undergraduate GPAs; Fewer career changes; Work habits and task persistence; Math achievement
CONNECTING
Social Awareness & Relationships
Improved academic test scores; Better job performance; Improved grades; Healthier relationships; Improved social skills; Improved self-efficacy, problem-solving
Communication
Healthier social relationships; Improved self-efficacy
Teamwork
Improved academic achievement; Improved self-efficacy
THRIVING
Growth Mindset
Improved performance and academic achievement; Lower stress levels
Efficacy
Improved academic achievement; Academic persistence; College GPA, persistence; Improved work performance; Improved perseverance
Self-Regulation
Positive academic outcomes (grade promotion, test scores, course grades); College GPA and retention; Positive social outcomes (better impulse control, more stable relationships, increased empathy and perspective taking, more constructive response to anger)
ACT Skills Framework: what skills are important for a child’s success (research, stakeholder input, etc.)
Network, program, and participant security: ensures we use data responsibly to meet network, program, and participant goals

6. Guiding Principles of Measurement: Data Security
Consent forms and memorandums of understanding (MOUs) protect program and students data and ensure we are using data responsibly to meet network, program, and participant goals.

7. Measurement Overview Youth Outcomes Program Quality
Tools Used: Cityspan; APT; SAYO Y;
SAYO, Teacher version (SAYO T) Pre and Post
Data Received: Program Report for Improvement & System Measurement (PRISM), Individual Student SAYO T Reports
Program Quality
Tools Used: Cityspan
Assessment of Program Practices Tool (APT)
Survey of Academic and Youth Outcomes, Youth version (SAYO Y)
Data Received: Program Report for Improvement & System Measurement (PRISM)
Enrollment & Attendance
Tools Used: Cityspan
Data Received: Enrollment & Attendance Reports

8. Enrollment and Attendance
GOAL: To understand who is being served by Boston programs, and how often. HOW: Sites share demographic, geographic info on participants, and daily attendance data. TOOLS USED: Cityspan database BENEFITS: Access to variety of documents and reports to assist with operations and attendance reporting needs.
“In your folder” COLOR CODE
The main goal of the enrollment and attendance level is to understand who is being served by Boston programs, and how often those youth are attending quality programs. To do this, programs share participant demographic, geographic, and their daily attendance data. This responsibility applies to this and all subsequent membership levels (include these responsibilities on subsequent slides).
The main tool used at this level is Cityspan—Boston Beyond’s city-wide database. The database provides a simple, yet comprehensive interface to track participant information and their daily attendance in the program. Through the database, programs have access to a variety of attendance tracking documents and reports to assist with their daily operations or attendance reporting needs.

9. MOU: Overview & Logistics Memorandum of Understanding
TWO PARTS:
Purpose:
Includes:
Who Should Sign:
Occurs:
Due:
Master Agreement
Statement of Work
To create mutual understanding of the rights and governance of each organization.
Data Sharing, Security, and Confidentiality; Mutual Indemnification; etc.
Executive Director or Equivalent
Once, at start of partnership
Before executing first Statement of Work
To outline roles and responsibilities of each organization in work done together (e.g. measurement).
Project description; Responsibilities of Boston Beyond and Partner; etc.
Whoever the program deems appropriate (may be program director, site coordinator)
At the start of each initiative
Before program begins
Enrollment & Attendance
Program Quality
Youth Outcomes

10. Parental Consent Forms: Overview & Logistics
Purpose:
To protect student’s right to privacy
Consent for:
BASB and research partner access to demographic data, youth survey participation, media release
Who Should Sign:
Parents of all youth participants
Occurs:
Once every year
Due:
Within first week of program
Which languages they’re available in
Pro Tip: Incorporate consent form into your own enrollment/registration package, or distribute during a parent orientation.
Enrollment & Attendance
Program Quality
Youth Outcomes

11. Enrollment & Attendance
Cityspan: Overview
boston.cityspan.com
Remember to include list of preferred participant information
Enrollment & Attendance
Program Quality
Youth Outcomes

12. Cityspan: Logistics
Enter Site Information
Add Youth Participants
Create a Program
Enroll Youth Participants
Track Daily Attendance
Once per year
Daily or Weekly
First four steps completed in by end of first two weeks of program.
User tutorials available on Boston Beyond’s Youtube channel.
Fall data entered by December 1
Importance of different data: data entry rationale
Show attendance tracking sheets
Enrollment & Attendance
Program Quality
Youth Outcomes

13. Reporting at the Enrollment and Attendance Level
Overall
Students Served
Total Days Scheduled
Total Hours Scheduled
Average Rate of Attendance
Drop Out Rate*
No Show Rate** 16 12
15.00
65.61%
18.75%
0.00%
Enrollment & Attendance
Program Quality
Youth Outcomes

14. Program Quality
GOAL: To understand your program’s strengths and areas in need of improvement. HOW: Sites host third-party observation of their program practice and administer a youth survey on program experience. Sites nominate a staff member to join the Certified Observer Network. TOOLS USED: Cityspan, Assessment of Program Practices Tool (APT) and the Survey of Academic and Youth Outcomes, Youth version (SAYO Y) BENEFITS: Data on program quality from two perspectives: an external adult observer, and their own students. Program report comparing own data to that of the network. *Recommended membership level for first-year partners
Color code this slide.
Goal: To understand your program quality. To do this, sites host a third-party observation of their program practice and administer a youth survey on program experience. Tools used: the Assessment of Program Practices Tool and the Survey on Academic and Youth Outcomes (SAYO Y)
Benefits: Programs receive qualitative and quantitative data on their program quality from two perspectives: an external adult observer, and their own students. Programs receive a report at the end of the programming that compares their own data to that of the network at large.

15. Program Quality Through APAS Afterschool Program Assessment System by the National Institute on Out of School Time
Gathers evidence to answer:
What is the quality of our program?
How are youth experiencing our program?
How can we improve?
Provides common language of quality for:
Program staff and leadership
Boston Beyond’s network of programs
Program Quality
Youth Outcomes
Youth Outcomes

16. APT: Overview Assessment of Program Practices Tool
Third party observation from NIOST Master or BASB Certified observer
Measures quality of practice in three domains:
Supportive Social Environment
Opportunities for Engagement and Skill Building
Program Organization & Structure
Program Quality
Youth Outcomes

17. APT: Logistics Scheduling a visit Preparing your program for the visit
Choose a typical day
Expect to host the observer all day
Information you may need to share:
Parking instructions
Point of contact on the day of
The program’s schedule for the day of
Preparing your program for the visit
Staff
Youth
Program Quality
Youth Outcomes

18. SAYO Y: Overview Survey on Academic and Youth Outcomes, Youth version
Youth survey on program experience and quality
Provides youth perspective on two of the quality domains:
Supportive Social Environment
Opportunities for Engagement and Skill Building
Includes three scales from the Youth Experiences Survey:
Self regulation
Teamwork
Problem solving
Program Quality
Youth Outcomes

19. SAYO Y: Logistics For youth in grades 4 and above
Within last month of the program
Allow half hour to administer
Only to youth with consent
Paper or electronic versions available
Instructions for administration will accompany the survey
Program Quality
Youth Outcomes

20. BASB Observer Certification
3) Observe and assess fellow youth-serving programs.
1) Train to use the APT in a program setting.
2) Receive a BASB Certificate in program observation.
Benefits:
Access a validated tool to assess your own program
See best practices in action throughout Boston
Network with fellow professionals in the field
Next Training: October 4 and 5!
Do we have a photo of this?

21. Reporting at the Program Quality Level
Immediate feedback: Site Visit Summary
Highlights of each section of APT
Program strengths and areas in need of improvement
Received within one week of your observation
Program Report on Improvement & System Measurement (PRISM)
See How to Read Your PRISM on the Insight Center:
Program Quality
Youth Outcomes

22. Linking APT and SAYO Y to Participant Skill Outcomes
ACT Skills Framework
Associated Outcomes from Research Review
Critical Thinking
APT: Activity is challenging and stimulates thinking.
Youth Experiences Survey: I have observed how others solved problems and learning from them at my program.
Social Awareness & Relationships
APT: When minor conflicts occur, youth are able to problem-solve together to resolve conflicts.
SAYO Y: Are kids here friendly with each other?
Self-Regulation
APT: When an individual youth is having a problem or is upset, staff pay attention and try to help.
Youth Experiences Survey: I learned about controlling my temper at my program.
See Intentional Program Practices handout for more information.
Program Quality
Youth Outcomes

23. Youth Outcomes
GOAL: To understand skill development in the youth you serve. HOW: Administer an educator survey assessing individual youths’ skills. TOOLS USED: Cityspan; APT; SAYO Y; Survey of Academic and Youth Outcomes, Teacher version (SAYO T) BENEFITS: Receive individual and aggregate data on the skill growth and achievement of youth participants.

24. Youth Outcomes Through APAS Afterschool Program Assessment System by the National Institute on Out-of-School Time
Gathers evidence to answer:
Are youth developing in relationships, skills and behaviors they need to be successful?
Provides common language for:
Youth in program
Program staff and leadership
BASB network of programs
Youth Outcomes

25. SAYO T: Overview Survey on Academic and Youth Outcomes – Teacher version
Staff survey on youth skill development
Pre/Post to capture growth over course of program
90 hours of program time required
Measures students development in six outcomes:
Perseverance
Critical Thinking
Relationships with Peers
Relationships with Adults
Communication
Self-Regulation
Youth Outcomes

26. SAYO T: Logistics
Pre administration: First month of program Post administration: Last two months of program Same staff person completes both pre and post Administered through Cityspan Takes 5-10 minutes per survey
Youth Outcomes

27. Reporting at the Youth Outcomes Level
Individual student progress
SAYO T PRISM
Own programs data, comparison with cohort
See SAYO T PRISM section on Insight Center:
Youth Outcomes

28. Enrollment & Attendance
Next Steps
Complete Program Information Form by September 25
Look out for MOU
Look out for Cityspan training
Enrollment & Attendance
Program Quality
Youth Outcomes