1. Badge User Debrief: School Year 2015-16 and Summer 2016
September 30, 2016

2. Agenda Welcome and introductions
Badging in 2016: system-wide trends and the user experience
The road ahead
Next steps and closing

3. 11 organizations used badges in school year and summer 2016, serving 937 youth in grades 4 through 12
MathPOWER, DotHouse, Thompson Island, Hale Reservation, and Sociedad Latina participated in the Summer 2015 pilot. They were joined this past school year by Citizen Schools at Orchard Gardens. In Summer 2016, one more middle school program, 826 Boston participated, along with four high school serving programs.

4. 10 different badges available to earnX
5 Skills
Communication
Critical Thinking
Engagement in Learning
Perseverance
Teamwork
2 Thresholds
Achievement
Growth

5. Process for awarding badges
Educator who knows (or will know) each student best records pre and post SAYO T observations of student in Cityspan
Program records student attendance
Students with >80% attendance are eligible to earn badges

6. Process for awarding badges (cont.)
Students who have been observed to use these skills “most of the time” (4 out of 5 on the SAYO T) are eligible to earn an achievement badge
Students whose SAYO T score increases by at least 1 point between the pre and post observation are eligible to earn a growth badge
Given this criteria, badges are awarded by the program.

7. Total Eligible Students % of Students who are Earners
The percentage of students who earned badges increased from 2015 to 2016
Year
Total Eligible Students
Total Badge Earners
% of Students who are Earners
2016
937
560
60%
2015
470
297
54%
The figures for 2015 only include incidents in which students met the criteria to earn badges– they do not include badges that were awarded using Cityspan’s override function. There were no recorded cases of such overrides in In this presentation, reference to “2016” includes all badges awarded in both school year and summer– unless stated otherwise.

8. Achievement badges outnumbered growth badges by about 2:1 in 2016
Total Badges Earned
Total
Achievement
Growth
2016
2590
1790
800
2015
1139
854
285
2016
2015

9. Distribution of students by number of badges earned
Number of students
Number of badges earned

10. Badges remain similarly distributed across skill areas, with the fewest earned for Critical Thinking and the most earned for Teamwork
2016
2015

11. Overall, the percentage of students earning three of the Growth badges significantly increased since 2015
Percentage of students earning * * *
Compared with the 2015 pilot, students earned significantly more Growth in Critical Thinking, Growth in Engagement in Learning, and Growth in Perseverance badges in 2016.
2015
2016 Achievement
2016 Growth
* Denotes significant difference, p<0.05

12. In 2016, students were more likely to earn achievement badges during the school year and growth badges during the summer *
Percentage of students earning * *
Achievement
Growth SY
Summer SY
Summer
* Denotes significant difference, p<0.05

13. High school students were significantly more likely than students in grades 4-8 to earn achievement badges in 2016 * * * *
Percentage of students earning
Achievement
Growth
G4-8 HS
G4-8 HS
* Denotes significant difference, p<0.05

14. Students continue to be more likely to earn both an achievement and a growth badge than a growth badge alone

15. Bigger picture questions raised by this data
Are increases in Critical Thinking, Engagement, and Perseverance badges related to 2016 professional development efforts?
What are the expectations to which program staff hold their students? Are they appropriate for the grade levels they serve?
Is it reasonable to award Growth badges during the summer using this criteria, given the relatively short time frame?

16. Questions for badge users
Do you feel that your experience using badges this past year is reflected in this data?
How did students react to earning (and not earning) badges this year?
Were there any changes from last year?
How did your team approach situations in which a student did not earn a badge?
How was the technical user experience?
Tech support and reminders from BASB
Cityspan functionality
Timeline and staff capacity

17. Agenda Welcome and introductions
Badging in 2016: system-wide trends and the user experience
The road ahead
Next steps and closing

18. Technical updates to the badging system that are in progress
Report on badges issued through Cityspan
View of badges by session and by student
Acceptable use policy
Badge assertion page

19. The badge report in Cityspan

20. Technical updates to the badging system that are in progress
Report on badges issued through Cityspan
View of badges by session and by student
Acceptable use policy
Badge assertion page

21. Achievement in Communication Badge Description: About: Criteria:
Awarded To: <<FIRST NAME>> <<LAST NAME>>
Program: <<ORGANIZATION>>, <<SITE/ACTIVITY>>
Date Issued: <<xx/xx/xxxx>>
Issued By: <<INTERMEDIARY>>
Endorsed By: <<Hide if none are available>>
Achievement in
Communication
Badge Description:
Students who earned this badge consistently demonstrated effective communication skills while participating in a high-quality education program.
About:
This badge was issued by <<INTERMEDIARY>>, and has been awarded by <<ORGANIZATION>>. <<Automatically pull program description and activities>> …
Criteria:
To receive this badge, the student attended <<PROGRAM>> at least 80% of the time they were enrolled. The student demonstrated communication skills by:
regularly volunteering to ask or answer questions
effectively expressing ideas to others
speaking clearly and loudly enough for others to hear,
demonstrating active learning skills
contributing to group discussions
Educators at <<ORGANIZATION>> assessed communication using the SAYO T, a survey tool developed by the National Institute on Out-of-School Time. For an Achievement badge, the student earned at least a 4.0 out of 5.0 on the SAYO T, meaning that this student demonstrated their communication skills “most of the time” while at <PROGRAM>>.
Draft of the assertion page– the webpage that a student can direct the public to view upon earning a badge. It will contain information about the program in which the student earned the badge as well.
Academic Standards:
This badge aligns with the following Next Generation Science Standards practices for grades 6-8:
Asking Questions and Defining Problems
Engaging in Argument from Evidence
Obtaining, Evaluating, and Communicating Information
Evidence:
Attendance
Average daily attendance: x.xx%
Total hours attended: xxx
Skill Assessment
Average SAYO T score: earned at least a 4.0 out of 5.0
Demonstration
<<Hide if none are available>>

22. Badges for high school students
Success Boston supported BASB and Ann Coles to develop the concept of four college readiness badges for high school students over the past six months
Boston Public Schools is determining how to integrate badges throughout summer learning and high school redesign
Nellie Mae Education Foundation is considering supporting BPS’s efforts, in partnership with BASB and Youth On Board

23. Evaluation of students’ experience
Market research conducted in 2016 revealed that there is great potential value in awarding badges to recognize skill development
More research on students’ reactions to earning or not earning badges is necessary
Are badges motivating?
Is it detrimental to not earn a badge?
Are there other risks we have not identified?

24. How should we navigate these upcoming developments and events?
What is the best way to collect data on youth experience with badges?
What additional training and support is needed to use the system to its fullest potential?
How could the Cityspan experience improve?

25. Next steps for badging
3 program sites to use the existing badge system in school year
Evaluation of middle school youth experience to take place in spring and summer 2017
If funded, high school badge development to begin in Jan 2017 with goal of testing criteria in summer 2017