Badge User Debrief: School Year 2015-16 and Summer 2016 September 30, 2016

Agenda Welcome and introductions Badging in 2016: system-wide trends and the user experience The road ahead Next steps and closing

11 organizations used badges in school year 2015-16 and summer 2016, serving 937 youth in grades 4 through 12 MathPOWER, DotHouse, Thompson Island, Hale Reservation, and Sociedad Latina participated in the Summer 2015 pilot. They were joined this past school year by Citizen Schools at Orchard Gardens. In Summer 2016, one more middle school program, 826 Boston participated, along with four high school serving programs.

10 different badges available to earn X 5 Skills Communication Critical Thinking Engagement in Learning Perseverance Teamwork 2 Thresholds Achievement Growth

Process for awarding badges Educator who knows (or will know) each student best records pre and post SAYO T observations of student in Cityspan Program records student attendance Students with >80% attendance are eligible to earn badges

Process for awarding badges (cont.) Students who have been observed to use these skills “most of the time” (4 out of 5 on the SAYO T) are eligible to earn an achievement badge Students whose SAYO T score increases by at least 1 point between the pre and post observation are eligible to earn a growth badge Given this criteria, badges are awarded by the program.

Total Eligible Students % of Students who are Earners The percentage of students who earned badges increased from 2015 to 2016 Year Total Eligible Students Total Badge Earners % of Students who are Earners 2016 937 560 60% 2015 470 297 54% The figures for 2015 only include incidents in which students met the criteria to earn badges– they do not include badges that were awarded using Cityspan’s override function. There were no recorded cases of such overrides in 2016. In this presentation, reference to “2016” includes all badges awarded in 2016-- both school year and summer– unless stated otherwise.

Achievement badges outnumbered growth badges by about 2:1 in 2016 Total Badges Earned Total Achievement Growth 2016 2590 1790 800 2015 1139 854 285 2016 2015

Distribution of students by number of badges earned Number of students Number of badges earned

Badges remain similarly distributed across skill areas, with the fewest earned for Critical Thinking and the most earned for Teamwork 2016 2015

Overall, the percentage of students earning three of the Growth badges significantly increased since 2015 Percentage of students earning * * * Compared with the 2015 pilot, students earned significantly more Growth in Critical Thinking, Growth in Engagement in Learning, and Growth in Perseverance badges in 2016. 2015 2016 Achievement 2016 Growth * Denotes significant difference, p<0.05

In 2016, students were more likely to earn achievement badges during the school year and growth badges during the summer * Percentage of students earning * * Achievement Growth SY Summer SY Summer * Denotes significant difference, p<0.05

High school students were significantly more likely than students in grades 4-8 to earn achievement badges in 2016 * * * * Percentage of students earning Achievement Growth G4-8 HS G4-8 HS * Denotes significant difference, p<0.05

Students continue to be more likely to earn both an achievement and a growth badge than a growth badge alone

Bigger picture questions raised by this data Are increases in Critical Thinking, Engagement, and Perseverance badges related to 2016 professional development efforts? What are the expectations to which program staff hold their students? Are they appropriate for the grade levels they serve? Is it reasonable to award Growth badges during the summer using this criteria, given the relatively short time frame?

Questions for badge users Do you feel that your experience using badges this past year is reflected in this data? How did students react to earning (and not earning) badges this year? Were there any changes from last year? How did your team approach situations in which a student did not earn a badge? How was the technical user experience? Tech support and reminders from BASB Cityspan functionality Timeline and staff capacity

Agenda Welcome and introductions Badging in 2016: system-wide trends and the user experience The road ahead Next steps and closing

Technical updates to the badging system that are in progress Report on badges issued through Cityspan View of badges by session and by student Acceptable use policy Badge assertion page

The badge report in Cityspan

Technical updates to the badging system that are in progress Report on badges issued through Cityspan View of badges by session and by student Acceptable use policy Badge assertion page

Achievement in Communication Badge Description: About: Criteria: Awarded To: <<FIRST NAME>> <<LAST NAME>> Program: <<ORGANIZATION>>, <<SITE/ACTIVITY>> Date Issued: <<xx/xx/xxxx>> Issued By: <<INTERMEDIARY>> Endorsed By: <<Hide if none are available>> Achievement in Communication Badge Description: Students who earned this badge consistently demonstrated effective communication skills while participating in a high-quality education program. About: This badge was issued by <<INTERMEDIARY>>, and has been awarded by <<ORGANIZATION>>. <<Automatically pull program description and activities>> … Criteria: To receive this badge, the student attended <<PROGRAM>> at least 80% of the time they were enrolled. The student demonstrated communication skills by: regularly volunteering to ask or answer questions effectively expressing ideas to others speaking clearly and loudly enough for others to hear, demonstrating active learning skills contributing to group discussions Educators at <<ORGANIZATION>> assessed communication using the SAYO T, a survey tool developed by the National Institute on Out-of-School Time. For an Achievement badge, the student earned at least a 4.0 out of 5.0 on the SAYO T, meaning that this student demonstrated their communication skills “most of the time” while at <PROGRAM>>. Draft of the assertion page– the webpage that a student can direct the public to view upon earning a badge. It will contain information about the program in which the student earned the badge as well. Academic Standards: This badge aligns with the following Next Generation Science Standards practices for grades 6-8: Asking Questions and Defining Problems Engaging in Argument from Evidence Obtaining, Evaluating, and Communicating Information Evidence: Attendance Average daily attendance: x.xx% Total hours attended: xxx Skill Assessment Average SAYO T score: earned at least a 4.0 out of 5.0 Demonstration <<Hide if none are available>>

Badges for high school students Success Boston supported BASB and Ann Coles to develop the concept of four college readiness badges for high school students over the past six months Boston Public Schools is determining how to integrate badges throughout summer learning and high school redesign Nellie Mae Education Foundation is considering supporting BPS’s efforts, in partnership with BASB and Youth On Board

Evaluation of students’ experience Market research conducted in 2016 revealed that there is great potential value in awarding badges to recognize skill development More research on students’ reactions to earning or not earning badges is necessary Are badges motivating? Is it detrimental to not earn a badge? Are there other risks we have not identified?

How should we navigate these upcoming developments and events? What is the best way to collect data on youth experience with badges? What additional training and support is needed to use the system to its fullest potential? How could the Cityspan experience improve?

Next steps for badging 3 program sites to use the existing badge system in school year 2016-17 Evaluation of middle school youth experience to take place in spring and summer 2017 If funded, high school badge development to begin in Jan 2017 with goal of testing criteria in summer 2017