NSF Broadening Participation in Computing Alliances (BPC-A) Introduction to Common Core Indicators of Breadth and Depth of Participation Daryl E. Chubin, Ph.D., Senior Advisor American Association for the Advancement of Science CE21—Portland, Oregon January 14, 2013

Evaluation Technical Assistance Team Daryl E. Chubin, American Ass’n for the Advancement of Science Betsy Bizot, Computing Research Association Tom McK lin, The Findings Group Alan Peterfreund, SageFox Consulting Group In collaboration with BPC-A project evaluators... This has been a community-driven undertaking.

Broadening Participation in Computing Alliances History Program launched in with funding of 10 Alliances Multi-institutional interventions in any segment of the education pathway, precollege (formal and informal) to workforce Most Alliances focused on undergraduate segment In 2009, the Alliance evaluators undertook to develop, in addition to annual project report information, a set of Common Core Indicators that would allow reporting of progress at the program level In , 10 of 13 Alliances participated in submission to a review team of a Common Core report Review team assembled a program-level report (data + narrative), which was submitted to NSF in August 2012 Sources Chubin and Johnson, Telling the Stories of the BPC Alliances, June 2010, Chubin et al., NSF’s BPC Alliances Program—A Report on Common Core Indicators of Breadth and Depth of Participation, August 2012 (available on request)

BPC Alliances—Common Core Indicators ( ) The review team identified three indicators that, by consensus among BPC Alliance (BPC-A) evaluators, cover the “common core” at the heart of all the NSF- funded Alliances. Detailed descriptions of and measurement strategies for each indicator were developed. The three indicators are: 1.Individual Participation and Outcomes 2.Organization Capacity Development 3.Alliance Impact

CCI Methodology and Review Process Each Alliance is different, thus each report was reviewed individually to contextualize its metrics and terminology. Then each indicator was applied across all alliances. Most alliances were captured in their latter stages (typically year 4 or 5). The composite yielded a picture of program performance. The team notes the difference between:  “broadening participation” and measuring broadening participation  designed (by intention) and implemented (through project evolution)

Indicator 1 The review team used a single year, academic year , to capture both in-school and summer activities for all projects. Over 90% of K-12 students are participating in what BPC-A evaluations report as a level of “deeper engagement.”  Significant evidence is most compelling for participants who are more deeply engaged, with the least evidence found for undergraduate students merely “touched” by the program. The majority of faculty participation is limited engagement among K-12 teachers.  Faculty more deeply engaged in the BPC projects were more likely to effect positive change among their undergraduate students in attitudes toward and persistence in computing.

Indicator 2 For this indicator, all active years of the project were used, i.e., as a cumulative measure. All alliances except one intentionally build faculty communities; 8 of 10 alliances intentionally build student communities.  Aside from academic institutions, BPC projects have an impact through collaboration on youth- and community-based organizations, professional associations, industry partners, and public sector agencies. All 10 projects present evidence of changes in knowledge/skill development, generating/ disseminating tools, and expanding stakeholder awareness.

Indicator 3 Without the alliances fostering identity with the broader community, no claim of cultural change as a national phenomenon in computing can be made. Six of the 10 Alliances affirmed a transition in developing ties with entities not originally designated as partners, with expansion and progress toward sustainability observed.  Alliances have impacted a wide variety of organizations outside their initial partners, most notably other institutions of higher education, other K-12 institutions, and the broader CS community.

Community The idea of building community cuts across all common core indicators for BPC alliances.  Community is intertwined with the activities an Alliance conducts and with the experiences of its participants. The Alliances share no common framework for tabulating the value of community participation. Community alliances appear to serve three key functions:  a strategy for deepening engagement,  a way of leveraging or increasing the impact of formal instruction or interaction during an activity, and  a way of sustaining the effect of a short-term activity such as a workshop by providing an avenue for continuing support and engagement.

Recommendations Indicators should be assembled at the outset of new programs, rather than developed concurrently with them. NSF expectations for new programs should be made clear in the solicitation, with the caveat that overly-rigid expectations can stifle innovation. A common language is needed for describing both the processes and outcomes around community-building.  Words like “collaborator” and “partner” have different meanings for different practitioners, yet are used interchangeably. The story of the BPC Alliances cannot be told with numbers alone.  While vigilant measurement is essential, so too is attention to the unique characteristics of each Alliance and the circumstances in which it is expected to thrive.