Maximizing Evaluation Impact by Maximizing Methods: Social Network Analysis Combined with Traditional Methods for Measuring Collaboration Carl Hanssen, PhD & MaryAnn Durland, PhD American Evaluation Association Baltimore, MD November 7, 2007

11/07/20072 Agenda  Social Network Analysis: The Method  SNA Results and Interpretation  Next Steps

11/07/20073 SNA Methodology  Network Analysis is the study of the relationships formed by the interaction or links between components in a “set”.  MMP sets are schools  The components are individuals: Faculty, both math and non math MTL (School level Math Teacher Leaders) MTS (District level Math Teacher Specialists)  Relationship is communication about math

11/07/20074 Measures  Indegree – popularity  Density – how “thick”, how much, out of potential

11/07/20075 Role in Evaluation  How much does the communication structure actually fit the theory and the design of the project  Can the structure be correlated with other measures of implementation and impact? Activities Proximal measures

11/07/20076 MMP Evaluation Logic Model Student Achievement Teacher Content & Pedagogical Knowledge Math Faculty Involvement Learning Team Effort School Buy-in Teacher Involvement New Courses District Buy-in MPA Ownership MATC Buy-In UWM Buy-In Classroom Practice MMP Activities Proximal Outcomes Distal Outcomes

11/07/20077 MMP Report Card Indicators  19 indicators in 7 domains derived from in-school data collection, online surveys, and MPS data 1. MTS Assessment 2. Collaboration 3. Learning Teams 4. Classroom Practice 5. Professional Development 6. Teacher MKT 7. Student Achievement

11/07/20078 SNA In Context: Evaluation Results Student Achievement Teacher Content & Pedagogical Knowledge Learning Team Effort School Buy-in Teacher Involvement Classroom Practice WKCE Mean % Proficient = 44% Overall rating = 3.5 Gap MTL v. other teacher =.2 Teacher Engagement = 3.2 Overall IRT = Algebra IRT = Team Functioning = 3.5 MMP Principles = 3.6 LT Quality = 3.1 PD Hrs. = 17.8 Facilitation Hrs. = 1.0 PD Quality = 3.1 Network density = 6.7% / School density = 17.6% MTL Role = 13.8 / MTS Role = 5.3 SR MTL Engagement = 4.4 / MTS Quality = 3.0 MTS Assessment = 38.3 of 55

11/07/20079 Data Collection  Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics  Statistical analysis focused on 1. Network and in-school density 2. Importance of MTL and MTS

11/07/ MMP Impact Continuum LowHigh Loose Web MTL Not Central Few Links to MTL MTS Outside Few Links to MTS Tight Web MTL Central Many Links to MTL MTS Inside Many Links to MTS

11/07/ Low

11/07/ Medium

11/07/ High

11/07/ Student Achievement & In-School Network Density

11/07/ Student Achievement & MTL In Degree

11/07/ Conclusions  Distributed leadership—a key program goal is manifested by a tightly webbed network  School-level adoption of program principals is manifested by positioning of key individuals within the network  There may be a natural evolution of school networks that is indicative of program impact in that school

11/07/ Next Steps  Continue school-level analysis to strengthen our hypothesis about the relationship between social networks and other proximal and distal outcomes  Develop cross-school (or district-wide) networks

11/07/ Contact Information  Carl Hanssen, PhD Hanssen Consulting, LLC  MaryAnn Durland, PhD Durland Consulting