1. Maximizing Evaluation Impact by Maximizing Methods: Social Network Analysis Combined with Traditional Methods for Measuring Collaboration Carl Hanssen, PhD & MaryAnn Durland, PhD American Evaluation Association Baltimore, MD November 7, 2007

2. 11/07/20072 Agenda  Social Network Analysis: The Method  SNA Results and Interpretation  Next Steps

3. 11/07/20073 SNA Methodology  Network Analysis is the study of the relationships formed by the interaction or links between components in a “set”.  MMP sets are schools  The components are individuals: Faculty, both math and non math MTL (School level Math Teacher Leaders) MTS (District level Math Teacher Specialists)  Relationship is communication about math

4. 11/07/20074 Measures  Indegree – popularity  Density – how “thick”, how much, out of potential

5. 11/07/20075 Role in Evaluation  How much does the communication structure actually fit the theory and the design of the project  Can the structure be correlated with other measures of implementation and impact? Activities Proximal measures

6. 11/07/20076 MMP Evaluation Logic Model Student Achievement Teacher Content & Pedagogical Knowledge Math Faculty Involvement Learning Team Effort School Buy-in Teacher Involvement New Courses District Buy-in MPA Ownership MATC Buy-In UWM Buy-In Classroom Practice MMP Activities Proximal Outcomes Distal Outcomes

7. 11/07/20077 MMP Report Card Indicators  19 indicators in 7 domains derived from in-school data collection, online surveys, and MPS data 1. MTS Assessment 2. Collaboration 3. Learning Teams 4. Classroom Practice 5. Professional Development 6. Teacher MKT 7. Student Achievement

8. 11/07/20078 SNA In Context: Evaluation Results Student Achievement Teacher Content & Pedagogical Knowledge Learning Team Effort School Buy-in Teacher Involvement Classroom Practice WKCE Mean % Proficient = 44% Overall rating = 3.5 Gap MTL v. other teacher =.2 Teacher Engagement = 3.2 Overall IRT = -0.34 Algebra IRT = -0.18 Team Functioning = 3.5 MMP Principles = 3.6 LT Quality = 3.1 PD Hrs. = 17.8 Facilitation Hrs. = 1.0 PD Quality = 3.1 Network density = 6.7% / School density = 17.6% MTL Role = 13.8 / MTS Role = 5.3 SR MTL Engagement = 4.4 / MTS Quality = 3.0 MTS Assessment = 38.3 of 55

9. 11/07/20079 Data Collection  Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics  Statistical analysis focused on 1. Network and in-school density 2. Importance of MTL and MTS

10. 11/07/200710 MMP Impact Continuum LowHigh Loose Web MTL Not Central Few Links to MTL MTS Outside Few Links to MTS Tight Web MTL Central Many Links to MTL MTS Inside Many Links to MTS

11. 11/07/200711 Low

12. 11/07/200712 Medium

13. 11/07/200713 High

14. 11/07/200714 Student Achievement & In-School Network Density

15. 11/07/200715 Student Achievement & MTL In Degree

16. 11/07/200716 Conclusions  Distributed leadership—a key program goal is manifested by a tightly webbed network  School-level adoption of program principals is manifested by positioning of key individuals within the network  There may be a natural evolution of school networks that is indicative of program impact in that school

17. 11/07/200717 Next Steps  Continue school-level analysis to strengthen our hypothesis about the relationship between social networks and other proximal and distal outcomes  Develop cross-school (or district-wide) networks

18. 11/07/200718 Contact Information  Carl Hanssen, PhD Hanssen Consulting, LLC carlh@hanssenconsulting.com 616-808-2867 carlh@hanssenconsulting.com  MaryAnn Durland, PhD Durland Consulting mdurland@durlandconsulting.com 630-650-9944 mdurland@durlandconsulting.com