Measuring Collaboration at state, regional, and local levels SIG Network Presentation April 24, 2013 Measuring Collaboration at state, regional, and local levels Pam William, Missouri Department of Elementary & Secondary Education Pattie Noonan, University of Kansas Amy Gaumer Erickson, University of Kansas

Collaboration "In the long history of humankind (and animal kind, too) those who learned to collaborate and improvise most effectively have prevailed." Charles Darwin “Collaboration is an unnatural act between two non-consenting adults.” Unknown

Levels of Collaboration (Frey, Lohmeier, Lee & Tollefson, 2006) Collaboration Independent agencies function as one entity or system. Frequent communication with mutual trust. Consensus reached on all decisions. Share ideas and resources frequently. Frequent and prioritized communication. All members have vote in decision-making. Coalition Sharing information and resources on a regular basis, defined roles, frequent communication and some shared decision-making. Coordination Cooperation Providing information to each other, somewhat defined roles, independent decision-making and formal communication. Networking Characterized by awareness of an organization and low levels of communication for referral purposes.

Measuring Collaboration STATE Missouri Department of Elementary & Secondary Education (DESE) Survey REGIONAL Regional Professional Development Centers (RPDCs) Survey DISTRICT/SCHOOL School Implementation Scale - District & School Results SCHOOL TEAMS Team Functioning Scale – School & Team Results INTERAGENCY Missouri Interagency Transition Team (MITT) Social Network Analysis & Focus Group

State Department Collaboration Annual Survey Involvement in the SPDG Change in knowledge Perceived change in collaboration Current level of collaboration through Social Networking Analysis

DESE Results 2011-12 Collaboration Coalition Coordinator Cooperation Networking No interaction At all Response Total Mean Office of Quality Schools 1 2 3 2.33 Office of Special Education 5.00 Office of Early and Extended Learning 5 1.20 Office of Adult Learning and Rehabilitation Services 4 0.20 Office of Educator Quality 1.60 Office of College and Career Readiness 1.40 Office of Data System Management 2.80 Regional Professional Development Centers 3.80 MIM Schools 1.00 MIM Districts Parent Centers (e.g., MPACT) 1.50 Representatives of Higher Education (e.g., MU, UMKC, KU evaluators)

Regional Professional Development Center Consultants Collaboration

RPDC Results 2011-12 Collaboration Coalition Coordinator Cooperation Networking No interaction At all Response Total Mean Department of Elementary and Secondary Education 5 4 6 24 15 60 2.03 DESE Office of Special Education 3 12 8 27 1.45 DESE Office of Adult Learning and Rehabilitation Services 1 2 7 49 59 0.25 DESE Office of College and Career Readiness 26 1.07 Other Regional Professional Development Centers 13 11 58 2.95 MIM Schools 1.58 MIM Districts 10 57 1.44 Parent Centers (e.g., MPACT) 53 0.12 Representatives of Higher Education (e.g., MU, UMKC, KU evaluators) 14 1.27

District/School Collaboration School Implementation Scale Evaluates fidelity of implementation and the extent of implementation across a school and/or district. Appeals to teachers, administrators, other certified staff, and noncertified staff. All items are evidence-based and result in robust data. Addresses tiered supports in both academics and behavior. Is sensitive to change. Is quick and easy to complete online. Results in a high response rate. Supports school-level action planning and data-based decision-making. Correlates closely with gains in academic achievement for students with disabilities. Has high reliability (overall scale Alpha = .961) Applicable to elementary, middle, and high schools.

One School’s Collaborative Environment Results Percent of Staff

One School’s Family Engagement Indicators of Collaboration Percent of Staff

School Team Collaboration Team Functioning Scale 17 items Reports developed for schools and collaborative teacher teams Full Scale alpha: 0.948 Structure Subscale alpha: 0.830 Communication Subscale alpha: 0.939 Focus Subscale alpha: 0.888

Social Network Analysis Missouri Interagency Transition Team (MITT)

Social Networking Analysis What is SNA? Social network is to depict interconnections or associations among a set of actors or nodes. A core theoretical problem in network analysis is to detect and represent the relation structure and explain the variation in structural relations and their consequences. Some Terminologies in SNA --Actor (Node): can be persons, groups, organizations, etc. --Relation: defined as a specific kind of contact, connection, or tie between a pair of actors. --Density: Average strength of all ties across all relations --Nodal Indegree: the number of lines received by an actor from others --Nodal Outdegree: the number of lines sent by an actor to others

General Description of SNA Social network analysis Relations Matters! ---Individual characteristics only explain half the story. ---Hypotheses can be phrased in terms of correlations between relations So: ---The data of SNA is about relations ---It uses different statistical methods Binary Socialmatrix Friendship Tom Sue Jim Bob --- 1 Valued Socialmatrix Sender/Initiator Reciever Collaboration Tom Sue Jim Bob --- 4 5 1 3

Social Network Analysis Methods Ten Organizations comprised the network. The intensity or the strength of interaction between pairs of actors is illustrated by a 5-point scale questionnaire with 0 represents No Interaction At All, 1 Networking, 2 Cooperation, 3 Coordination, 4 Coalition, and 5 Collaboration. Social network analysis (SNA) was conducted with the program Ucinet 6. By convention, for directed relations, the actors in the rows are initiators or senders and the actors in the columns are terminators or receivers of the relations, therefore the descriptive analysis were conducted from both senders or receivers perspectives. Density was tested both before and after the training

Measuring Collaboration (Frey, Lohmeier, Lee & Tollefson, 2006) Collaboration Independent agencies function as one entity or system. Frequent communication with mutual trust. Consensus reached on all decisions. Share ideas and resources frequently. Frequent and prioritized communication. All members have vote in decision-making. Coalition Sharing information and resources on a regular basis, defined roles, frequent communication and some shared decision-making. Coordination Cooperation Providing information to each other, somewhat defined roles, independent decision-making and formal communication. Networking Characterized by awareness of an organization and low levels of communication for referral purposes.

Results Example--PTI receives an average tie strength of 1 (networking) before training and 2.05 (cooperation) after training

Results The density after sustained implementation is 2.413, while the density in early implementation is 1.331. The difference between the two is 1.082. The p value is less than 0.001, indicating it is significant at 0.001 level.

Tie Strength in Early Implementation

Tie Strength after Sustained Implementation

Side-by-side

Focus Group

Focus Group Method Topics: 90 minutes during the group’s regular meeting 10 participants Recorded and transcribed Trustworthiness through audits and code-recode cycles Topics: the origins, structure, and activities of the state transition team collaborative processes and experiences

Focus Group Results Collaborative Characteristics Variety of Partnerships Relationships Time Together Shared Vision Shared Leadership Collaborative Activities Site Visits Joint Planning Joint Training Sharing Information

Variety of Partnerships State Department of Education Vocational Rehabilitation Department of Corrections Department of Mental Health Parent Training & Information Center Partnerships led to an “increased awareness that there are opportunities to collaborate on projects and initiatives that they might not have thought about before.”

Relationships “Trust…makes someone a lot more likely to invest time and effort in something. We’ve all been on those committees where you know so-and-so’s never going to follow through…Well I don’t think that’s… the case with this group [because] we can count on each other to follow through with what we said we were going to do, and luckily and through our hard work, we’ve seen some results.” Longevity of collaboration: not tied to a single initiative “It…goes back to the old saying [that] it really does take a village to a raise a child, and…when it comes to transition, I think [the state transition team] is a village.”

Time Together Site visits, conferences, training events, planning activities “We keep showing up. We keep coming back to the table, and I think that’s one of the key things, because you can have partners fade out easily. That’s probably been a strength: longevity.”

Shared Vision Annual strategic planning “Ultimately our purpose is to find ways to improve student outcomes for graduation and dropout and employability and transition planning.”

Shared Leadership “Everybody feels empowered to jump in and commit to whatever they feel they can bring to the table.” “Brainstorming and voting” to reach consensus “When local teams see a state team that’s working together, it can be very inspirational to them…We’re modeling.”

Site Visits Vocational training centers, career and technical education schools, model transition programs, women’s prison “That was very eye-opening because it gave us the opportunity to see different pieces [which helped us see the] whole piece and how everybody fits into the process.”

Joint Planning Implementing statewide online career exploration tool Collaboration with the National Dropout Prevention Center for Students with Disabilities (NDPC-SD) Targeting state needs through reviewing multiple agencies’ data during annual retreats

Joint Training Attending conferences together Inviting national speakers to joint trainings Inviting staff from multiple agencies to trainings and state conferences Multiple agencies presenting to community transition teams “We all look at everything from our perspective [of] what we do for transition, and we don’t realize that there are whole parallel groups of people out there who are also working to accomplish the same thing…This has allowed us to say, ‘Okay, you’re doing that. Maybe we can do a joint training or maybe there are things we can share…,’ which makes us all stronger.”

Sharing Information “I can point people in school districts or parents to other agencies, because I know more about what they do now than I did before and hopefully help them make good connections for their kids as well. The group “has been very, very helpful…for passing on what we’ve learned to people in the field...and start doing the work at the local level.”