Technical Assistance Partnerships in Action: Lessons Learned from Evaluation Pat Mueller, EEC David Merves, EEC Vitaliy Shyyan, NCEO OSEP Project Directors’ Conference, August 2016

Purpose of Presentation To SHARE… Evaluation Design Logic Model Instrumentation Case Studies Reporting Formats Discuss Focus Questions

National Center on educational Outcomes Mission Statement To support the development of inclusive assessment in a rapidly changing state (and consortium) assessment system environment in order to promote improved educational results for students with disabilities. Collaboration, influence, and outcome mapping Measure numbers and strengths of partnerships (e.g., frequency of contacts) Indicate collaborative TA activities and collaborative activities not specific to TA (e.g., collaborator participates in a jointly sponsored event vs. collaborating on a TA activity) Track state TA and identify when/who collaborated on the TA Collect data from clients, state/consortium team leaders, case studies Displaying the data – Network Analysis

EvALUATION DESIGN Evaluation Questions Development of Theoretical Model Evaluation Methodology Triangulation of Data Sets Collaboration, influence, and outcome mapping Measure numbers and strengths of partnerships (e.g., frequency of contacts) Indicate collaborative TA activities and collaborative activities not specific to TA (e.g., collaborator participates in a jointly sponsored event vs. collaborating on a TA activity) Track state TA and identify when/who collaborated on the TA Collect data from clients, state/consortium team leaders, case studies Displaying the data – Network Analysis

NCEO Partner collaborative Logic model Employment of common strategies and language Development of common products, tools, and services Non-duplication of efforts Cost efficiency Handout the Logic Model with NCEO & EEC logos

Instrumentation Internal TA tracking system TA Documentation Where in the World is NCEO? Website linkage analysis Surveys with key partners and clients Interviews with key partners and clients

Case studIES Consortium Topical Focus State Collaborative Multi-Case Report

Case studY Methodology Selected Success Case Method (SCM) that aligned with goals & logic model: What is happening? What results are being achieved? What is the value of the results? How can the initiative be improved? Conducted interviews with 3 partnership types: Consortium Topical Focus State Collaborative Written reports, briefs, & comparative study describe collaborative work: Products Benefits of process Results Impact

Case study findings PROCESS RESULTS IMPACT Understanding of deliverables Expert facilitation, leadership and responsiveness Solicitation and openness to feedback RESULTS Relationships Collaborative Presence Collaborative TA IMPACT Effective Work with Partners Enhanced Assessment Policies and Practices

Reporting formats Standard evaluation reports Infographics Have hand out with logic model on 1 side and infographic on the other side

Discussion questions What methods do Centers employ to evaluate their collaboration efforts, outcomes, and impacts? What types of collaboration assist Centers in developing high-quality products, tools, and services? What types of reporting formats do Centers use to communicate the efforts and effects of their work?

Contact information Pat Mueller & David Merves Evergreen Evaluation & Consulting, Inc. pat@eeecvt.com; david@eecvt.com Vitaliy Shyyan National Center on Educational Outcomes shyya001@umn.edu