Evaluating the Impact of a System- Level Intervention Using a Developmental Evaluation Approach S. Elizabeth McGee, MA Evangeline Danseco, PhD Kyle Ferguson Purnima Sundar, PhD American Evaluation Association Annual Conference, October 23, 2012, Minneapolis, Minnesota

Presentation Overview 1. Context 2. The Task (challenges and opportunities) 3. Understanding our systems intervention through systems thinking and approaches 4. Why System Evaluations made sense 5. Our Approach 6. Limitations and Future Considerations

The Task

The Context Ontario, Canada Child and Youth Mental Health sector Ontario Centre of Excellence for Child and Youth Mental Health Six-Year Strategic Plan Evaluation Framework

What is the Centre? Vision Optimal mental health and well-being for children and youth. Mission We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth, and their families and caregivers. 5

Strategic Goals Six year strategic plan and it’s three key strategic priorities: Goal 1 Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes Goal 2 Build and develop collaborative partnerships to sustain capacity within mental health services Goal 3 Be a true learning organization and lead by example 6

Key Centre programs Information products and tools Support Services Training To improve the implementation, evaluation and sustainability of Evidence Informed Practices

The Challenge, the opportunity

Need for Evaluation Information Program Improvements Accountability Understanding our broader impact

The Centre’s work, as a System Intervention Using System Interventions System interventions seek to change system- wide patterns of behavior among actors by changing underlying system dynamics, structures, and conditions (Wheatley 2001; Eoyang 2007).

The Challenges The Centre’s unique position in the sector Capacity-building involves collective behavior, evaluation techniques are less widespread. Changes in organizational behavior may not occur for several years. Measuring changes in organizational process and decision-making are challenging.

Understanding our systems intervention through systems thinking and approaches

Systems Theory and Evaluation Planning Systems Thinking Systems thinking provides a perspective that emphasizes the patterns of interrelationships between parts and the whole rather than the parts in isolation (Trochim et al. 2006; Parsons 2007; Hargreaves and Parsons 2009).

Simple, Complicated or Complex? fixed, static, as well as linear, direct cause and effect relationship between system parts. Simple Systems Organization Leaders plan and coordinate the activities of multiple parts -Self-correcting feedback loops between parts help maintain system equilibrium by counterbalancing and neutralizing a change in the system Complicated Systems Organization Characterized by massively entangled webs of relationships, from which unpredicted outcomes emerge through the self-organizing interactions of many parts or actors within and across system levels Adaptive, learn and coevolve as they interact with one another and respond to changes in their environment, inherently unstable Complex Systems Organization

Components of Complexity Components of our system intervention Complexity Factors 1. Child and Youth Mental Health sector funding changes system transformation intersections with other sectors (health, addictions, etc.) evolving and changing health trends 2. Centre of Excellence (i.e. our programs, services, resources) collaborating with the Ministry and Regional Offices. changing programs and resources to meet needs of sector/clients new evidence/best practices 3. Clients (i.e. child and youth mental health agencies/programs) 140 core agencies through the province child welfare, hospitals, community health centres, faith-based centres rural vs. urban, aboriginal and franaphone populations, etc.

Centre’s Theory of Change model Improved EIP evaluation and implementation uptake Stakeholder engagement and collaboration Information Dissemination Knowledge Exchange Education, Training and consultation Relationship-building and coaching Improved child and youth mental health outcomes

Why systems evaluations made sense

What we need to measure

Our Approach

Our Evaluation Context Our Strategic Plan (six-year) Strategic Goals Associated Outcomes

The context in which we are working in, 1. Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes 2. Build and develop collaborative partnerships to sustain capacity within mental health services 3. Be a true learning organization and lead by example Improved mental health outcomes for children and youth in Ontario Enhanced capacity of the Centre’s as a ‘Learning Organization’ Enhanced capacity of the Centre to reach key stakeholders and enhanced partnerships among stakeholders Enhanced partnerships and collaborations among stakeholders Improved relevance, effectiveness and usefulness of Centre programs and services Enhanced stakeholder knowledge on evidence-informed practices, improved stakeholder skills and enhanced organizational capacity Improved evidence-informed practices and growth as learning organizations Increase in and improved engagement with primary stakeholders

Our Approach to the Framework Ensuring all Centre programs fall within strategic goals.

Linking Outcome Maps to Strategic Outcomes Consultations - Supporting activities via teleconferences, site visits, communications Consultations - Supporting activities via teleconferences, site visits, communications Learning resources Webinars, measures database, toolkits, online learning modules, methods mini-kits Learning resources Webinars, measures database, toolkits, online learning modules, methods mini-kits Enhanced awareness of the value of evaluation Increased knowledge of evaluation practices in evidence-informed programs Increased agency buy-in around evaluation Increased agency buy-in around evidence informed programs Increased program evaluation skills Increased internal capacity to support evaluation systems and structures Increased use of evaluation to inform practices/program improvements using an organizational learning approach Strategic Core Outcome: Enhanced understanding of what is meant by “evidence” among service providers Activities Program Outcomes Link to Strategic Goal Strategic Core Outcome: Enhanced readiness among agencies to implement evidence-informed practices

How do our programs align? Strategic Goal #1 Implementation Support Service Mental Health Literary Youth and Family Engagement Evaluation Support Service Evidence-Insight Dare to Dream Program Strategic Goal #2 Regional Partners Initiative eMentalHealth.ca Knowledge Exchange Initiative Emerging Leaders Strategic Goal #3 Staff Development Improved Financial Processes Evaluation and Performance Measurement Improve Staff Retention and Job Satisfaction

Our Approach To Measuring Impact Developing high-level indicators (and sub- indicators) to measure the impact of our organization’s work.

Strategic Goals and Outcomes 1. Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes 2. Build and develop collaborative partnerships to sustain capacity within mental health services 3. Be a true learning organization and lead by example Improved mental health outcomes for children and youth in Ontario Enhanced capacity of the Centre’s as a ‘Learning Organization’ Enhanced capacity of the Centre to reach key stakeholders and enhanced partnerships among stakeholders Enhanced partnerships and collaborations among stakeholders Improved relevance, effectiveness and usefulness of Centre programs and services Enhanced stakeholder knowledge on evidence-informed practices, improved stakeholder skills and enhanced organizational capacity Improved evidence-informed practices and growth as learning organizations Increase in and improved engagement with primary stakeholders

Impact Areas and Indicators 3. Be a true learning organization and lead by example 3. Be a true learning organization and lead by example 2. Build and develop collaborative partnerships to sustain capacity within mental health services 1. Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes 1. Demonstrated leadership 2. Reach 4. Use and Impact of Knowledge 5. Relevance, Effectiveness and Usefulness 6. Sustainability 3. Engagement Sub- Indicators

What we need to measure

Process Evaluation How do Data Feedback Loops work at the Centre? Program-specific quantitative and qualitative data is aggregated multiple times a year (and annually). Managers and teams analyze data, look for trends, and implement changes based on stakeholder feedback. Larger trends across multiple programs are discussed with management and considered in Operations Plan/Strategic Planning (in development).

Limitations and Future Considerations

Limitations One of the major limitations of our approach was the inability to make causal inferences regarding the influences of system level activities on system level outcomes

Moving forward,

For more information S. Elizabeth McGee, MA Research Associate ext 3313 Evangeline Danseco, PhD Director of Evaluation and Research ext 3319