1. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Useful Tools for Integrating Systems Concepts into System Change Evaluations November 10, 2010 American Evaluation Association Professional Development Session 34 Meg Hargreaves ● Marah Moore ● Beverly Parsons 1

2. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Welcome and Introduction 2

3. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Workshop Objectives To describe a situation systemically and to understand its attributes and dynamics To describe and understand the attributes and dynamics of a systems change intervention To integrate systems concepts into the 4 phases of an evaluation: designing evaluation, collecting data, making meaning from data, and shaping practice 3

4. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Four Phases of Evaluation 4 Collect Data Make Meaning from Data Design Evaluation Shape Practice

5. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Morning Agenda Overview of systems concepts Describing a situation systemically Describing a systems change intervention Design Evaluation: a systems change approach Collect Data: selecting appropriate methods Make Meaning: data analysis and interpretation of complex data Shape Practice: using evaluation results 5

6. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Systems Concepts 6

7. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Many System Definitions A configuration of interacting, interdependent parts that are connected through a web of relationships, forming a whole that is more than the sum of its parts (Holland 1998) Systems are overlapping, nested, and networked; they have subsystems and operate within broader systems (von Bertalanffy 1955; Barabasi 2002) 7

8. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Systems Thinking A way of seeing and understanding a situation that emphasizes both the parts and the relationships among the parts rather than the parts in isolation 8

9. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Systems Boundaries Delineate what is inside/outside the system or intervention, its parts, or situation of inquiry ◦ Geographical (location) ◦ Organizational (department, unit, function) ◦ Physical (money, materials, staff) ◦ Conceptual (goals, mission, purpose, rules) ◦ Intangibles (perceptions, awareness, mental models) ◦ Natural or human-made 9

10. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Systems Interrelationships Relationships, connections, and exchanges among parts, whole, and environment (context) ◦ Social relationships, formal and informal ◦ Organizational relationships ◦ Flows of information, data, knowledge ◦ Funding flows, streams, budget authorizations ◦ Communication channels and types ◦ Collaborative partnerships ◦ Cause and effect 10

11. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Systems Perspectives System perspectives or purposes that focus the energy, attention, action of system agents System parts/agents may differ in worldviews, purposes, or agendas in a given situation Diversity in system perspectives or purposes produces tension and energy within a system (might be productive or destructive) Coherence of purpose or mission among parts can focus, shift patterns of system activity 11

12. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Describe This Situation Systemically 12

13. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Partner Exercise Pick a partner and select a situation Describe the situation systemically What are the boundaries? What are the relationships? What are key perspectives? Your partner’s turn 13

14. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Schools of Systems Theory Multiple schools of systems theory ◦ Cybernetics ◦ General systems theory ◦ Systems dynamics modeling ◦ Complexity theory ◦ Soft and critical systems ◦ Learning systems 14

15. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Early Cybernetics Early Cybernetics Early leaders include Gregory Bateson, Norbert Wiener, Warren McCulloch, Margaret Mead, and Ross Ashby Contributions ◦ Feedback and information ◦ Parallels between cognitive/human and engineered/ machine behavior Implications for evaluation

16. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Late Cybernetics Late Cybernetics Leaders include Heinz von Foerster, Stafford Beer, Humberto Maturana, Niklas Luhmann, and Paul Watzlawick Contributions ◦ Inclusion of observer and observed in same system ◦ Continuation of early cybernetics work with application to management, biology, sociology, and psychology Implications for evaluation

17. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com General Systems Theory Leaders include Ludwig von Bertalanffy, Kenneth Boulding, Geoffrey Vickers, and Howard Odum Contributions ◦ Open vs. closed systems ◦ Sum greater than parts ◦ System boundaries and webs ◦ Nested system hierarchies Implications for evaluation

18. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Systems Dynamics Leaders include Jay Forrester, Donella Meadows, and Peter Senge Contributions ◦ Reinforcing and balancing feedback ◦ Circularity (feedback loops) ◦ Stocks and flows ◦ Computer modeling of underlying dynamics of organizational, societal, and global systems ◦ Mental models and system archetypes ◦ Levels of system leverage Implications for evaluation

19. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Feedback Loops 19

20. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Complexity Theory Leaders include Ilya Prigogine, John Holland, Stuart Kauffman, and James Lovelock Contributions ◦ Based on cybernetics and general systems theory ◦ Complex adaptive systems ◦ Conditions of self-organization—far from equilibrium ◦ Irreversible past, unpredictable future ◦ Nonlinearity (small initial differences—large effects) ◦ Adaptation and co-evolution Implications for evaluation

21. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Multiple Systems Dynamics Multiple Dynamics Concurrently Exist in Systems Unorganized—random Organized—simple Organized—complicated Self-organizing—complex adaptive Select dynamics to attend to in evaluation 21

22. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Random Independent Actions Random Independent Actions 22

23. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Simple Dependent Relationships 23

24. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Complex Interdependencies 24

25. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Dynamics of a Social System and Its Context Unorganized (random) Organized (simple, complicated) Self- Organizing (complex, adaptive) Relationships High Predictability Low Predictability Perspectives High Agreement Low Agreement Context

26. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Soft and Critical Systems Leaders include C. West Churchman, Russell Ackoff, Peter Checkland, Werner Ulrich, and Michael C. Jackson Contributions ◦ Applications in management and public policy ◦ Multiple perspectives and power; boundary critique ◦ Addressing intractable problems/situations Implications for evaluation

27. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Learning Systems Systems of learning in individual practice, groups, and organizations Leaders include Kurt Lewin, Eric Trist, Chris Argyris, Donald Schon, Mary Catherine Bateson Contributions ◦ Way people learn (in organizations, primarily) and systems within which they learn ◦ Group dynamics ◦ Action research Implications for evaluation

28. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Systems Change Interventions 28

29. The Systems Iceberg 29 Events and Behaviors Patterns Structures Paradigms Conditions What is happening now? How do patterns play out over time and space? What are the drivers and deep structures? How are they related? M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com

30. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Is Systems Change? Underlying patterns and structures influence system-wide behaviors System change—shifts in patterns and paradigms/structures/conditions of the system These shifts manifest as changes in boundaries, relationships, perspectives, and dynamics over time and space These changes influence and are influenced by changes in events and behaviors 30

31. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Is the Nature of the Intervention? What is the intervention’s governance—its funding, management, organizational structure, and implementation? What is the intervention’s theory of change—its causal mechanisms and pathways of change related to deep structures, patterns, and events and behaviors? What are the intervention’s intended outcomes—how many, how focused, and at what levels? 31

32. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Intervention Theory of Change System Intervention theory of change ◦ How an intervention plans to trigger the system change process (Funnell and Rogers 2010) Some interventions focus on changing complex systems Some interventions focus on changing individuals operating within complex systems Both approaches benefit from a theory of change (TOC) that attends to different aspects of the system 32

33. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Example: City Integration Initiative 33

34. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Is the Situation? Describe the situation—the whole, parts, and boundaries Describe the dynamics of the situation’s relationships (where are dynamics random or unknown, simple, complicated, or complex) Describe the diversity of purposes or perspectives within the situation How do deep structures, patterns, and events and behaviors factor into the situation? 34

35. Current Situation: Independent Systems Source: Mount Auburn Associates and Mathematica Policy Research Current Situation: Independent Systems M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.com 35mhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com

36. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Is the Intervention? What is the intervention’s governance—its funding, management, organizational structure, and implementation? What is the intervention’s theory of change—its causal mechanisms and pathways of change? What are the intervention’s intended outcomes—how many, how focused, and at what levels? How does the intervention attend to deep structures, patterns, and events and behaviors? 36

37. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com System Change: Integrated System Source: Mount Auburn Associates and Mathematica Policy Research

38. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Is the Evaluation’s Design? Who are the evaluation’s users? What are the evaluation’s purposes? (developmental, formative, monitoring, or summative) What are the evaluation’s research questions? What are the evaluation’s methods? How will the data be analyzed and interpreted? How does the evaluation attend to deep structures, patterns, and events and behaviors? 38

39. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Is the Evaluation’s Purpose? Who are the evaluation’s users? The national client, initiative’s funders, local grantees, and other stakeholders What are the evaluation’s purposes? The evaluation will focus on the intervention’s development and early implementation, providing formative feedback at multiple levels 39

40. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Are the Evaluation Questions? What systems changes are occurring? ◦ How have the system’s boundaries been expanded or reconfigured? ◦ Geographic boundaries, stakeholder groups, discipline areas ◦ Have stakeholders’ perspectives changed? ◦ Orientation of problem, understanding of challenges and opportunities, commitment to project, charge attitudes ◦ Have intensity, types of relationships changed? ◦ Level of coordination, formality of linkages, flow of resources, closeness of ties, diversity of actors 40

41. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Are the Evaluation Questions? What is the role of the client in influencing systemic change and benefits for low-income people? Integration of financing and programmatic strategies, how blended funds are structured, introduction of new financial intermediary, client consultation and technical assistance How has the community’s context interacted with and influenced systemic change and benefits for low-income people? Economic conditions, racial dynamics, political environment, community norms, cultural norms 41

42. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Are the Evaluation Questions? How are site-specific strategies, activities, and structures influencing systemic change and benefits to low-income people? Site strategies and projects Initiative staffing, management, governance structure Common agenda Capacity and structure of financing partners Leadership of stakeholders Public sector role and leadership 42

43. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Evaluation Methods for Unknown Dynamics Case studies, interviews, focus groups, observation of activities Mapping of community assets Environmental scans Needs assessments Situational analyses 43

44. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Evaluation Methods for Simple Dynamics Randomized experiments Quasi-experimental comparisons Regression discontinuity analyses Hierarchical linear modeling Performance measurement, monitoring Program audits, inspections 44

45. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Evaluation Methods for Complicated Dynamics Computer simulation models of stocks, flows, feedback, and causal loops Social network analysis Pre-post measurements of change Interrupted time series analysis Comparative measurement and monitoring 45

46. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Evaluation Methods for Complex Dynamics GIS spatial analysis Agent-based modeling Time trend analysis Observational or cross-sectional studies Retrospective analysis Adaptive learning measurement systems 46

47. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com What Are the Evaluation’s Methods? Network analysis—social network surveys and ecosystem mapping of sites Key informant interviews—phone interviews and periodic calls with site-based informants Site visits—focus groups, on-site interviews Observation—of program activities, events Document review—program documents, products Secondary data—environmental indicators 47

48. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Data Analysis and Interpretation Network survey—two rounds Follow-up site visits and interviews Tracking of grantee-level policy changes Tracking indicators of grantee outputs, project-specific, city-level, and resident outcomes Spatial analysis of neighborhood, city, and region change 48

49. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Group Exercise Select a systems change initiative and describe: The dynamics of the situation The dynamics of the intervention The evaluation’s design—users, purpose, questions, methods, and analyses How does the design address each level of the iceberg? How do systems concepts and dynamics change the design? 49

50. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Summary of Morning Session 50

51. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Setting the Stage for the Afternoon 51

52. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com The Afternoon Will: Build on concepts from the morning Look at paradigm shifts in systems change Explore the Iceberg Present three tools that can help with your work We will begin promptly at 12:00 Enjoy your lunch!!! 52

53. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Afternoon Agenda Using another evaluation example, explore: ◦ Paradigm shifts in evaluation ◦ Building a theory of change around systemic points of influence ◦ Assessing patterns that connect deep structures with events and behaviors Three new tools Group work Close 53

54. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Example: Quality Improvement Center for Early Childhood (QIC-EC) 54

55. Revisiting the Systems Iceberg Events and Behaviors Patterns Structures Paradigms Conditions What is happening now? How do patterns play out over time and space? What are the drivers and deep structures? How are they related? 55 M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com

56. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Situation Many interventions in place, but abuse and neglect rates not improving Multiple partners related to child abuse and neglect (CAN) prevention (programs, communities, policymakers, researchers, etc.) Affects families across socioeconomic spectrum Social norms stigmatize families who are thought to be at risk Families are isolated socially and isolated from the larger system Predominantly focuses on reducing risk and individual behavior change rather than more systems approach No underlying paradigm driving policy—policy often not related to CAN paradigms at all! 56

57. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Intervention Goals: Decrease in child abuse and neglect through a systemic paradigm shift from focusing on risk to building PROTECTIVE FACTORS in families and communities Move to a focus on identifying and understanding patterns across the systems and the paradigms, structures, and conditions that reinforce and/or support change in the patterns 57

58. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Intervention Strategies: Research project focused on creating and integrating new knowledge about using Protective Factors in CAN prevention Overall project focused on a systemic approach across multiple levels Four research demonstration sites ◦ Different approaches to CAN prevention ◦ Different levels of systems involvement and integration ◦ Varying degrees of integrating protective factors into interventions 58

59. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Protective Factors as Paradigm Shift Parental resilience Social connections Concrete support in time of need Knowledge of parenting and child development Social and emotional competence of child 59

60. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Intervention The Theory of Change identifies “Points of Influence” at multiple levels: Caregiver-Child (individual level) Social Support (relationship level) Neighborhood (community level) Organizational Programs (community level) Policy and Social Norms (“systems” level) 60

61. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Intervention Points of Influence in the Theory of Change are subsystems that: Have their own coherence Interact with other subsystems Change in different ways or rates Past research shows system impact 61

62. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Intervention At each level, the TOC identifies: Baseline of fundamentals and system dynamics Testing applications of new fundamentals and system dynamics Tipping point to new fundamentals and system dynamics balance Sustaining an adaptive balance of new fundamentals and system dynamics in a shifting context 62

63. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Intervention A tool for understanding the paradigms/structures/conditions that influence events and behaviors 63

64. Theory of Change in Paradigms, Structures, and Conditions of Complex Systems Example from Cross-Site Evaluation of Quality Improvement Center on Early Childhood M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.com 64mhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com

65. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Group Activity (20 minutes) Use the example you worked on this morning Identify the systemic points of influence (paradigms/structures/conditions) Identify what the “sustained adaptive balance” would look like for these points of influence 65

66. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Four Phases of Evaluation 66 Figure 1. Phases of Evaluation Collect Data Make Meaning from Data Design Evaluation Shape Practice 66

67. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Evaluation Design 67

68. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Evaluation Design Two levels of Evaluation: Individual Research Demonstration Sites Cross-Site These levels are separate but interrelated: Both shared and separate methods Results from each expected to inform the other Ongoing communication 68

69. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com “Insider-Outsider” Evaluation: A Creative Tension The extent to which evaluation process and evaluation results explicitly contribute to an intervention varies widely (e.g., action research vs. “pure” research) For the QIC-EC, this is a tension In complex systems tensions do not always need to be resolved! 69

70. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design Purpose: Increase understanding of the movement toward and results of a paradigm shift in CAN prevention Identify interrelationships between “points of influence” in development and implementation of CAN prevention interventions Identify how protective factors are built with families and within communities and programs Identify the role of protective factors in CAN prevention 70

71. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design Users: QIC-EC Leadership Team QIC-EC Learning Network Four research demonstration projects The “field” 71

72. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design Overall Approach: Use a systemic TOC to design the evaluation Move from simply evaluating the efficacy/effectiveness of interventions in changing events and behaviors to evaluating: ◦ The effect of systemic patterns and underlying structures on CAN ◦ Effectiveness of efforts to change systemic patterns and underlying structures Test specific models at individual sites and then layer additional evaluation methods on top of that for Cross-Site 72

73. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design Methods: Randomized Controlled Trial (RCT) and Quasi-Experimental (QE) (at project level) Cross-site incorporates data from project-level RCTs/Q-E Structural Equation Modeling to integrate data from multiple levels 73

74. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design Methods (cont.): Mixed methods—both in tandem and in parallel ◦ Various quantitative measures for caregiver/child- level outcomes ◦ Quantitative and qualitative measures at other levels ◦ Qualitative exploration of intervention implementation leading to quantitative measures ◦ Social Network Mapping to understand relationships at the caregiver/child level, the community level, and the program level 74

75. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design Using the Theory of Change in Paradigms, Structures, and Conditions of Complex Systems tool to identify systems outcomes for points of influence 75

76. Theory of Change in Paradigms, Structures, and Conditions of Complex Systems Example from Cross-Site Evaluation of Quality Improvement Center on Early Childhood M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.com 76mhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com

77. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design “Sustained adaptive balance” for the QIC-EC points of influence 77

78. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Group Activity (15 minutes) Use the same example Identify what the “sustained adaptive balance” would look like for the points of influence 78

79. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design A tool for understanding the patterns in how paradigms/structures/conditions influence events and behaviors 79

80. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com The 7 C’s Framework Understanding Patterns 7 Cs Framework 80

81. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Cross-Site Evaluation Design Paradigm Shift as a Social Movement 81

82. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Change Agent QIC-EC Social Change Movement Loosely organized, collective systems change effort by people or organizations with a common purpose and solidarity in sustained interactions with the systems they are focused on changing Systems Change Changes in Systemic Points of Influence Impact Reduction of CAN 82 M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com

83. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Paradigm Shift and Social Movements Feeding the Social Movement: At individual Sites ◦ Collaborations ◦ Sharing of process and outcomes ◦ Learning through research and evaluation Across the “field” ◦ New knowledge dissemination through QIC-EC ◦ Broad dissemination of results ◦ Informal dissemination 83

84. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Group Activity (30 minutes) Continue with the same example Identify how you might use the 7 Cs to understand the relevant patterns in the movement toward a “sustained adaptive balance” for the “points of influence” Pick one example that would look at the change agent level; one for the social movement; and one for the systems change How does this help you to understand change in events and behaviors? 84

85. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Small Group Discussion and Report Out One important thing that you learned this afternoon using the two tools for evaluation design One thing that you will do differently in your practice One question with which you are leaving 85

86. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com How are you integrating systems thinking into your approach? 86

87. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Implications for Data Collection How integrated is the evaluation with the intervention? How are stakeholders from different levels involved? What is the timing and frequency of data collection, based on the TOC? 87

88. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Implications for Making Meaning How integrated is the evaluation with the intervention? How are stakeholders from different levels involved? How are different needs/purposes balanced? 88

89. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com QIC-EC: Implications for Shaping Practice How integrated is the evaluation with the intervention? How are stakeholders from different levels involved? Who is trying to shape whose practice? 89

90. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.comZIPPER Z = ZOOMING IN AND OUT OF EVALUAND AND ITS ENVIRONMENT I = INTERCONNECTING THE PARTS P = PLUNGING INTO PARADIGMS, STRUCTURES, AND CONDITIONS P = PERCEIVING PATTERNS E = ENVISIONING ENERGY R = RECOGNIZING RESULTS 90

91. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Wrap-Up Discussion and Session Evaluation 91

92. M.Hargreaves, mhargreaves@mathematica-mpr.com; M.Moore, marah@i2i-institute.com; P.Parsons, bparsons@insites.commhargreaves@mathematica-mpr.commarah@i2i-institute.combparsons@insites.com Contact Information Margaret Hargreaves ◦ mhargreaves@mathematica-mpr.com ◦ 617-301-8994 Marah Moore ◦ marah@i2i-institute.com ◦ 575-758-7513 Beverly Parsons ◦ bparsons@insites.org ◦ www.insites.org ◦ 360-638-1442; 970-226-1003 92