1 Meg Hargreaves, Ph.D. Mathematica Policy Research, Inc. Beverly Parsons, Ph.D. InSites Human Systems Dynamics Theory Applied to Evaluation Practice American Evaluation Association 2008

2

3

4

5 Introduction to a Systems Perspective In Evaluation This section presents: System definitions System features System characteristics Types of systems Examples of types

6 Multiple definitions: A group of interacting, interrelated, or interdependent parts forming a complex whole A configuration of parts joined together by a web of relationships The parts form a whole, which is greater than the sum of its parts Systems Definitions

7 Boundaries define who or what lies inside or outside the system Differences among the parts influence the system’s dynamics Relationships among parts, between parts and whole, and between whole and its environment, are key focus of systems System Features Systems are as much an “idea” about the real world as a physical description of it:

8 System Characteristics Common patterns, behaviors, and properties: Patterns – unorganized, organized, or organic (self-organized) Behaviors – random, simple, complicated, or complex adaptive; linear or nonlinear Properties – independent, interrelated, or interdependent relationships Scale – small to large, self-similarity across levels (fractals)

9 System Types Systems can be grouped by their level of complexity or organization: Random (no system) - unorganized Simple system - organized Complicated system – organized Complex adaptive system – organic

10 Random (Unorganized) Random, chaotic activity – no pattern Independent, unconnected parts No cause-effect relationships – constant chaos and surprise Turbulence - no equilibrium Random parts without a system No leadership - people react blindly Unknowable

11 Random System Examples War zone: Civilians caught in crossfire, random flight to escape conflict Natural disaster: At landfall or in the eye of the storm, residents react instinctively to events Leadership transitions: During changes in administration old patterns are suspended before new patterns are established

12 Simple System (Organized) Stable, static pattern Parts connected in linear ways Predictable cause-effect relationships Set equilibrium System reducible to parts and replicated Directive leadership - designed change Known knowns – answers are evident

13 Simple System Examples Baking a cake: Follow a recipe to assemble and combine ingredients into a batter that is baked at a pre-set temperature with predictable results Flu shot clinics: Nurses use consistent procedures to administer the same shots to each person, following a set protocol in assembly-line fashion

14 Complicated (Organized) Dynamic pattern of feedback loops Many interrelated parts across subsystems, levels Complex, nonlinear cause-effect relationships Feedback can stabilize equilibrium – thermostat System can be reduced to parts and replicated Multiple answers – investigate options Unknowns become known through expert analysis at multiple levels

15 Complicated System Examples Space Shuttle Challenger disintegrated (1986) when O-ring failure caused a rocket booster breach, creating flare that damaged external fuel tank, spilling fuel that exploded In large healthcare institutions, human behaviors are part of wider systems of causality, in which medical errors can occur in organizational and policy contexts that result in long (36-hour) shifts, large caseloads, and strained staff relations

16 Complex Adaptive System (Organic) Dynamical patterns – parts adapting to each other and to environment as a whole Parts are massively entangled, interdependent Parts self-organize, learn, coevolve organically Equilibrium in flux - sensitive to initial conditions System not replicable, can’t repeat past Emergent change – manage conditions of organic development and experimentation Unknown unknowns – trial and error

17 Complex Adaptive System Examples Economic system – interactions of homeowners, mortgage lenders, stock market traders, investors, federal banking institutions, and worried consumers are coevolving into global crisis and recession, despite governments’ interventions User networks (Diabetes, AA) facilitate exchange of information and advice on care for chronic conditions among participants, learning from each other

18 Background about Systems Theories This section presents: General systems theory Cybernetics – systems dynamics Complex adaptive systems Implications for evaluation

19 General Systems Theory Holistic change ideas – ancient Greeks General systems theory - von Bertalanffy (1930’s); earliest work by Bogdanov (1910) Open systems – nonrandom elements organized into interacting, interrelated components that seek to survive through interactions with environment Each system level nested in higher level (cells, organisms, families, organizations, communities, societies)

20 Implications for Evaluation The whole can enable/constrain parts and the parts can contribute to and/or challenge stability of the whole Because open systems are structured in hierarchies; useful to look one level above and one level below the ‘system in focus’ Evaluate system viability – does system have both the parts and the information and decision flows among the parts that are needed to survive?

21 Cybernetics and System Dynamics System dynamics founded by Forrester at MIT (1950’s) for electrical engineering Method for calculating and modeling how many circular, interlocking, sometimes time- delayed relationships among parts are important in shaping system-wide behavior Through negative feedback, adjustments made to keep system in balance; positive feedback used to move system in same direction, moving out of balance

22 Implications for Evaluation Assess influence of feedback loops on behavior of system’s parts and on whole Behavior of whole not only explained by behavior of parts (e.g. medical errors) Feedback loops undermine sustainability of public interventions (policy resistance) Evaluators cannot step outside social and ecological systems to observe (not value- neutral); self-reflection needed

23 Complex Adaptive Systems Key CAS writers – Weaver (1948), Simon (1962), Prigogine (1987), Stacey (1997, 2007), Zimmerman et al (2001), Eoyang (2006) CAS – many semi-independent and diverse agents, who are free to act in unpredictable ways, continually interact with each other, adapting to each other and to environment as a whole, creating system-wide patterns Key concepts – emergence, organic self- organization, co-evolution, simple rules

24 Implications for Evaluation Currently relevant evaluation criteria and measures may need to be updated as new conditions emerge Measure frequently for emerging patterns Avoid grand modeling projects for prediction; use smaller projects for ongoing experimentation and learning Also visualize system interactions as networks; look outside nested levels for system patterns, drivers, and constraints Ask what, so what, now what?

25 Three Dynamics of a Social System and its Context close to certainty far from certainty far from agreement close to agreement Unorganized dynamics (random unpatterned seemingly chaotic) organic dynamics (emerging patterns coherent but not predictable) Agreement Organized dy namics (predictable orderly controlled) C C O O N N T T E E X X T T Certainty

26 Match of Evaluation Designs to Dynamics of Social Systems and Their Context close to certainty far from certainty Certainty far from agreement close to agreement Agreement Exploratory Design Predictive Design Organic Design unorganized dynamic organized dynamic organic dynamic Initiative Renewal Design C C O O N N T T E E X X T T

27 Understanding Organic Dynamics (Activity) Divide into triads Selects one other triad member (doesn’t tell) and uninvolved person in refreshment area Stay at least two feet apart and equidistant from the other two Do this for about 1-2 minutes while trying to reach refreshments Reflect on experience

28 Case Study Introduction Do the preconference professional development offerings contribute to effective evaluation-related work of association members? If so, how?

29 Unorganized System Dynamics What is happening? What boundaries, differences, similarities, and relationships might shape how the offerings contribute to participants’ evaluation-related work?

30 Organized System Dynamics Do participants receive high-quality content that is relevant to their evaluation-related work and is delivered through high-quality instructional methods?

31 Organized System Dynamics How do the format and content of the session support or hinder participants in understanding and using the session to apply the learning from the session to their evaluation work?

32 Organic System Dynamics What patterns among participants (including the session facilitators) before and during the session are likely to affect the participants’ understanding and application of the learning to their evaluation-related work?

33 Patterns

34 Patterns

35 Patterns

36 Patterns

37 Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 10)

38 Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 27)

39 Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p.42)

40 Patterns Centers for Medicare & Medicaid Services. (2008). System and Impact Research and Technical Assistance for CMS FY2005, FY2006, and FY2007 RCSC Grants (2008). [Annual Report]. Cambridge, MA: Abt Associates, Inc. (p. 78)

41 Patterns

42 Fractals: Patterns, Patterns Everywhere In nature... Mathematical construct of iterating nonlinear equation and plotting on complex number plane—Mandelbrot Set Similar shapes at all scales—Broccoli Biological scaling gives coherence in widely diverse entities— Oak tree Scale-free networks

43 Recognizing patterns is critical: similarities, differences, and relationships that have meaning across space and time Basic values or simple rules generate diverse, but self-similar behavior across scales Naming and telling stories about dynamics in a system help reinforce and shape fractal patterns Fractals: Patterns, Patterns Everywhere

44 Fractals

45 Looking at the Dynamics as a Whole What is the overall picture of system dynamics affecting how the preconference professional development offerings contribute to effective evaluation-related activities of AEA members? Given the findings from the three system dynamics within the preconference session, how might the preconference professional development process and offerings be modified to contribute more substantially to the quality of AEA members’ evaluation-related work?