1. Evaluating Complex System Interventions Evaluation Professional Development Workshop Beverly Parsons and Meg Hargreaves

2. What is a System?
A group of interacting, interrelated, and interdependent elements forming a complex whole
A configuration of parts connected and joined together by a web of relationships
The whole is different from, and greater than, the sum of its parts

3. Parts of an Elephant
One way to see this system – reduce it to its parts, when in reality the sum is more than and different from the sum of its parts – refers to a Hindustani tale in which people are blindfolded and asked to describe what they touch. All are right, but they misidentify the whole – an elephant

4. Systems Thinking
A way of understanding reality that emphasizes the relationships among a system’s parts, rather than the parts themselves.
Concerned about interrelationships among parts and their relationship to a functioning whole
Sees underlying patterns and structures

5. Foundations of Systems Theory
Cybernetics: system feedback, information; differences (that make a difference); human – machine analogy; inclusion of the observer and the observed in the system
General systems theory: open systems; system integrity; nested system hierarchy, boundaries, webs, emergence (sum greater than parts)
Recent theories developed starting in the 1920’s to describe the interrelatedness of organisms in ecosystems
Cybernetics –early (Gregory Bateson, Norbert Weier, Warren McMulloch, Margaret Mead, Ross Ashby, Talcott Parsons
General systems theory – Ludwig von Bertalanffy, Kenneth Boulding, Geoffry Vickers, Howard Odum and Fritjof Capra
Late cybernetics – Heiz von Foerster, Stafford Beer, Humbarto Maturana, Niklas Luhmann, Paul Watzilawick

6. Systems Theories
Soft and critical systems: human systems - multiple perspectives, power issues, intractable problems without simple solutions
Systems dynamics: systems have reinforcing and balancing feedback loops, circularity, system archetypes, mental models, unintended consequences
Soft and critical systems – C. West Churchman, Russell Ackoff, Peter Checkland, Werner Ulrich, Michael C. Jackson
System Dynamics – Jay Forrester, Donnella Meadows, Peter Senge

7. More Systems Theories
Complexity theory: complex adaptive systems; semi-independent, interacting agents; self-organization; emergence; nonlinearity; co-evolution; past is irreversible; future is unpredictable
Learning systems: the way that people learn and the systems in which they learn
Complexity theory – Ilya Prigogine, Stuart Kauffman, James Loveback, Ralph Stacey
Learning Systems – Kurt Lewin, Eric Twist, Chris Argyris, Donald Schon, Mary Catherine Bateson

8. System Boundaries Shows what is inside and outside of the system
Geographical (location)
Organization (department, unit or function)
Physical (money, material, information)
Conceptual (goals, mission, purpose, rules)
Intangibles (perceptions, awareness, models)
Natural or man-made
The change of a boundary can affect an entire system. We just experienced the 20th anniversary of a boundary shift that changed history – the fall of the Berlin Wall – it was part of a cascading set of changes from Perestroika to the break-up of the Soviet Union – that was one of the major paradigm shifts in the last century – the elimination of an “iron curtain”.

9. System Relationships (Interconnections)
Connections and exchanges among system parts, parts and the whole, and the whole and its environment
Flows of information
Flows of funding
Client referrals
Collaborative partnerships
Family, community, and social networks
The creation of new relationships is an important tool of diplomacy for system change/paradigm shifts. Nixon’s visit to China to meet with Mao Zedong, Zhoe Enlai and others led to normalization of relations between the U.S. – Chinese.

10. System Perspectives Stakeholders’ worldviews and purposes
System agents who have different perspectives may pursue different purposes within a given situation
Patterns of (mis)alignment of purposes and processes within and across system levels

11. System Change
System differences generate creative tension or energy within a system
Positive or negative, energy provides potential for system change
System change: shifts in patterns (similarities and differences) of system relationships, boundaries, focus, timing, events and behaviors over time and space

12. System Dynamics Random (unorganized) Organized (simple or complicated)
Adaptive (organic, self-organizing)
All three system dynamics can be present in a complex situation

13. Random System Attributes
Random activity – no pattern
Unconnected collection of parts
No cause-effect relationships
Turbulence – no stability or equilibrium
Answers are unknowable
No purpose or direction – people react blindly in a war zone or natural disaster

14. Random System: Hurricane Katrina

15. Organized (Simple) System Attributes
Stable, static pattern
Parts tightly connected machines
Predictable cause-effect relationships
System can be reduced to parts and processes and replicated
Directive leadership, designed change
Answers are knowable, with recipes or prescriptions for action

16. Single Organized System: Ring-Around the Rosie
Simple system – parts are tightly connected, process is simple and straightforward – sing a verse and fall down

17. Simple Organized System: Riding a Bicycle
Another simple system that includes a human with machinery.

18. Organized (Complicated) System Attributes
Dynamic patterns of feedback loops with many interrelated parts within and across subsystem levels
Recursive, non-linear cause-effect relationships; reinforcing and balancing feedback loops maintain equilibrium
Expert analysis can identify causal loops, deep structural causes to actions

19. Insider Trading: A Tangled Web of Tips and Trades
New York Times Friday November 6, 2009
“14 are Charged with Insider Trading as Galleon Case Grows” The charges are against hedge funds and the networks of market gossip that are endemic on trading floors against money managers, lawyers, and other investors.

20. Adaptive (Complex) System Attributes
Dynamical patterns – parts adapting, co-evolving with each other and environment
Parts are massively entangled and interdependent; nested webs, networks
Parts self-organize, learn, and change
Equilibrium in flux, sensitive to initial conditions; system change emerges through interactions among parts

21. Ecological View of an Elephant
When you can the boundaries of inquiry, you can see an elephant’s adaptation and co-evolution in its context – hot, dry, grassy, with dangerous predators and need for migration across large spaces for food and water

22. Complex Interdependencies

23. Alignment of Context, Program, and Evaluation Dynamics
Context can be random, organized, adaptive, or combination of dynamics
Program design uses random, organized (entity-based), or adaptive (paradigm-based) or a combination of dynamics
Evaluation design (content and process) can be entity-focused (organized), paradigm-focused (adaptive) or a combination of both

24. System Dynamics of Family Nutrition

25. Simple Organized Dynamics of Family Nutrition
Context: hungry family
Intervention: buy ingredients, bake a cake, serve family at dinner
Evaluation: quality of cake, family satisfaction
Not focus on overall nutrition – focus on specific event
Straightforward, linear relationship between hunger, eating cake, and outcome (satisfaction)

26. Complicated Organized Dynamics of Family Nutrition
Context: hungry family with different tastes and preferences
Intervention: ask for family preferences, create optional dishes, serve family multiple dishes at dinner
Evaluation: quality and variety of dinner options, matching of dishes to tastes
Family more multi-dimensional – so organize parts (family members by food preferences) and create food for each category. Still assumes one person is making decisions about when is dinner, what for dinner and who is coming to dinner. Also linear, but a feedback loop in the evaluation if there is a mismatch.

27. Complex, Adaptive Dynamics of Family Nutrition
Context: hungry family with different tastes, schedules, and cooking ability
Intervention: Buy and store meal options, make dishes for non-cooks, agree on dinner schedule, adapt shopping patterns to use of food and supplies
Evaluation: trends, patterns of food use, meals, family nutrition, overall health
Many semi-independent agents who are making own decisions about what to eat, when to eat, and how to prepare food. Family dinner is no longer assumed, but the focus of family debate and negotiation

28. System Dynamics of H1N1 Flu

29. Simple Organized Dynamics of H1N1 Flu
Context – everyone should be protected through vaccination
Program design – universal flu shot clinics
Evaluation design - How many clinics were conducted, how many people were vaccinated, how many people contracted the H1N1 flu virus
Some states used this approach - got the seasonal and H1N1 vaccines and distributed them, first come, first serve, until they ran out. A quick fix with unintended consequences – high risk children and pregnant women did not have access to the prevention of a potentially life-threatening illness

30. Complicated Organized Dynamics of H1N1 Flu
Context – people are at different risk levels for contracting the H1N1 flu
Program design – allocate, administer flu shots by risk level, triage patients by level of risk
Evaluation design - What proportion of people with high/medium/low risk receive the vaccine? What proportion of people at each risk level contract the H1N1 flu? How many deaths and hospitalizations are avoided as result of shots?
Assumed that the only intervention is prevention – simple linear theory of action, although implemented in a complicated way to account for subpopulations at different risk levels. Organized, top-down, focused on people as passive recipients of treatment

31. Complex Adaptive Dynamics of H1N1 Flu
Context – Timing of two interacting epidemics (H1N1 and seasonal flu) is ahead of current vaccine production
Program design – Multi-level intervention: national media messages, provider triage by risk, populations self-organize multiple responses
Evaluation design – What are changing patterns of twin epidemics? How are governments, providers, populations reacting and interacting in response to situation? Population health impacts?
Because there are two simultaneous epidemics, vaccination production assumptions and production capacity is overtaxed, partly because of use of old technology (live eggs) that is based on an annual flu cycle, not an ongoing global pandemic co-evolving with the annual flu cycle. Multiple interventions are needed to prevent spread of both flu strains, involving multiple partners at multiple levels – government, producers, medical providers, employers, adults, schools, child care centers and children who are self-organizing many different responses – changing hand-washing techniques, installing hand sanitizers in public buildings, frequent government updates on status of vaccine production, public education of who is high risk, provider triage and gatekeeping of distribution of shots, use of tama-flu medicine for high-risk patients, employers also offer tamaflu instead of vaccine clinics, liberalization of working at home policies, public pressure for more vaccines, child care centers changing their policies of allowing children with colds to remain in day care.

32. System Dynamics Discussion
What are the situations’ boundaries, focus, interconnections, perspectives, power, timing, and dynamics?
What are the risks of not understanding the system attributes and dynamics?
What are the benefits of understanding the system attributes and dynamics?

33. System Dynamics of Child Abuse Prevention – Home Visiting

34. U.S. Child Abuse and Neglect Trends
National Child Abuse and Neglect Data System (NCANDS)

35. Context, Program Design of Child Abuse Prevention
Context: Many programs exist but child abuse and neglect rates are increasing
Program design: AFC funding for 17 grants for the adaptation, implementation, spread, and sustainability of evidence-based home visiting programs through infrastructure development and system change
Organized (entity-focused) and complex adaptive (paradigm-focused) program design elements. Conceptualization of tangible (staff, training, funding, evaluation, program materials) and intangible infrastructure (communication, collaboration, community support) Recognition of mix of dynamics

36. Evaluation Design of Evidence-based Home Visiting Initiative
Program evaluation – tracking of cross-site cost, implementation, fidelity, and child and family outcomes of 17 EBHV programs
System evaluation – tracking of cross-site and grantee-specific system infrastructure, theories of action, measures of system change, partner collaboration and network analysis; system unit of analysis
Standardized cross-site measures of entity-based program components – 5-6 evidence-based home visiting models. Models and program participants are units of analysis. Grantee-specific definitions of systems and system change, contemporaneous data collection of goals, activities, partners, and context, tracking changes (expected and unexpected) over time. Unit of analysis is the system

37. Open Space Technology: System Dynamics Exercise
What are the dynamics (i.e., the nature and balance of types of system dynamics) of the situation as a whole?
What are the system dynamics of the intervention?
What are the implications for the evaluation design and process?

38. Three Dynamics of a Social System and its Context

39. Initiative Renewal Design
Match of Evaluation Designs to Dynamics of Social Systems and Their Context
Exploratory Design
Initiative Renewal Design
Organic Design
Predictive Design

40. Complex Adaptive Systems and Adaptive (Self-organizing) Dynamics
Self-organizing/adaptive/organic
Sensitivity to initial conditions
Emergence
Macro pattern

41. Complex Adaptive Systems and Adaptive (Self-organizing) Dynamics (cont
Feedback
Co-evolution
Pattern formation and points of influence

42. Implications for Evaluation and Action
Small differences can create large effects.
The past influences but does not predict the future.
Many points of influence exist.
Boundaries, differences, and relationships are levers of influence toward a purpose.

43. Implications for Evaluation and Action
Simple rules underlie patterns.
Pattern-based feedback and actions are iterative.
Tensions are not resolved.
Patterns are outcomes.

44. Four Stages of Evaluation
Design Evaluation
Shape Practice
Collect Data
Make Meaning from Data

46. Learning through Engineering Design and Practice
Example: LEAP
Learning through Engineering Design and Practice

47. Example: LEAP Research Design
Quasi-experimental design embedded in curriculum development process
Pre-post assessments of
Content knowledge
Perceptions of engineers at work
Tinkering
Self-efficacy
Engineering notebooks
Career behaviors survey

48. External Evaluation Design
The external evaluation focused on:
Confirmation of effectiveness
Scale-up
Sustainability

49. Conceptual Shifts
The fundamental conceptual shift in this project was from:
teacher-directed de-contextualized learning to student-engaged project-based learning
fixed skills and knowledge as learning outcomes to the desired outcomes being that students are actively engaged; develop the capacity to explore and figure things out; and act like an engineer.

50. Confirmation of Effectiveness
Knowledge and skills related to project topics and STEM concepts
Enjoyment and pride in project work
Development of teamwork, collaboration and workplace skills
Interest in STEM courses and pursuit of STEM career and educational pathways

51. Scale-Up Scale-up involved two tracks:
Greater use of the curriculum within the school system
Transfer of the curriculum to Boys and Girls Clubs

52. Sustainability Program sustainability
Sustainability of learning of participating students
Sustainability of collaborations
Sustainability of teaching capacity

53. Program Sustainability
Maintain relationships with the district
Professional development for teachers
Shape research related to the project
Explore ways to continue project at 9th grade
Track STEM course selection of project students in high school
Collaborate on additional community dissemination and funding

54. Sustainability of Learning of Students
Classroom
Extracurricular activities
Career-related activities
Focused attention through high school
Continued involvement of university faculty and students
Continued contact with science center

55. Sustainability of Collaborations
Use current collaborations to spur others over time

56. Sustainability of Teaching Capacity
Build capacity through formal and informal professional development approaches
Training for new teachers
Coaching by master teachers
Summer professional development activities
Technology enhanced training

57. Communities of Learning, Inquiry, and Practice (CLIPs)
Example 2
Communities of Learning, Inquiry, and Practice (CLIPs)
(video at

58. Strengthening Families
Example 3
Strengthening Families

59. Strengthening Families Protective Factors
Parental resilience
Social connections
Concrete support in time of need
Knowledge of parenting and child development
Social and emotional competence of child

61. World Café Exercise
What data gathering and/or analysis tools have you found helpful in gaining a deeper understanding of complex systems or interventions?
What practices help you develop your capacity to recognize patterns?
Do certain practices seem more related to finding surface patterns and others more related to finding deep patterns?
Some states used this approach - got the seasonal and H1N1 vaccines and distributed them, first come, first serve, until they ran out. A quick fix with unintended consequences – high risk children and pregnant women did not have access to the prevention of a potentially life-threatening illness

62. Contact Information
Beverly Parsons, Ph.D (360)
Meg Hargreaves, Ph.D. (617)