Matching Evaluation to the Nature & Conceptualization of the Problem Simple, Complicated and Complex Problem Framings and the Implications for Evaluation Brenda Zimmerman, Schulich School of Business York University, Toronto, Canada Presentation to the Joint Canadian Evaluation Society/American Evaluation Association Conference Toronto Oct 29, 2005

Social Transformation – our definition An alteration of what is established by the introduction of new elements or forms (including new ideas, practices, or resource flows). In particular the alteration of social relationships to allow for an improvement in or transformation of intransigent and broadly based social problems

Our approach to social innovation Connecting understanding of large scale patterns or dynamics to action at the level of individuals or groups Concerned with how to understand these interactions in such a way that we could help those trying to make a difference

The Decision to Use a Complexity Lens Complexity theory leads us to look at the relationship between the micro and the macro It suggests how individuals act effectively in arenas which they cannot control (complex is more than complicated) Complexity theory points to what is happening “in the between”; in the relationships between things. Implications for evaluation??

Following a Recipe A Rocket to the Moon Raising a Child ComplicatedComplex zThe recipe is essential zRecipes are tested to assure replicability of later efforts zNo particular expertise; knowing how to cook increases success zRecipe notes the quantity and nature of “parts” needed zRecipes produce standard products zCertainty of same results every time Simple

Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome ComplicatedComplex zThe recipe is essential zRecipes are tested to assure replicability of later efforts zNo particular expertise; knowing how to cook increases success zRecipes produce standard products zCertainty of same results every time Simple

Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains ComplicatedComplex zThe recipe is essential zRecipes are tested to assure replicability of later efforts zNo particular expertise; knowing how to cook increases success zRecipes produce standard products zCertainty of same results every time Simple

Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains ComplicatedComplex zThe recipe is essential zRecipes are tested to assure replicability of later efforts zNo particular expertise; knowing how to cook increases success zRecipe notes the quantity and nature of “parts” needed zRecipes produce standard products Certainty of same results every time Simple

Sources for charts of next three slides Begun, Zimmerman and Dooley, “Health Care Organizations as Complex Adaptive Systems”, in S. M. Mick and M. Wyttenbach (eds.), 2003 Advances in Health Care Organization Theory San Francisco: Jossey-Bass, pp Also can be found at

Complexity ScienceEstablished Science --- “complicated” HolismReductionism IndeterminismDeterminism Relationships among entitiesDiscrete entities Nonlinear relationships – critical mass thresholds Linear relationships – marginal increases Quantum physics – influence through iterative nonlinear feedback – expect novel and probabilistic world Newtonian physics – influence as direct result of force from one object to another – expect predictable world Understanding; sensitivity analysisPrediction Focus on variationFocus on averages Local controlGlobal control Behavior emerges from bottom upBehavior specified from top down Metaphor of morphogenesisMetaphor of assembly Adapted from Dent 1999

Established Perspectives*Complexity Science Perspective TEMPORAL FRAMING VIEW OF THE FUTURE Relatively knowableRelatively unknowable RELEVANCE OF HISTORY None (transaction cost) to high (institutional). When high, history is deterministic. High, but history may or may not be deterministic. SPATIAL FRAMING DOMAIN OF STUDY Reified organization in the environment Relationships among individuals, subsystems, systems VIEW OF THE ENVIRONMENT Outside the organization; evolves separately Part of the domain of study; coevolves with the organization LEVELS OF ANALYSIS Single to few, relatively independent Multitude of nested levels CONSTRUCT FRAMING STRATEGYRelatively designedRelatively emergent STRUCTUREEquilibrium; relatively centralized Non-equilibrium; relatively decentralized KEY INFO. FOR THE ORGANIZATION External environmental intelligence Functioning of relationships INFORMATION PROCESSOR Reified organizationIndividuals; complex systems of individuals

Complexity Science Perspective Implications for Research TEMPORAL FRAMING VIEW OF THE FUTURERelatively unknowablePatterns may repeat, but without predictive power. Anticipate surprise. Study emergence. RELEVANCE OF HISTORY High, but history may or may not be deterministic. Requisite to study history (vs. cross- sectional only); conduct longitudinal analysis SPATIAL FRAMING DOMAIN OF STUDYRelationships among ind., subsystems, systems Study patterns of interaction among agents. VIEW OF THE ENVIRONMENT Part of the domain of study; coevolves with the org. Study co-evolution of organization and environment LEVELS OF ANALYSISMultitude of nested levelsView issue from multiple, nested levels of systems CONSTRUCT FRAMING STRATEGYRelatively emergentStudy changes in strategy and conditions that facilitate change STRUCTURENon-equilibrium; relatively decentralized Assess flexibility of structures; simple rules; min specs PURPOSE OF ORGANIZATIONAL RELATIONSHIPS Learning; co-creation of meaning; functioning of relationships Assess degrees of co-participation, learning, sharing; study quality of relationships KEY INFORMATION FOR THE ORGANIZATION Functioning of relationshipsStudy quality of relationships INFORMATION PROCESSOR Individuals; complex systems of individuals Study individuals and coalitions, vs. reified organization

The Brazil Case Study annual per capita income is less than $5000 In the 1980s, Brazil’s AIDS problem was worse than South Africa’s By 2000, South Africa’s HIV infection rate was 25% whereas Brazil’s was 0.6% In 1992, the World Bank predicted that Brazil would have 1.2 million AIDS cases by 2000 …but the actual count was 0.5 million. Brazil Analysis is from Begun, Zimmerman and Dooley, 2002

The Brazil Complex Problem How do you respond to an AIDS epidemic in a developing country? No money for expensive drugs problems of affordability of drugs A very iffy health care system problems of treatment High levels of illiteracy problems of compliance even if there were drugs High levels of poverty and hunger problems of nutritional needs for drugs

Making the Brazil Problem Complicated “What will drug costs be for our infected population?” “What resources are needed to manage drug therapies for illiterate patients?” “What resources are needed to assure compliance with drug associated nutrition in this population?” “What are the resources needed for an effective prevention program?”

World Bank Responds to AIDS as Complicated Meaningful solutions require sophisticated, integrated national health care systems We cannot provide treatment to all when the drug costs are so high We cannot afford resources to manage treatment compliance With our limited resources, we should focus more on prevention than treatment It will therefore take a long time for the problem to work itself through

Brazil Implicitly Recognized the Complexity Began to address the AIDS issue head-on in 1994 Were unwilling to accept the answers of the World Bank Hence – had to change the questions… Changing the questions changes the focus, changes what is “analyzed”, changes what is seen as possible/impossible

The Brazil Questions Assume Complexity “What will drug costs be for our infected population?” “What resources are needed to manage drug therapies for illiterate patients?” “ How can we reduce costs so that we can provide treatment to all who need it?” “What methods of communication will work to convey the drug therapy routine to a patient – even a homeless, illiterate patient?” World Bank QuestionsBrazil Questions

The Brazil Questions “What resources are needed to assure compliance with drug associated nutrition in this population?” “What are the resources needed for an effective prevention program?” “If food is an issue, how can we ensure greater compliance with the routine by linking up with charities that can provide food at the right times of day?” “How can we achieve our prevention goals while treating all of those currently infected?” World Bank QuestionsBrazil Questions

The Brazil Conclusions Meaningful solutions require sophisticated, integrated national health care systems We cannot provide treatment to all when the drug costs are so high We cannot afford resources to manage treatment compliance With limited resources, focus more on prevention than treatment It will therefore take a long time for the problem to work itself through Find ways to use the resources we have to respond to the problem Provide drugs to all by finding ways to reduce drug costs Use our informal system to train people to care for themselves Prevention will be part of the treatment Seek short and long term results World Bank Conclusions Brazil Conclusions

The Brazil Responses A stable container: Brazil built on existing infrastructure (natural network existed) A somewhat shaky health system of hospitals and clinics Added to by 600 NGOs, churches (hubs) Free drugs to all AIDS patients: faced down drug companies in pursuit of national interest

The Brazil Responses - cont’d Patients managed their own drugs Illiterate people were taught by local “trustworthy” folks (hubs were sought) Free treatment spread prevention ideas No labeling of those affected Prevention information readily available

Brazil saw the value in the relationships as the key to change

The Brazil AIDS story - optimism COURAGE – to challenge WTO, USA, large pharmaceuticals, World Bank COMPLEXITY – lived with the complex nature of the society, and the AIDS problem RELATIONSHIPS – used the power of existing relationships to learn and enhance their connectedness

Matching Evaluation to Cycles of Intervention & Program Development Ecocycle & Panarchy as a Framework

The ecocycle as a metaphor for successful social innovation Based on the work of ecologists studying natural systems (C.S. Hollings in particular) Suggests four distinct stages in a “life cycle”, linked to two key dynamics: connectedness or sameness and stored capital or potential

BIRTH GROWTH MATURITY

The birth, growth, destruction and renewal of a forest Variety CONNECTEDNESS Sameness Exploitation 1 Released POTENTIAL Stored

12/3/2015 exploitation

The Psychosocial place called “exploitation” Triggers: Choice! Groups of people have converged on some promising products or processes and are devoting time and energy to these. Decision to invest in one or more and build capacity People: the implementers come into their own. Invest in people who love to take ideas and develop them into reality, the team-builders and the engineers come into their own. Experience: Excitement, flow, high energy, doors opening, long hours, time flying by….often correlated with peak experiences. A time of intense activity, flow and productivity with a steep organizational “learning curve” Evaluation: “Formative” – look for a continuous feedback systems as the start- up phase moves towards greater goal clarity and efficiency in delivery Management: Supporting capacity building. Careful attention to the need for training, hiring and developing systems to support bringing a program “to market” and moving it up the scale. Evaluation geared to providing feedback for increasing effectiveness and efficiency. Traps: Failure to build sufficient capacity or sufficient “lock-in” to achieve economies of scale and delivery efficiency

The birth, growth, destruction and renewal of a forest Variety CONNECTEDNESS Sameness Exploitation 1 Released POTENTIAL Stored

12/3/2015 conservation

The Psycho-social place called “conservation” Triggers: Reaching a level of efficiency and learning where improvements become incremental. The “mature” phase, productivity high and measurable, peak “earnings” and efficiency. People: Invest in those excellent at management, system design excel, others may be getting restless, or even a little bored. Experience: satisfaction at success, the pride in the well oiled machine, anxiety about mushrooming demand, loss of momentum Evaluation: Summative evaluation – success based on goals should be measurable by quantitative or qualitative means. Management: awareness of vulnerability, standing still (again), openess to creative destruction. Encourage “letting go” Traps: rigidity, loss of peripheral vision, over-commitment. Fear of change.

The birth, growth, destruction and renewal of a forest Variety CONNECTEDNESS Sameness Exploitation 1 Released POTENTIAL Stored Creative Destruction Conservation

Creative destruction Not devastation of “soil” Crisis…. from root word “to sift” Creation requires destruction To honor creation… open to destruction Externally imposed Internally driven

12/3/2015 Creative destruction

12/3/2015 Creative destruction

The psychosocial space called “creative destruction” Triggers: most clearly signaled by the discontinuation, destruction or collapse of a given program or process and the simultaneous release of resources and energies for other things. People: those who thrive on crisis, on new beginnings are happy here – others may be depressed or in mourning Experience: of confusion, identity crisis,, change in relationships, anxiety, elation. Management: providing reassurance, “standing still”, ‘active” reflection, contemplation, listening. Evaluation: organizations should be “harvesting”: identifying lessons learned and thinking about implications for the future. Traps: Fear and guilt can act to isolate – communication is needed. Outputs: Focus on new ideas, investment of capital in exploration.

Renewal Also known as Exploration or Reorganization

12/3/2015 Renewal/exploration

The psychosocial space called “exploration” Triggers: The need for innovation. Development of multiple “random walks”, experiments, initiatives which lead to little in the way of measurable outcomes for considerable time People: people who learn by doing are happy here; invest in individuals who are entrepreneurs/ innovators with a track record. Experience: Reflection moving to experimentation, lots of false starts and sometimes frustration and mounting anxiety about inputs/output ratios Management: Generate resources and connections and exchange of information, encourage experimentations and learning, have faith in the self- organizing properties of information and groups. Introduce new ingredients as required. Exert increasing pressure for output. Evaluation: geared to capturing learning, building transparent accounts of innovation. Traps: “spinning wheels”, the incapacity to select a course and commit resources to it to the required degree. Ungovernable competition.

Conservation 2 The birth, growth, destruction and renewal of a forest Weak CONNECTEDNESS Strong Exploitation 1 Renewal 4 3 Creative Destruction Little CAPITAL STORED Much

The birth, growth, destruction and renewal of a forest Variety CONNECTEDNESS Sameness Exploitation 1 Released POTENTIAL Stored Creative Destruction ConservationRenewal

POVERTY TRAP RIGIDITY TRAP Variety Sameness Released Stored

The psycho-social space of poverty traps Organization, processes, initiatives: a sense of spinning wheels, initiatives and interactions are infrequent or are not generative Experience: “chronic disaster”, increasing hopelessness or helplessness, as sense of the poverty of the very ground of being

Psycho-social experience of rigidity traps Organizations, systems initiatives: rule- bound, unresponsive, resistant to change, too may resources tied up in non-productive systems. Experience: a kind of rigor mortis, nothing new seems to happen, creativity discouraged, emphasis on evaluation and performance

Stored Released Variety Sameness

Do we evaluate Resiliency? Connectedness is an indicator of how much similarity vs. variety characterizes the behavior/composition of the system Potential (or capital) is an indicator of the degree to which resources (time, energy, money, skill, knowledge) are committed to existing projects as opposed to available to attach to new projects Resilience is a measure of the ability of the system to avoid “traps”, i.e. getting stuck in any phase

Developmental Evaluation Formative Evaluation Summative Evaluation