1. Michael Quinn Patton May, 2008 Evaluating the Complex: Getting to Maybe Michael Quinn Patton Oslo, Norway 29 May, 2008

2. Michael Quinn Patton May, 2008 Evolving Understandings I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they haven’t been thinking. Noam Chomsky “The Professor Provaocateur,” The New York Times Magazine, Nov. 2, 2003: 13.

3. Michael Quinn Patton May, 2008 Interpretive Frameworks May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber. They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment."

4. Michael Quinn Patton May, 2008 They further concluded that it is a waste of resources to spend a lot of money increasing the marginal accuracy of data available to senior executives compared to the value of enhancing their capacity to interpret whatever data they have. Executives were more limited by a lack of capacity to make sense of data than by inadequate or inaccurate data. In essence, they found that interpretive capacity, or "mind-sets," distinguish high- performance more than data quality and accuracy.

5. Michael Quinn Patton May, 2008 Evaluation’s Traditional Interpretive Framework In the beginning…

6. Michael Quinn Patton May, 2008 Original Primary Options Formative and Summative Evaluation (Mid-term and End-of-Project Reviews)

7. Michael Quinn Patton May, 2008 Evidence-based Practice Evaluation grew up in the “projects” testing models under a theory of change that pilot testing would lead to proven models that could be disseminated and taken to scale: The search for best practices and evidenced-based practices

8. Michael Quinn Patton May, 2008 Fundamental Issue: How the World Is Changed Top-down dissemination of “proven models” versus Bottoms-up adaptive management

9. Michael Quinn Patton May, 2008 Models vs. Principles Identifying proven principles for adaptive management (bottoms-up approach) versus Identifying and disseminating proven models (top down approach)

10. Michael Quinn Patton May, 2008 Conditions that challenge traditional model-testing evaluation High innovation Development High uncertainty Dynamic Emergent Systems Change Adaptive Management

11. Michael Quinn Patton May, 2008 Mintzberg on Strategy Two types of strategy: Intended & Emergent Unrealized Strategy Intended Strategy Deliberate Strategy Realized EmergentStrategy Strategy

12. Michael Quinn Patton May, 2008 Re-conceptualizing Use Use is a process not a event Use involves an interaction not just a report Use involves training for use not just delivery of results

13. Michael Quinn Patton May, 2008 Some premises : Evaluation is part of initial program design, including conceptualizing the theory of change Evaluator’s role is to help users clarify their purpose, hoped-for results, and change model. Evaluators can/should offer conceptual and methodological options. Evaluators can help by questioning assumptions. Evaluators can play a key role in facilitating evaluative thinking all along the way. Interpretative dialogue is critical. Designs can be emergent and flexible.

14. Michael Quinn Patton May, 2008 Three ways of conceptualizing and mapping theories of change  Linear Newtonian causality  Interdependent systems relationships  Complex nonlinear dynamics

15. Michael Quinn Patton May, 2008 Linear Logic Model INPUTS (people, materials)  ACTIVITIES (processes)  OUTPUTS  OUTCOMES  CHANGES IN PEOPLES LIVES  IMPACTS  CHANGES IN COMMUNITIES

16. Michael Quinn Patton May, 2008 Pushing Force (Non-directional) Dissatisfaction with the Status Quo (Inertia) a.k.a. “Cost of Change” Resistance to Change Pulling Force (Directional) a.k.a. “Desirability of the end state” Compelling Vision

17. Michael Quinn Patton May, 2008 Pushing Force (Non-directional) Dissatisfaction with the Status Quo (Inertia) a.k.a. “Cost of Change” Resistance to Change Pulling Force (Directional) a.k.a. “Desirability of the end state” Compelling Vision First StepsBelievability Causal Diagram of Beckhard’s Change Formula

18. Michael Quinn Patton May, 2008 Feedback

19. Michael Quinn Patton May, 2008

20. Step Two: THEN A MIRACLE OCCURS “I think you should be more explicit here in step two.”

21. Michael Quinn Patton May, 2008 Systems Parts are interdependent such a change in one part changes all parts The whole is greater than the sum of the parts Focus on interconnected relationships Systems are made up of sub-systems and function within larger systems

22. Michael Quinn Patton May, 2008 Systems Concepts in Evaluation – An Expert Anthology. 2006. Bob Williams and Iraj Imam AEA Monograph, EdgePress/AEA Point Reyes CA.

23. Michael Quinn Patton May, 2008

24. Understanding the Elephant from a Systems Perspective

25. Michael Quinn Patton May, 2008 The relationship between what goes in and what comes out What conceptual framework informs front-end evaluation work?

26. Michael Quinn Patton May, 2008 Teen Pregnancy Program Example

27. Michael Quinn Patton May, 2008 Logic Model for Pregnant Teens Program 1. Program reaches out to pregnant teens 2. Pregnant teens enter and attend the program (participation) 3. Teens learn prenatal nutrition and self-care (increased knowledge) 4. Teens develop commitment to take care of themselves and their babies (attitude change) 5. Teens adopt healthy behaviors: no smoking, no drinking, attend prenatal clinic, eat properly (behavior change) 6. Teens have healthy babies (desired outcome)

28. Michael Quinn Patton May, 2008 Systems web showing possible influence linkages to a pregnant teenager Teachers/ other adults Young pregnant woman's attitudes & behaviors Her parents & other family members Child's father & peers Prenatal program staff Her peer group

29. Michael Quinn Patton May, 2008 Program systems web showing possible institutional influences affecting pregnant teenagers: SCHOOL SYSTEM Young pregnant women's attitudes & behaviors Prenatal Clinic and Hospital Outreach Church Prenatal program Other community- based youth programs Other Systems -- welfare -- legal -- nutrition programs -- transportation -- child protection -- media messages Context factors -- politics -- economic incentives -- social norms -- culture -- music Youth Culture

30. Michael Quinn Patton May, 2008 Using Different System Lenses to Understand a “particular” System Biologic System Emergence Coordination/synergy Structure, Process, Pattern Vitality Sociologic System Relationships Conversations Interdependence Loose-tight coupling Meaning/sense Mechanical / Physical System Flow Temporal Sequencing Spatial Proximities Logistics Information Economic System  Inputs/Outputs  Cost/Waste/Value/Benefits  Customers/Suppliers Political System Power Governance Citizenship Equity Anthropologic System Values Culture/Milieu Information System Access Speed Fidelity/utility Privacy/security Storage Psychological System Organizing Forces Field Ecological/Behaviour Settings SYSTEM DIMENSIONS

31. Michael Quinn Patton May, 2008

32. Complex Dynamic Systems Configuration

33. Michael Quinn Patton May, 2008 HIV/AIDS Example Hits every system: health, family, social, religious, economic, political, community, international Requires multiple interventions on multiple fronts in all subsystems simultaneously Resulting reactions, interactions, consequences dynamic, unpredictable, emergent, and ever changing

34. Michael Quinn Patton May, 2008 Challenges: Situation Recognition and Appropriate Evaluation Designs

35. Michael Quinn Patton May, 2008 The nature of EXPERTISE

36. Michael Quinn Patton May, 2008 Contingency-based Evaluation Situational analysis & responsiveness Context sensitivity Clarify and focus on intended users: stakeholder analysis Clarify and focus on intended uses Methodological appropriateness Criteria for evaluating the evaluation: credibility, meaningfulness

37. Michael Quinn Patton May, 2008 Seeing Through A Complexity Lens “You don't see something until you have the right metaphor to let you perceive it”. Thomas Kuhn

38. Michael Quinn Patton May, 2008 Complex Nonlinear Dynamics Nonlinear: Small actions can have large reactions. “The Butterfly Wings Metaphor” Emergent: Self-organizing, Attractors Dynamic: Interactions within, between, and among subsystems and parts within systems can volatile, changing Getting to Maybe: Uncertainty, unpredictable, uncontrollable

39. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008 SNOWDEN’s CYNAFIN FRAMEWORK Simple, Complicated, Complex, Chaotic and Disordered Behaviours

40. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008

41. Michael Quinn Patton May, 2008 Conceptual Options Simple Complicated Complex

42. Michael Quinn Patton May, 2008 Types of Community Issues The Stacey Matrix Certainty Agreement Close to Far from Close to

43. Michael Quinn Patton May, 2008 Simple Certainty Agreement Close to Far from Close to Simple Plan, control

44. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008 Simple (Known) Inter-relationships between elements : Tight, centralised connections. Anyone can see the things the way they are. Very simple linear cause and effect. Everyone knows the right answer within the current context (which of course may not be self-evident or known to others – hence importance of context).

45. Michael Quinn Patton May, 2008 Following a Recipe A Rocket to the Moon Raising a Child ComplicatedComplex z The recipe is essential z Recipes are tested to assure replicability of later efforts z No particular expertise; knowing how to cook increases success z Recipes produce standard products z Certainty of same results every time Simple

46. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008 Complicated (Knowable) Inter-relationships between elements: Relationships are looser but still clustered around a central core. Cause and effect is non-linear. Relationships able to be modelled and able to predicted. An expert would know the right answer(s)

47. Michael Quinn Patton May, 2008 Technically Complicated Certainty Agreement Close to Far from Close to Simple Plan, control Technically Complicated Experiment, coordinate expertise

48. Michael Quinn Patton May, 2008 Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome ComplicatedComplex zThe recipe is essential zRecipes are tested to assure replicability of later efforts zNo particular expertise; knowing how to cook increases success zRecipes produce standard products zCertainty of same results every time Simple

49. Michael Quinn Patton May, 2008 Socially Complicated Certainty Agreement Close to Far from Close to Simple Plan, control Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground

50. Michael Quinn Patton May, 2008 Socially complicated Implementing human rights agreements, like gender equity or outlawing child labor Environmental Initiatives  Many different and competing stakeholders  Diverse vested interests  High stakes

51. Michael Quinn Patton May, 2008 Socially complicated situations pose the challenge of coordinating and integrating many players

52. Michael Quinn Patton May, 2008 Stakeholder Mapping High Interest/ Low Power THE INVOLVED High Interest/ High Power THE PLAYERS THE CROWD Low interest/ Low Power CONTEXT SETTERS Low Interest/ High Power

53. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008 Complex Inter-relationships between elements: Centre is loosely connected to network. Cause effect difficult to understand in current setting. Situation understandable only in retrospect. Not predictable.

54. Michael Quinn Patton May, 2008 Know When Your Challenges Are In the Zone of Complexity Certainty Agreement Close to Far from Close to Simple Plan, control Zone of Complexity Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground Systems Thinking Relationship Building Collaboration Good Enough Vision Chunking Around Drivers Minimum Specifications Multiple Actions Adaptability & Organic

55. Michael Quinn Patton May, 2008 Following a Recipe A Rocket to the Moon Raising a Child Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Every child is unique Uncertainty of outcome remains ComplicatedComplex zThe recipe is essential zRecipes are tested to assure replicability of later efforts zNo particular expertise; knowing how to cook increases success zRecipes produce standard products zCertainty of same results every time Simple

56. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008 Chaotic (Unordered) Cause and effect unknowable, unattributable even in retrospect. No right answer(s)

57. Michael Quinn Patton May, 2008 Know When Your Challenges Are In the Zone of Complexity Certainty Agreement Close to Far from Close to Simple Plan, control Zone of Complexity Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground Chaos Massive Avoidance

58. Michael Quinn Patton May, 2008 Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains ComplicatedComplex zThe recipe is essential zRecipes are tested to assure replicability of later efforts zNo particular expertise; knowing how to cook increases success zRecipe notes the quantity and nature of “parts” needed zRecipes produce standard products zCertainty of same results every time Simple

59. Michael Quinn Patton May, 2008 Michael Quinn Patton TEI 2008

60. Michael Quinn Patton May, 2008 Global Economic Complexity Arthur Greenspan, Final speech to world’s Central Bankers, Jackson Hole, Wyoming August 26, 2005

61. Michael Quinn Patton May, 2008 “In the absence of a single variable, or at most a few, that can serve as a reliable guide, policymakers have been forced to fall back on an approach that entails the interpretation of the full range of economic and financial data.”

62. Michael Quinn Patton May, 2008 “Despite extensive efforts to capture and quantify what we perceive as the key macroeconomic relationships, our knowledge about many critical linkages is far from complete and, in all likelihood, will remain so. Every model, no matter how detailed or how well conceived, designed, and implemented, is a vastly simplified representation of the world, with all of the intricacies we experience on a day-to-day basis.”

63. Michael Quinn Patton May, 2008 “We all temper the outputs of our models and test their results against the ongoing evaluations of a whole array of observations that we do not capture in either the data input or the structure of our models. We are particularly sensitive to observations that appear inconsistent with the causal relationships of our formal models.”

64. Michael Quinn Patton May, 2008 Oct 8, 2007 on The Daily Show “I was telling my colleagues the other day...I’d been dealing with these big mathematical models for forecasting the economy, and I’m looking at what’s going on the last few weeks and I say, “Y’know, if I could figure out a way to determine whether or not people are more fearful, or changing to euphoric... I don’t need any of this other stuff. I could forecast the economy better than any way I know. The trouble is, we can’t figure that out. I’ve been in the forecasting business for 50 years, and I’m no better than I ever was, and nobody else is either.” Alan Greenspan.

65. Michael Quinn Patton May, 2008 The Age of Turbulence: Adventures in a New World, 2007, by Alan Greenspan

66. Two evaluation locations for Evaluating the Complex: Prospective and Retrospective Michael Quinn Patton May, 2008

67. Prospective Example The McGill-McConnell Leadership Program Example Simple elements Complicated elements Complex elements

68. Simple outcomes Increase knowledge and skills of participants Evaluation: Pre-post data and documentation of learning Michael Quinn Patton May, 2008

69. Complicated Impacts Change participants’ organizations Evaluation: Case studies of organizational change Michael Quinn Patton May, 2008

70. Complex Vision Infuse energy into the moribund not- for-profit (voluntary) sector Make the sector more dynamic Create network of leaders who actively engage in change Michael Quinn Patton May, 2008

71. Evaluating the Complex Real time follow-up of network connections and actions Follow-up is an intervention Rapid feedback of findings permits infusion of resources in support of emergent outcomes Michael Quinn Patton May, 2008

72. Process Use Infusing evaluative thinking as a primary type of process use. Capacity-building as an evaluation focus of process use.

73. Michael Quinn Patton May, 2008 Complex Interdependencies

74. Michael Quinn Patton May, 2008 Insert action into the system

75. Michael Quinn Patton May, 2008 EMERGENCE

76. Michael Quinn Patton May, 2008 Retrospective Example Advocacy Evaluation Final Push Campaign to overthrow the Juvenile Death Penalty

77. In late 2003 several petitions on behalf of juvenile offenders facing the death penalty were filed with the U.S. Supreme Court. On January 26, 2004, the Court granted certiorari in Roper v. Simmons and the case was argued before the Court on October 13, 2004. The decision was announced March 1, 2005. The Court ruled 5-4 that capital punishment for juveniles was unconstitutional. Michael Quinn Patton May, 2008

78. In the brief window of time between when the Court agreed to hear the case and the case was argued, roughly nine months, a coordinated campaign was organized and funded aimed at overturning the juvenile death penalty. Organizing, public education, networking & communications continued through to the Court's ruling in March, 2005. Michael Quinn Patton May, 2008

79. Retrospective Evaluation To what extent, if at all, was the Court’s decision influenced by the campaign? Modus Operandi or General Elimination Method Michael Quinn Patton May, 2008

80. 1. Strong high capacity coalitions. Working through coalitions is a common centerpiece of advocacy strategy. 2.Strong national-state-grassroots coordination. Effective policy change coalitions in the United States have to be able to work bottoms-up and top-down, with national campaigns supporting and coordinating state and grassroots efforts, while state efforts infuse national campaigns with local knowledge and grassroots energy. Strengthening strong national-state coordination is part of coalition development and field building. 3.Disciplined and focused messages with effective communications. Effective communications must occur within movements (message discipline) and to target audiences (focused messaging). Strengthening communications has been a key a key component of advocacy coalition building. 4. Solid research and knowledge base. The content of effective messages must be based on solid research and timely knowledge. In the knowledge age, policy coalitions must be able to marry their values with relevant research and real time data about dynamic policy environment. 5. Timely, opportunistic lobbying and judicial engagement. The evaluation findings emphasize that effective lobbying requires connections, skill, flexibility, coordination, and strategy. 6. Collaborating funders engaged in strategic funding. Effective funding involves not only financial support, but infusion of expertise and strategy as part of field building.

81. Michael Quinn Patton May, 2008 Targeted State Campaigns & Grassroots Organizing Omnibus Coordinated Integrated Strategy and Implementation Effective LitigationCredible, Useful and Amicus Up-to-date Research Briefs United Focused Coalition Communications Partners Campaign Knowledgeable Funders

82. Michael Quinn Patton May, 2008 Overall Lesson Learned for Effective Advocacy In essence, strong national/state/grassroots coordination depends on having a high capacity coalition. A solid knowledge and research base contributes to a focused message and effective communications. Message discipline depends on a strong coalition and national-state coordination, as does timely and opportunistic lobbying and judicial engagement. To build and sustain a high capacity coalition, funders must use their resources and knowledge to collaborate around shared strategies. These factors in combination and mutual reinforcement strengthen advocacy efforts. In classic systems framing, the whole is greater than the sum of parts, and the optimal functioning of each part is dependent on the optimal integration and integrated functioning of the whole.

83. Michael Quinn Patton May, 2008 National/ Grassroots Coordination Opportunistic Lobbying & Judicial Engagement Strong High Capacity Coalitions EFFECTIVE ADVOCACY Disciplined Focused Message/ Effective Communications Collaborating Funders/ Strategic Funding SIX INTERCONNECTED FACTORS, DYNAMICALLY INTERACTING, THAT STRENGTHEN ADVOCACY Solid Knowledge & Research Base

84. Michael Quinn Patton May, 2008 Strong, high capacity coalition Collaborating funders/ strategic funding Targeted timely lobbying National state grassroots coordinatio n Disciplined focused message Relevant research The interdependent system of factors that contribute to effective advocacy and change

85. Dealing with the Unexpected and Unpredicted Michael Quinn Patton May, 2008

86. Photo by Lynsey Gornick

87. Michael Quinn Patton May, 2008 Creative Challenge Situational adaptability:  Contingency-based evaluation  Appropriateness --Using standard forms of evaluation and -- Going beyond standard forms when appropriate and useful

88. Michael Quinn Patton May, 2008 Being Open: How hard is this to do? My Colorado experience….

89. Michael Quinn Patton May, 2008 “The range of what we think and do is limited by what we fail to notice. And because we fail to notice that we fail to notice, there is little we can do to change, until we notice how failing to notice shapes our thoughts and deeds.” Scottish psychiatrist, R. Laing

90. Paradigms and Lenses The importance of interpretive frameworks Complexity as an interpretive framework Michael Quinn Patton May, 2008

91. Getting to Maybe: How the World Is Changed? 2006 Frances Westley, Brenda Zimmerman, Michael Q. Patton Random House Canada,

92. Michael Quinn Patton May, 2008 Complex Situations Highly emergent (difficult to plan and predict) Highly dynamic, rapidly changing Relationships are interdependent and non-linear rather than simple and linear (cause-effect)

93. Michael Quinn Patton May, 2008 Certainty Agreement Close to Far from Close to ? Zone of Complexity

94. Michael Quinn Patton May, 2008 Certainty Agreement Close to Far from Close to Search for root cause

95. Michael Quinn Patton May, 2008 Certainty Agreement Close to Far from Close to Search for systemic features

96. Michael Quinn Patton May, 2008 Contingency-based Developmental Evaluation

97. Michael Quinn Patton May, 2008 Improvement versus Development

98. Michael Quinn Patton May, 2008 Beyond just Summative and Formative

99. Michael Quinn Patton May, 2008 Beyond Static Accountability Models

100. Michael Quinn Patton May, 2008 Example of an emergent option: Developmental Evaluation

101. DEVELOPMENTAL EVALUATION DEFINED Evaluation processes, including asking evaluative questions and applying evaluation logic, to support program, product, staff and/or organizational development. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation and intentional change. The evaluator's primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and facilitate data-based decision-making in the developmental process. Michael Quinn Patton May, 2008

102. Other names  Real time evaluation  Emergent evaluation  Action evaluation  Adaptive evaluation

103. Michael Quinn Patton May, 2008 CONTRASTS Traditional evaluations… Testing models Complexity-based, Developmental Evaluation… Supporting innovation and adaptation

104. Michael Quinn Patton May, 2008 Traditional Evaluation… Render definitive judgments of success or failure Developmental Evaluation… Provide feedback, generate learnings, support direction or affirm changes in direction in real time

105. Michael Quinn Patton May, 2008 Traditional Evaluation… Render definitive judgments of success or failure Measure success against predetermined goals Developmental Evaluation… Provide feedback, generate learnings, support direction or affirm changes in direction Develop new measures and monitoring mechanisms as goals emerge & evolve

106. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation… Evaluation… Evaluator external, independent, objective Evaluator part of a team, a facilitator and learning coach bringing evaluative thinking to the table, supportive of the organization’s goals

107. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation…Evaluation… Evaluator determines the design based on the evaluator’s perspective about what is important. The evaluator controls the evaluation. Evaluator collaborates with those engaged in the change effort to design an evaluation process that matches philosophically and organizationally.

108. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation… Evaluation… Design the evaluation based on linear cause-effect logic models Design the evaluation to capture system dynamics, interdependencies, and emergent interconnections

109. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation… Evaluation… Aim to produce generalizable findings across time & space. Aim to produce context-specific understandings that inform ongoing innovation

110. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation… Evaluation… Accountability focused on and directed to external authorities and funders. Accountability centered on the innovators’ deep sense of fundamental values and commitments – and learning.

111. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation… Evaluation… Accountability to control and locate blame for failures Learning to respond to lack of control and stay in touch with what’s unfolding And thereby respond strategically

112. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation… Evaluation… Evaluation often a compliance function delegated down in the organization Evaluation a leadership function: Reality-testing, results-focused, learning-oriented leadership

113. Michael Quinn Patton May, 2008 TraditionalDevelopmental Evaluation…Evaluation… Evaluation engenders fear of failure. Evaluation supports hunger for learning.

114. Michael Quinn Patton May, 2008 Conditions High innovation Development High uncertainty Dynamic Emergent

115. Michael Quinn Patton May, 2008 SenseMaker software Dave Snowden, Founder of Cognitive Edge, former Director of Knowledge Management at IBM SenseMaker can code and map 95,000 stories in 24 hours See the world as others see it; anti-terror applications. See the quantitative patterns in the meta- data with qualitative context and meaning

116. Michael Quinn Patton May, 2008 PANARCHY MODEL

117. Michael Quinn Patton May, 2008 Routine Change Foreloop Growth r Conservation K

118. Michael Quinn Patton May, 2008 Turbulent Change Release  Backloop Reorganization 

119. Michael Quinn Patton May, 2008 Adaptive Cycle Foreloop Growth r Conservation K Release  Backloop Reorganization 

120. Stored Leadership/initiative Exploration phase System Capacity Building Mature product and scaling up Phases of Technological and Social Innovation

121. The birth, growth, destruction and renewal of a forest Weak CONNECTEDNESS Strong Little CAPITAL STORED Much Creative Destruction 1

122. The birth, growth, destruction and renewal of a forest Weak CONNECTEDNESS Strong Little CAPITAL STORED Much Creative Destruction 1 2 Renewal/Expl oration

123. The birth, growth, destruction and renewal of a forest Weak CONNECTEDNESS Strong Exploitation 3 Little CAPITAL STORED Much

124. The birth, growth, destruction and renewal of a forest Weak CONNECTEDNESS Strong Exploitation 3 Little CAPITAL STORED Much Conservation 4

125. Stored HARVESTING LESSONS DEVEOPMENTAL EVALUATION FORMATIVE SUMMATIVE Phases of Technological & Social Innovation

126. Michael Quinn Patton May, 2008 Taking Emergence Seriously Beyond “unanticipated consequences” to genuine openness

127. Michael Quinn Patton May, 2008 SURPRISES The psychologist Baruch Fischhoff wrote: "The occurrence of an event increases its reconstructed probability"—in other words, surprises are psychologically untenable in some ways, and we reshape our memories and expectations until we believe that the surprising event was, in fact, likely.

128. Michael Quinn Patton May, 2008 The Epistemology of Surprise MALCOLM GLADWELL We hate surprises. We try to erase them from our memory. This is part of what keeps us sane. If, after all, we were always fully aware of the possibility of completely unpredictable events, would we be able to walk out the front door in the morning? Would we ever invest in the stock market? Would we have children? Generally speaking, people who have an accurate mental picture of why and how things happen tend to occupy mental hospitals—or, at the very least, a psychiatrist's office….

129. Michael Quinn Patton May, 2008 Information is like a stream teeming with fish, and if you stick out a net you'll collect something— but to decide what information is consequential. How does one do that?

130. Michael Quinn Patton May, 2008 At the heart is the process of distinguishing signal from noise. How is that done? I have no idea, nor does anyone, I think, who isn't a seasoned analyst. Pattern recognition is something that comes only with experience. It's a matter of intuition, as much as anything. People always want to reduce this sort of thing to a formula, or a system, and I'm not sure you can do that. I suspect that there are some artificial-intelligence systems that can help to sort through certain kinds of data. But that could only be a first cut, and eventually human judgment has to be involved.

131. Michael Quinn Patton May, 2008 What role can evaluation play with complex dynamic innovations?

132. Michael Quinn Patton May, 2008 Challenge: Matching the evaluation process and design to the nature of the situation: Contingency-based Evaluation

133. Michael Quinn Patton May, 2008 References Getting to Maybe: How the World Is Changed? Frances Westley, Brenda Zimmerman, Michael Q. Patton, Random House Canada, 2006 Utilization-Focused Evaluation, 4 th ed., Michael Quinn Patton, Sage, 2008.