Evaluating the Complex: Getting to Maybe Michael Quinn Patton

Evaluating the Complex: Getting to Maybe Michael Quinn Patton
Oslo, Norway 29 May, 2008 Michael Quinn Patton May, 2008

Evolving Understandings
I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they haven’t been thinking. Noam Chomsky “The Professor Provaocateur,” The New York Times Magazine, Nov. 2, 2003: 13. Michael Quinn Patton May, 2008

Interpretive Frameworks
May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber. They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment." Michael Quinn Patton May, 2008

Michael Quinn Patton May, 2008
They further concluded that it is a waste of resources to spend a lot of money increasing the marginal accuracy of data available to senior executives compared to the value of enhancing their capacity to interpret whatever data they have. Executives were more limited by a lack of capacity to make sense of data than by inadequate or inaccurate data. In essence, they found that interpretive capacity, or "mind-sets," distinguish high-performance more than data quality and accuracy. Michael Quinn Patton May, 2008

Evaluation’s Traditional Interpretive Framework
In the beginning… Michael Quinn Patton May, 2008

Original Primary Options
Formative and Summative Evaluation (Mid-term and End-of-Project Reviews) Michael Quinn Patton May, 2008

Evidence-based Practice
Evaluation grew up in the “projects” testing models under a theory of change that pilot testing would lead to proven models that could be disseminated and taken to scale: The search for best practices and evidenced-based practices Michael Quinn Patton May, 2008

Top-down dissemination of Bottoms-up adaptive management
Fundamental Issue: How the World Is Changed Top-down dissemination of “proven models” versus Bottoms-up adaptive management Michael Quinn Patton May, 2008

Models vs. Principles Identifying proven principles for adaptive management (bottoms-up approach) versus Identifying and disseminating proven models (top down approach) Michael Quinn Patton May, 2008

Conditions that challenge traditional model-testing evaluation
High innovation Development High uncertainty Dynamic Emergent Systems Change Adaptive Management Michael Quinn Patton May, 2008

Mintzberg on Strategy Two types of strategy: Intended & Emergent Unrealized Strategy Intended Strategy Deliberate Strategy Realized Emergent Strategy Michael Quinn Patton May, 2008

Re-conceptualizing Use
Use is a process not a event Use involves an interaction not just a report Use involves training for use not just delivery of results Michael Quinn Patton May, 2008

Some premises: Evaluation is part of initial program design, including conceptualizing the theory of change Evaluator’s role is to help users clarify their purpose, hoped-for results, and change model. Evaluators can/should offer conceptual and methodological options. Evaluators can help by questioning assumptions. Evaluators can play a key role in facilitating evaluative thinking all along the way. Interpretative dialogue is critical. Designs can be emergent and flexible. Michael Quinn Patton May, 2008

Three ways of conceptualizing and mapping theories of change
Linear Newtonian causality Interdependent systems relationships Complex nonlinear dynamics Michael Quinn Patton May, 2008

Linear Logic Model INPUTS (people, materials)
ACTIVITIES (processes)  OUTPUTS  OUTCOMES  CHANGES IN PEOPLES LIVES  IMPACTS  CHANGES IN COMMUNITIES Michael Quinn Patton May, 2008

Resistance to Change Compelling Vision
Pushing Force (Non-directional) Dissatisfaction with the Status Quo (Inertia) a.k.a. “Cost of Change” Resistance to Change Pulling Force (Directional) a.k.a. “Desirability of the end state” Compelling Vision Michael Quinn Patton May, 2008

Causal Diagram of Beckhard’s Change Formula
Pushing Force (Non-directional) Dissatisfaction with the Status Quo (Inertia) a.k.a. “Cost of Change” Resistance to Change Pulling Force (Directional) a.k.a. “Desirability of the end state” Compelling Vision Believability First Steps Michael Quinn Patton May, 2008

Feedback Michael Quinn Patton May, 2008

Step Two: THEN A MIRACLE OCCURS “I think you should be more explicit here in step two.” Michael Quinn Patton May, 2008

Systems Parts are interdependent such a change in one part changes all parts The whole is greater than the sum of the parts Focus on interconnected relationships Systems are made up of sub-systems and function within larger systems Michael Quinn Patton May, 2008

Systems Concepts in Evaluation – An Expert Anthology. 2006.
Bob Williams and Iraj Imam AEA Monograph, EdgePress/AEA Point Reyes CA. Michael Quinn Patton May, 2008

Understanding the Elephant from a Systems Perspective Michael Quinn Patton May, 2008

The relationship between what goes in and what comes out
What conceptual framework informs front-end evaluation work? Michael Quinn Patton May, 2008

Teen Pregnancy Program Example
Michael Quinn Patton May, 2008

Logic Model for Pregnant Teens Program 1. Program reaches out to pregnant teens 2. Pregnant teens enter and attend the program (participation) 3. Teens learn prenatal nutrition and self-care (increased knowledge) 4. Teens develop commitment to take care of themselves and their babies (attitude change) 5. Teens adopt healthy behaviors: no smoking, no drinking, attend prenatal clinic, eat properly (behavior change) 6. Teens have healthy babies (desired outcome) Michael Quinn Patton May, 2008

Teachers/ other adults
Systems web showing possible influence linkages to a pregnant teenager Teachers/ other adults Young pregnant woman's attitudes & behaviors Her parents & other family members Child's father & peers Prenatal program staff Her peer group Michael Quinn Patton May, 2008

Other community-based youth programs
Program systems web showing possible institutional influences affecting pregnant teenagers: SCHOOL SYSTEM Young pregnant women's attitudes & behaviors Prenatal Clinic and Hospital Outreach Church program Other community-based youth programs Other Systems -- welfare -- legal -- nutrition programs -- transportation -- child protection -- media messages Context factors -- politics -- economic incentives -- social norms -- culture -- music Youth Culture Michael Quinn Patton May, 2008

Using Different System Lenses to Understand a “particular” System
Economic System Inputs/Outputs Cost/Waste/Value/Benefits Customers/Suppliers Political System Power Governance Citizenship Equity Biologic System Emergence Coordination/synergy Structure, Process, Pattern Vitality Sociologic System Relationships Conversations Interdependence Loose-tight coupling Meaning/sense Anthropologic System Values Culture/Milieu SYSTEM DIMENSIONS Information System Access Speed Fidelity/utility Privacy/security Storage Mechanical / Physical System Flow Temporal Sequencing Spatial Proximities Logistics Information Psychological System Organizing Forces Field Ecological/Behaviour Settings Michael Quinn Patton May, 2008

Complex Dynamic Systems Configuration

HIV/AIDS Example Hits every system: health, family, social, religious, economic, political, community, international Requires multiple interventions on multiple fronts in all subsystems simultaneously Resulting reactions, interactions, consequences dynamic, unpredictable, emergent, and ever changing Michael Quinn Patton May, 2008

Situation Recognition Appropriate Evaluation Designs
Challenges: Situation Recognition and Appropriate Evaluation Designs Michael Quinn Patton May, 2008

The nature of EXPERTISE Michael Quinn Patton May, 2008

Contingency-based Evaluation
Situational analysis & responsiveness Context sensitivity Clarify and focus on intended users: stakeholder analysis Clarify and focus on intended uses Methodological appropriateness Criteria for evaluating the evaluation: credibility, meaningfulness Michael Quinn Patton May, 2008

Seeing Through A Complexity Lens
“You don't see something until you have the right metaphor to let you perceive it”. Thomas Kuhn Michael Quinn Patton May, 2008

Complex Nonlinear Dynamics
Nonlinear: Small actions can have large reactions. “The Butterfly Wings Metaphor” Emergent: Self-organizing, Attractors Dynamic: Interactions within, between, and among subsystems and parts within systems can volatile, changing Getting to Maybe: Uncertainty, unpredictable, uncontrollable Michael Quinn Patton May, 2008

Simple, Complicated, Complex, Chaotic and Disordered Behaviours
SNOWDEN’s CYNAFIN FRAMEWORK Simple, Complicated, Complex, Chaotic and Disordered Behaviours Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

Conceptual Options Simple Complicated Complex Michael Quinn Patton May, 2008

Types of Community Issues The Stacey Matrix
Far from Agreement Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Simple Far from Agreement Simple Plan, control Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Inter-relationships between elements : Tight, centralised connections.
Simple (Known) Inter-relationships between elements : Tight, centralised connections. Anyone can see the things the way they are. Very simple linear cause and effect. Everyone knows the right answer within the current context (which of course may not be self-evident or known to others – hence importance of context). Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

Simple Complicated Complex
Following a Recipe A Rocket to the Moon Raising a Child The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Michael Quinn Patton May, 2008

Complicated (Knowable)
Inter-relationships between elements: Relationships are looser but still clustered around a central core. Cause and effect is non-linear. Relationships able to be modelled and able to predicted. An expert would know the right answer(s) Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

Technically Complicated
Far from Agreement Technically Complicated Experiment, coordinate expertise Simple Plan, control Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Michael Quinn Patton May, 2008

Socially Complicated Socially Complicated Agreement Certainty Far from
Build relationships, create common ground Agreement Simple Plan, control Technically Complicated Experiment, coordinate expertise Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Socially complicated Implementing human rights agreements, like gender equity or outlawing child labor Environmental Initiatives Many different and competing stakeholders Diverse vested interests High stakes Michael Quinn Patton May, 2008

Socially complicated situations pose the challenge of coordinating and integrating many players Michael Quinn Patton May, 2008

Stakeholder Mapping High Interest/ Low Power THE INVOLVED High Power THE PLAYERS THE CROWD Low interest/ CONTEXT SETTERS Low Interest/ Michael Quinn Patton May, 2008

Inter-relationships between elements:
Complex Inter-relationships between elements: Centre is loosely connected to network. Cause effect difficult to understand in current setting. Situation understandable only in retrospect. Not predictable. Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

Know When Your Challenges Are In the Zone of Complexity
Far from Systems Thinking Relationship Building Collaboration Good Enough Vision Chunking Around Drivers Minimum Specifications Multiple Actions Adaptability & Organic Socially Complicated Build relationships, create common ground Zone of Complexity Agreement Simple Plan, control Technically Complicated Experiment, coordinate expertise Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Following a Recipe A Rocket to the Moon Raising a Child Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Every child is unique Uncertainty of outcome remains The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Michael Quinn Patton May, 2008

Cause and effect unknowable, unattributable even in retrospect.
Chaotic (Unordered) Cause and effect unknowable, unattributable even in retrospect. No right answer(s) Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

Know When Your Challenges Are In the Zone of Complexity
Chaos Massive Avoidance Far from Socially Complicated Build relationships, create common ground Agreement Zone of Complexity Simple Plan, control Technically Complicated Experiment, coordinate expertise Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Complicated Complex Simple
Following a Recipe A Rocket to the Moon Raising a Child Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipe notes the quantity and nature of “parts” needed Recipes produce standard products Certainty of same results every time Michael Quinn Patton May, 2008

Michael Quinn Patton TEI 2008 Michael Quinn Patton May, 2008

Global Economic Complexity
Arthur Greenspan, Final speech to world’s Central Bankers, Jackson Hole, Wyoming August 26, 2005 Michael Quinn Patton May, 2008

“In the absence of a single variable, or at most a few, that can serve as a reliable guide, policymakers have been forced to fall back on an approach that entails the interpretation of the full range of economic and financial data.” Michael Quinn Patton May, 2008

“Despite extensive efforts to capture and quantify what we perceive as the key macroeconomic relationships, our knowledge about many critical linkages is far from complete and, in all likelihood, will remain so. Every model, no matter how detailed or how well conceived, designed, and implemented, is a vastly simplified representation of the world, with all of the intricacies we experience on a day-to-day basis.” Michael Quinn Patton May, 2008

“We all temper the outputs of our models and test their results against the ongoing evaluations of a whole array of observations that we do not capture in either the data input or the structure of our models. We are particularly sensitive to observations that appear inconsistent with the causal relationships of our formal models.” Michael Quinn Patton May, 2008

Oct 8, 2007 on The Daily Show “I was telling my colleagues the other day...I’d been dealing with these big mathematical models for forecasting the economy, and I’m looking at what’s going on the last few weeks and I say, “Y’know, if I could figure out a way to determine whether or not people are more fearful, or changing to euphoric... I don’t need any of this other stuff. I could forecast the economy better than any way I know. The trouble is, we can’t figure that out. I’ve been in the forecasting business for 50 years, and I’m no better than I ever was, and nobody else is either.” Alan Greenspan. Michael Quinn Patton May, 2008

The Age of Turbulence: Adventures in a New World, 2007, by Alan Greenspan Michael Quinn Patton May, 2008

Two evaluation locations
for Evaluating the Complex: Prospective and Retrospective Michael Quinn Patton May, 2008

Prospective Example The McGill-McConnell Leadership Program Example Simple elements Complicated elements Complex elements Michael Quinn Patton May, 2008

Evaluation: Pre-post data and documentation of learning
Simple outcomes Increase knowledge and skills of participants Evaluation: Pre-post data and documentation of learning Michael Quinn Patton May, 2008

organizational change
Complicated Impacts Change participants’ organizations Evaluation: Case studies of organizational change Michael Quinn Patton May, 2008

Complex Vision Infuse energy into the moribund not-for-profit (voluntary) sector Make the sector more dynamic Create network of leaders who actively engage in change Michael Quinn Patton May, 2008

Evaluating the Complex
Real time follow-up of network connections and actions Follow-up is an intervention Rapid feedback of findings permits infusion of resources in support of emergent outcomes Michael Quinn Patton May, 2008

Process Use Infusing evaluative thinking as a primary type of process use. Capacity-building as an evaluation focus of process use. Michael Quinn Patton May, 2008

Complex Interdependencies Michael Quinn Patton May, 2008

Insert action into the system Michael Quinn Patton May, 2008

EMERGENCE Michael Quinn Patton May, 2008

Retrospective Example
Advocacy Evaluation Final Push Campaign to overthrow the Juvenile Death Penalty Michael Quinn Patton May, 2008

In late 2003 several petitions on behalf of juvenile offenders facing the death penalty were filed with the U.S. Supreme Court. On January 26, 2004, the Court granted certiorari in Roper v. Simmons and the case was argued before the Court on October 13, The decision was announced March 1, The Court ruled 5-4 that capital punishment for juveniles was unconstitutional. Michael Quinn Patton May, 2008

In the brief window of time between when the Court agreed to hear the case and the case was argued, roughly nine months, a coordinated campaign was organized and funded aimed at overturning the juvenile death penalty. Organizing, public education, networking & communications continued through to the Court's ruling in March, 2005. Michael Quinn Patton May, 2008

Retrospective Evaluation
To what extent, if at all, was the Court’s decision influenced by the campaign? Modus Operandi or General Elimination Method Michael Quinn Patton May, 2008

1. Strong high capacity coalitions. Working through coalitions is a common centerpiece of advocacy strategy. Strong national-state-grassroots coordination. Effective policy change coalitions in the United States have to be able to work bottoms-up and top-down, with national campaigns supporting and coordinating state and grassroots efforts, while state efforts infuse national campaigns with local knowledge and grassroots energy. Strengthening strong national-state coordination is part of coalition development and field building. Disciplined and focused messages with effective communications. Effective communications must occur within movements (message discipline) and to target audiences (focused messaging). Strengthening communications has been a key a key component of advocacy coalition building. 4. Solid research and knowledge base. The content of effective messages must be based on solid research and timely knowledge. In the knowledge age, policy coalitions must be able to marry their values with relevant research and real time data about dynamic policy environment. 5. Timely, opportunistic lobbying and judicial engagement. The evaluation findings emphasize that effective lobbying requires connections, skill, flexibility, coordination, and strategy. 6. Collaborating funders engaged in strategic funding. Effective funding involves not only financial support, but infusion of expertise and strategy as part of field building. Michael Quinn Patton May, 2008

Campaigns & Grassroots Strategy and Implementation
Targeted State Campaigns & Grassroots Organizing Effective Litigation Credible, Useful and Amicus Up-to-date Research Briefs United Focused Coalition Communications Partners Campaign Knowledgeable Funders Omnibus Coordinated Integrated Strategy and Implementation Michael Quinn Patton May, 2008

Overall Lesson Learned for Effective Advocacy
In essence, strong national/state/grassroots coordination depends on having a high capacity coalition. A solid knowledge and research base contributes to a focused message and effective communications. Message discipline depends on a strong coalition and national-state coordination, as does timely and opportunistic lobbying and judicial engagement. To build and sustain a high capacity coalition, funders must use their resources and knowledge to collaborate around shared strategies. These factors in combination and mutual reinforcement strengthen advocacy efforts. In classic systems framing, the whole is greater than the sum of parts, and the optimal functioning of each part is dependent on the optimal integration and integrated functioning of the whole. Michael Quinn Patton May, 2008

Funders/ Strategic Funding
SIX INTERCONNECTED FACTORS, DYNAMICALLY INTERACTING, THAT STRENGTHEN ADVOCACY Solid Knowledge & Research Base Disciplined Focused Message/ Effective Communications Collaborating Funders/ Strategic Funding Michael Quinn Patton May, 2008

The interdependent system of factors that contribute to
Strong, high capacity coalition Targeted timely lobbying National state grassroots coordination The interdependent system of factors that contribute to effective advocacy and change Relevant research Disciplined focused message Collaborating funders/ strategic funding Michael Quinn Patton May, 2008

Dealing with the Unexpected and Unpredicted

Photo by Lynsey Gornick Michael Quinn Patton May, 2008

Creative Challenge Situational adaptability: Contingency-based evaluation Appropriateness --Using standard forms of evaluation and -- Going beyond standard forms when appropriate and useful Michael Quinn Patton May, 2008

Being Open: How hard is this to do?
My Colorado experience…. Michael Quinn Patton May, 2008

“The range of what we think and do is limited by what we fail to notice. And because we fail to notice that we fail to notice, there is little we can do to change, until we notice how failing to notice shapes our thoughts and deeds.” Scottish psychiatrist, R. Laing Michael Quinn Patton May, 2008

Paradigms and Lenses The importance of interpretive frameworks Complexity as an interpretive framework Michael Quinn Patton May, 2008

Getting to Maybe: How the World Is Changed? 2006 Frances Westley, Brenda Zimmerman, Michael Q. Patton Random House Canada, Michael Quinn Patton May, 2008

Complex Situations Highly emergent (difficult to plan and predict) Highly dynamic, rapidly changing Relationships are interdependent and non-linear rather than simple and linear (cause-effect) Michael Quinn Patton May, 2008

? Zone of Complexity Far from Agreement Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Far from Agreement Search for root cause Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Search for systemic features
Far from Agreement Search for systemic features Close to Close to Certainty Far from Michael Quinn Patton May, 2008

Contingency-based Developmental
Evaluation Michael Quinn Patton May, 2008

Improvement versus Development Michael Quinn Patton May, 2008

Beyond just Summative and Formative Michael Quinn Patton May, 2008

Static Accountability
Beyond Static Accountability Models Michael Quinn Patton May, 2008

Example of an emergent option: Developmental Evaluation Michael Quinn Patton May, 2008

DEVELOPMENTAL EVALUATION DEFINED
Evaluation processes, including asking evaluative questions and applying evaluation logic, to support program, product, staff and/or organizational development. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation and intentional change. The evaluator's primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and facilitate data-based decision-making in the developmental process. Michael Quinn Patton May, 2008

Other names Real time evaluation Emergent evaluation Action evaluation Adaptive evaluation Michael Quinn Patton May, 2008

CONTRASTS Traditional evaluations… Testing models Complexity-based, Developmental Evaluation… Supporting innovation and adaptation Michael Quinn Patton May, 2008

Developmental Evaluation… Provide feedback, generate learnings, support direction or affirm changes in direction in real time Traditional Evaluation… Render definitive judgments of success or failure Michael Quinn Patton May, 2008

Developmental Evaluation… Provide feedback, generate learnings, support direction or affirm changes in direction Develop new measures and monitoring mechanisms as goals emerge & evolve Traditional Evaluation… Render definitive judgments of success or failure Measure success against predetermined goals Michael Quinn Patton May, 2008

Traditional Developmental Evaluation… Evaluation… Evaluator external, independent, objective Evaluator part of a team, a facilitator and learning coach bringing evaluative thinking to the table, supportive of the organization’s goals Michael Quinn Patton May, 2008

Traditional Developmental Evaluation… Evaluation…
Evaluator determines the design based on the evaluator’s perspective about what is important. The evaluator controls the evaluation. Evaluator collaborates with those engaged in the change effort to design an evaluation process that matches philosophically and organizationally. Michael Quinn Patton May, 2008

Design the evaluation based on linear cause-effect logic models Design the evaluation to capture system dynamics, interdependencies, and emergent interconnections Michael Quinn Patton May, 2008

Aim to produce generalizable findings across time & space . Aim to produce context-specific understandings that inform ongoing innovation Michael Quinn Patton May, 2008

Accountability focused on and directed to external authorities and funders. Accountability centered on the innovators’ deep sense of fundamental values and commitments – and learning. Michael Quinn Patton May, 2008

Accountability to control and locate blame for failures Learning to respond to lack of control and stay in touch with what’s unfolding And thereby respond strategically Michael Quinn Patton May, 2008

Evaluation often a compliance function delegated down in the organization Evaluation a leadership function: Reality-testing, results-focused, learning-oriented leadership Michael Quinn Patton May, 2008

Evaluation engenders fear of failure. Evaluation supports hunger for learning. Michael Quinn Patton May, 2008

Conditions High innovation Development High uncertainty Dynamic Emergent Michael Quinn Patton May, 2008

SenseMaker software Dave Snowden, Founder of Cognitive Edge, former Director of Knowledge Management at IBM SenseMaker can code and map 95,000 stories in 24 hours See the world as others see it; anti-terror applications. See the quantitative patterns in the meta-data with qualitative context and meaning Michael Quinn Patton May, 2008

PANARCHY MODEL Michael Quinn Patton May, 2008

Routine Change Conservation K Foreloop Growth r Michael Quinn Patton May, 2008

Turbulent Change Release  Backloop Reorganization  Michael Quinn Patton May, 2008

Adaptive Cycle Conservation K Foreloop Release  Reorganization  Backloop Growth r Michael Quinn Patton May, 2008

Mature product and scaling up Exploration phase
Mature product and scaling up Exploration phase Stored System Capacity Building Leadership/initiative Phases of Technological and Social Innovation

1 Creative Destruction Little CAPITAL STORED Much
The birth, growth, destruction and renewal of a forest Little CAPITAL STORED Much 1 Creative Destruction Weak CONNECTEDNESS Strong

2 1 Creative Destruction Renewal/Exploration Renewal/Exploration
The birth, growth, destruction and renewal of a forest 2 Renewal/Exploration Renewal/Exploration Little CAPITAL STORED Much 1 Creative Destruction Weak CONNECTEDNESS Strong

3 Exploitation Little CAPITAL STORED Much Weak CONNECTEDNESS Strong
The birth, growth, destruction and renewal of a forest Little CAPITAL STORED Much 3 Exploitation Weak CONNECTEDNESS Strong

4 Conservation 3 Exploitation Little CAPITAL STORED Much
The birth, growth, destruction and renewal of a forest 4 Conservation Little CAPITAL STORED Much 3 Exploitation Weak CONNECTEDNESS Strong

DEVEOPMENTAL EVALUATION SUMMATIVE
DEVEOPMENTAL EVALUATION SUMMATIVE Stored FORMATIVE HARVESTING LESSONS Phases of Technological & Social Innovation

Taking Emergence Seriously
Beyond “unanticipated consequences” to genuine openness Michael Quinn Patton May, 2008

SURPRISES The psychologist Baruch Fischhoff wrote: "The occurrence of an event increases its reconstructed probability"—in other words, surprises are psychologically untenable in some ways, and we reshape our memories and expectations until we believe that the surprising event was, in fact, likely. Michael Quinn Patton May, 2008

The Epistemology of Surprise MALCOLM GLADWELL We hate surprises. We try to erase them from our memory. This is part of what keeps us sane. If, after all, we were always fully aware of the possibility of completely unpredictable events, would we be able to walk out the front door in the morning? Would we ever invest in the stock market? Would we have children? Generally speaking, people who have an accurate mental picture of why and how things happen tend to occupy mental hospitals—or, at the very least, a psychiatrist's office…. Michael Quinn Patton May, 2008

Information is like a stream teeming with fish, and if you stick out a net you'll collect something— but to decide what information is consequential. How does one do that? Michael Quinn Patton May, 2008

At the heart is the process of distinguishing signal from noise. How is that done? I have no idea, nor does anyone, I think, who isn't a seasoned analyst. Pattern recognition is something that comes only with experience. It's a matter of intuition, as much as anything. People always want to reduce this sort of thing to a formula, or a system, and I'm not sure you can do that. I suspect that there are some artificial-intelligence systems that can help to sort through certain kinds of data. But that could only be a first cut, and eventually human judgment has to be involved. Michael Quinn Patton May, 2008

What role can evaluation play with complex dynamic innovations? Michael Quinn Patton May, 2008

Challenge: Matching the evaluation process and design to the nature of the situation: Contingency-based Evaluation Michael Quinn Patton May, 2008

References Getting to Maybe: How the World Is Changed? Frances Westley, Brenda Zimmerman, Michael Q. Patton, Random House Canada, 2006 Utilization-Focused Evaluation, 4th ed., Michael Quinn Patton, Sage, 2008. Michael Quinn Patton May, 2008

Evaluating the Complex: Getting to Maybe Michael Quinn Patton

Similar presentations

Presentation on theme: "Evaluating the Complex: Getting to Maybe Michael Quinn Patton"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Evaluating the Complex: Getting to Maybe Michael Quinn Patton

Similar presentations

Presentation on theme: "Evaluating the Complex: Getting to Maybe Michael Quinn Patton"— Presentation transcript:

Similar presentations

About project

Feedback