The Logic Model and Systems Thinking: Can They Co-Exist? American Evaluation Association Conference November 2009 Orlando, FL Robert F. Richard, Ph.D.

In theory there’s no difference between theory and practice. In practice there is. Yogi Berra

What are we here to think about? Logic Model background Systems Thinking background How do they mix/blend/co-mingle?

Logic Model

Program DevelopmentProgram Performance SEEC Practices KASA Reactions Participation Activities Resources SEEC Practices KASA Reactions Participation Activities Evaluate processes Evaluate impacts Assess needs/opportunities and program design Bennett’s Model

Assess needs Define Extension’s role and nature of commitment Assess resources and develop program Initiate networks and coalitions Provide educational experiences Consequences Louisiana 4-H Evaluability Assessment Program Logic Model Richard (1992)

Logical Linkages: Series of If-Then Relationships Program invests time & money Resource inventory can be developed Families will know what is available Families will access services Families will have needs met Inputs OutputsOutcomes IfThen If Then If Then If Then

Basically a description of what has to happen to get what we desire. A tool that describes the theory of change underlying an intervention.

Program reaches out to pregnant teens Pregnant teens enter and attend the program (participation) Teens learn prenatal nutrition and self care (increased knowledge) Teens develop commitment to take care of themselves and their babies (attitude change) Teens adopt healthy behaviors; no smoking, no drinking; attend prenatal clinic, eat properly (behavior change) Teens have healthy babies (desired outcome)

Pregnant teen’s knowledge, attitudes and behavior School system Church Other community based programs Youth culture Prenatal clinic and hospital outreach Prenatal program Other Systems Welfare Legal Nutrition programs Transportation Child protection Media messages Context factors Politics Economic incentives Social norms Culture Music

11 A Logic Model will tell you that the after school program is an activity and improved reading scores is an outcome. It might tell you that attendance at the after school program is an intermediate outcome.

12 But it won’t tell you that: “students need to attend after-school programs at least 3 days per week for a minimum of 60 days, and the curricula must focus on love of reading and literacy, IN ORDER FOR test scores to rise”

Systems Thinking

Systems Parts are interdependent such that a change in one part changes all parts The whole is greater than the sum of the parts Focus on interconnected relationships Systems are made up of sub-systems and function within a larger system

Systems concepts in evaluation Offers more effective ways of dealing with complexity and complex situations Develops new ways of understanding situations Pays attention to coalitions Pays attention to properties that emerge unexpectedly Helps identify leverage points Recognizes the evolutionary nature of programs

It is unwise to focus on one view or definition of a system without examining its relationship with another system Systems concepts can be particularly useful to evaluators in situations where rigorous rethinking, reframing and unpacking complex realities and assumptions

 Recipes produce standard products  Certainty of same results every time  Formulas are critical and necessary  High level of expertise in many specialized fields + coordination  High degree of certainty of outcome Following a recipe Simple Going to the moon ComplicatedComplex Raising a child  No instructions; difficult to predict results  Interdependent and non-linear  Multitude of external factors

Simple CauseEffect Complicated Results Complex Parts

The Stacey Matrix Agreement Close to Far from Close to Far from Certainty Complicated Simple ANR - prescriptive Complex Environmental issues Global warming Poverty

In systems work, richness implies that the whole can only be understood as a product of its parts plus the dynamic relationship between those parts Does our use of the Logic Model sometime ignore the dynamic relationship between parts?

…logic models are far more than templates for preparing plans of work. Without such understanding…Extension educators tend to focus on “filling out the forms” rather than using logic models as a framework for articulating program theory. Consequently the linkages between inputs, outputs and outcomes tends to be weak or unconfirmed. Rennekamp & Arnold (2009) Journal of Extension

More engaged parents Opportunities for parents to work with children Agent time Materials School space School Garden Body Walk 4-H club Assumptions Parents desire to be with their children Community is interested in gardening School is supportive of effort Barriers Little experience with gardening Absentee parents/grandparents as main provider Limited resources InputsOutputsOutcomes Alice Boucher Project

More engaged parents School GardenBody Walk4-H Club Parental involvement? Leader/co-leader? Club structure to support sense of belonging, group behavior How many meetings attended before change in behavior can be expected? Improved social skills Project enrollment? Improved nutrition How are lessons provided before, during and after? How much repetition is necessary? Are nutritious foods available locally? How are parents involved? In lessons, in actual event? When – evenings, weekends? Involve Master Gardeners? What is their knowledge of youth and adult education? When can parents be involved? Evenings, weekends? How much can be produced? How is it distributed? Do parents know how to prepare? Subject matter, Teamwork, Cooperation Alice Boucher Project

More engaged parents School GardenBody Walk4-H Club Improved nutrition Improved social skills Subject matter, Teamwork, Cooperation School GardenBody Walk4-H Club Improved nutrition Improved social skills Subject matter, Teamwork, Cooperation

School GardenBody Walk4-H Club Improved nutrition Improved social skills Subject matter, Teamwork, Cooperation More engaged parents How are parenting skills purposefully introduced for each program component? What do we really want to measure? Do we need to measure something for each program component? How long (years) a project is this? What will the school consider a success? Is there agreement as to what better parenting skills look like? What will each component consider a success?

In preparing for battle I have always found that plans are useless, but planning is indispensible. Dwight D. Eisenhower

So what do we have here? I think…

28 Logic models graphically illustrate program components, and creating one helps stakeholders clearly identify outcomes, inputs and activities Systems Thinking help us to create a theory of change to explain HOW and WHY the desired change is expected to come about

Logic Models require identifying program components, so you can see at a glance if outcomes are out of sync with inputs and activities, but they don’t show WHY activities are expected to produce outcomes Systems Thinking require justifications at each step – you have to articulate the hypothesis about why something will cause something else

…boundaries in human systems are massively entangled. An individual participates in multiple natural systems: family, church, community etc. Emergent patterns within each of those boundaries shapes the individual who, in turn, shapes patterns as they emerge in other contexts.

Pregnant teen’s knowledge, attitudes and behavior School system Church Other community based programs Youth culture Prenatal clinic and hospital outreach Prenatal program Other Systems Welfare Legal Nutrition programs Transportation Child protection Media messages Context factors Politics Economic incentives Social norms Culture Music

A common understanding of complexity is that it results from the number of elements and/or interactions in a system being beyond the capacity of the observer to readily understand. …even a small number of elements and/or interactions may be complex when they are in a dynamic and difficult to predict process of change.

More engaged parents School GardenBody Walk4-H Club Parental involvement? Leader/co-leader? Club structure to support sense of belonging, group behavior How many meetings attended before change in behavior can be expected? Improved social skills Project enrollment? Improved nutrition How are lessons provided before, during and after? How much repetition is necessary? Are nutritious foods available locally? How are parents involved? In lessons, in actual event? When – evenings, weekends? Involve Master Gardeners? What is their knowledge of youth and adult education? When can parents be involved? Evenings, weekends? How much can be produced? How is it distributed? Do parents know how to prepare? Subject matter, Teamwork, Cooperation Alice Boucher Project

…building an understanding that elements of a system interact through feedback loops, that causes often are not linked directly in time with effects, and that delays can cause unanticipated behavior are extremely valuable for both the program designers and evaluators.

Emerging complexity is characterized by disruptive change whose challenges include: 1.The solution to the problem is unknown 2.The problem statement itself is still unfolding 3.Who the key stakeholders are is not clear

36 It’s obvious by now: Systems Thinking, aka Theories of Change or Program Theory take a lot of work and time. On the other hand, Logic Models may sometimes be too simplistic to be helpful…..

Do you agree/disagree? Are we asking the right questions? Do we have the correct boundaries?

Just because something doesn't do what you planned it to do doesn't mean it's useless. Thomas A. Edison

Eoyang, G.H. (2007). Human Systems Dynamics: Complexity-Based Approach to a Complex Evaluation. In B. Williams & I. Iman (eds). Systems Concepts in Evaluation: An Expert Anthology (pp ). Point Reye, CA: EdgePress of Inverness. Frechtling, J. A. (2007). Logic Modeling Methods in Program Evaluation San Francisco, CA: Jossey-Bass Patton, M. (1997). Utilization Focused Evaluation (3 rd ed.). Thousand Oaks, CA: Sage Publications. Scharmer, O. (2007). Theory of U. Cambridge, MA: The Society for Organizational Learning, Inc. Westley, F., Zimmerman, B. & Patton, M. (2006). Getting To Maybe. Random House, Canada. Wholey, J., Hatry, H., Newcomer, K. (2004). Handbook of Practical Program Evaluation. San Francisco, CA: Jossey-Bass

Chaos, Confusion, Disorder…. My work here is done

Phone: Robert F. Richard, Ph.D. Thinking Guide Seven Oaks Consulting Helping you think P.O. Box 916 Scott, LA 70583