1. How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary

2. Presentation outline Complex systems Complex interventions Role of program theory in complex evaluations Audience expectations Future Farming Strategy evaluation planning

3. Non linear Emergent Dynamical Adaptive Uncertain Co-evolutionary Complex systems Patton, M. (2010). Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, NY, p. 8.

4. Victorian agriculture system

5. Governance and implementation Simultaneous causal strands Alternative causal strands Non-linearity and disproportionate outcomes Emergent outcomes Complex interventions Rogers, P. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1), p. 32.

6. The Future Farming Strategy (FFS)

7. Interactions Complex interventions are events in systems. It is not the intervention itself but the context or setting into which it is introduced and with which it interacts which is the most significant aspect of the complexity. Hawe, P., Shiell, A. and Riley, T. (2009). Theorising interventions as events in systems. American Journal of Community Psychology, 43, 267-276.

8. Role of program theory

9. InputsOutputs Transformational potential

10. Future Farming Strategy logic model

12. Audience expectations Judgements of adequacy ‘Global’ net benefits Causal inference – level of confidence Quantification “The tension between the systematic evaluation approaches proposed by evaluation experts and the impact measurements required by clients for accountability purposes is not going to be solved easily.” Molas-Gallart, J. and Davies, A. (2006). Toward theory-led evaluation: The experience of European science, technology, and innovation policies. American Journal of Evaluation, 27(1), p. 78.

13. Evaluation planning Purpose Examine if FFS has made progress towards improving the productive, competitive and sustainable status of Victorian farm businesses Assess the benefits of Victorian Government investment in the FFS Provide opportunities for learning and improvement in whole-of-government strategy implementation and delivery to guide future policy and program development.

14. Evaluation planning Implications of complexity Outcomes definition Alignment between Action and strategy level outcomes Forms of evidence Emergent evaluation

15. Everything should be made as simple as possible, but not simpler. Albert Einstein, 1879 – 1955 Reality is complex. Nicoletta Stame, 2004 The art of dealing with the complicated and complex real world lies in knowing when to simplify and when, and how, to complicate. Patricia Rogers, 2008... and finally...