Telling the Difference between Snake Oil and Good Oil; Evaluation Capacity Building in the public sector Bronte Koop, Principal Consultant, Strategy and Evaluation John Collins, Manager, Program Evaluation

COAG Reform Agenda: Human Capital Development National Partnerships Literacy and Numeracy Teacher Quality Low SES communities

Not getting what you expected from an evaluation

What do we mean by Snake Oil Staff as passive recipients Evaluation questions not addressed Use lots of words Recommendations to continue the program and evaluate it again Lack of evidence based recommendations for improvement No discussion of data or design limitations No discussion of alternative explanations for program effect Evaluators reluctant to address client concerns about clarity or focus

Useful information to guide decision making

So what’s the Good Oil Staff critical participants Provides answers to specific questions Balances qualitative with quantitative data Reports are just sufficient in length Conclusions follow from the evidence Has logical recommendations Data or design limitations are discussed Alternative explanations for program effect are discussed

How do we get there? teamwork commitment learning proactivity

What are we trying to change? Attitudes and values Knowledge and skills Organisational behaviour

Overcoming your fears is the first step

Principles for evidence based program management Demand evidence Examine the logic Treat the program as an unfinished prototype Embrace an attitude of wisdom Jeffrey Pfeffer and Robert Sutton on evidence based management

1.Executive leadership 2.Consulting Services 3.Resources 4.Professional development 5.Policy and operations 6.Networks 7.Strategic Oversight Where should be put our efforts? CREATING CHANGE

Locate near the top Recruit the knowledge, skills and motivation Build credibility Provide an ‘evaluation perspective’ Consulting Services

Resources If you build it, they will come

What have others done?

Accessible Resources

Practical Resources

Provide a starting point

Project Management

Evaluation & planning

Professional Development Time to stretch your thinking Induction Customized training to needs

Procurement: Expression of Interest Two part contracts: plan + do Executive summaries Interviews Hard questions Policy and Operations Think outside the square

Network, network, network

Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

Strategic objective 2011 “We will foster a culture of evaluation and review to better direct our services.”

Strategic Plan We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes...

Strategic Plan We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes... current investments will be evaluated and we will reinvest where appropriate.

Strategic Plan We take seriously our responsibility to invest public funds effectively and efficiently. Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes... current investments will be evaluated and we will reinvest where appropriate.

Represents a major cultural change.

Represents a major cultural change From an ‘accountability’ model to one centred on the corporate goal.

Represents a major cultural change From an ‘accountability’ model to one centred on the corporate goal.

How?

A marketing approach

A marketing approach Product life-cycle

A marketing approach Product life-cycle

A marketing approach Product life-cycle Customer

A marketing approach Product life-cycle Customer –Needs and wants

A marketing approach Product life-cycle Customer –Needs and wants

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity Competition

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity Competition

A marketing approach Product life-cycle Customer –Needs and wants The grudge purchase Brands and brand equity Competition 4 Ps – Profit Product Place Promotion

Priorities? To continue development & refinement of our ‘system’ To address changing the focus of our evaluation culture To move from ‘good oil’ to ‘best value’ oil

Disclaimer The views and interpretations expressed in this presentation are those of the authors and do not necessarily reflect those of the Department of Education and Children's Services or the Government of South Australia

Contact