Engaging with external evaluators Kim Grey 15 June 2017 Disclaimer – the views expressed are those of the author

How do we get - Right skills Right time Right problem? What’s right? In general – it means: The ‘right’ technical needs The relevant ‘decision points’, and A significant or strategic focus But - what’s ‘right’ might depend on circumstances

When might this vary? When doing evaluation in: Chaotic situations Complex Complicated Simple

Complicated situations Complex situations Technical solutions are available Stakeholders in relative agreement Problems are knowable and stable Complex situations Techniques under development Stakeholder views differ Problems not clearly understood, multiple options & emergence

Complicated situations Complex situations Diagnosis relatively straightforward Use checklists Complex situations Greater uncertainty Adaptation needed

What to do in areas of complex social policy? Suggestions from my experience: 1. Technical challenges: test new approaches together For example Seek partners willing to implement new methods Diversify your partnerships Try mixed method combinations And build uncommon skills Be prepared to share risk and develop bespoke approaches.

Recognise the need to develop a tailored response What's the strategy? Recognise the need to develop a tailored response Implement bespoke evaluation design Recommended reference: Forss, Marra & Schwartz (2011) Evaluating the Complex. Attribution, Contribution and Beyond. Comparative Policy Evaluation, Volume 18 Transaction Publishers

2. Seek common ground among stakeholders Ask: What needs do we share? What facts do we all need to see? and Strive to see the world from others’ points of view

3. Explore the problem together External evaluators aren’t the font of all knowledge They can’t read your mind. Share program insights, build on your stock of knowledge Seek methods that reveal the unexpected … and the unintended effects Be prepared to question your assumptions

Options for working with external evaluators in areas of complex social policy Summary: 1. Technical challenges: test new approaches together 2. Find common ground among stakeholders 3. Explore the problem together

Tools for starting off well… 1st 2nd Clear requirements Careful selection… Tender document State assumptions about the evaluand Share your preferences Ask for ideas and suggestions Tender assessment Consider panel membership and debate strengths and weaknesses Get references for projects involving the staff listed

How to continue? Talking Debating Negotiating

Take-home messages: Three things we need to do - Build a relationship Be prepared to discuss your epistemological assumptions  Have the difficult conversations

Thank you Kim.grey@pmc.gov.au