Engaging with external evaluators Kim Grey 15 June 2017 Disclaimer – the views expressed are those of the author
How do we get - Right skills Right time Right problem? What’s right? In general – it means: The ‘right’ technical needs The relevant ‘decision points’, and A significant or strategic focus But - what’s ‘right’ might depend on circumstances
When might this vary? When doing evaluation in: Chaotic situations Complex Complicated Simple
Complicated situations Complex situations Technical solutions are available Stakeholders in relative agreement Problems are knowable and stable Complex situations Techniques under development Stakeholder views differ Problems not clearly understood, multiple options & emergence
Complicated situations Complex situations Diagnosis relatively straightforward Use checklists Complex situations Greater uncertainty Adaptation needed
What to do in areas of complex social policy? Suggestions from my experience: 1. Technical challenges: test new approaches together For example Seek partners willing to implement new methods Diversify your partnerships Try mixed method combinations And build uncommon skills Be prepared to share risk and develop bespoke approaches.
Recognise the need to develop a tailored response What's the strategy? Recognise the need to develop a tailored response Implement bespoke evaluation design Recommended reference: Forss, Marra & Schwartz (2011) Evaluating the Complex. Attribution, Contribution and Beyond. Comparative Policy Evaluation, Volume 18 Transaction Publishers
2. Seek common ground among stakeholders Ask: What needs do we share? What facts do we all need to see? and Strive to see the world from others’ points of view
3. Explore the problem together External evaluators aren’t the font of all knowledge They can’t read your mind. Share program insights, build on your stock of knowledge Seek methods that reveal the unexpected … and the unintended effects Be prepared to question your assumptions
Options for working with external evaluators in areas of complex social policy Summary: 1. Technical challenges: test new approaches together 2. Find common ground among stakeholders 3. Explore the problem together
Tools for starting off well… 1st 2nd Clear requirements Careful selection… Tender document State assumptions about the evaluand Share your preferences Ask for ideas and suggestions Tender assessment Consider panel membership and debate strengths and weaknesses Get references for projects involving the staff listed
How to continue? Talking Debating Negotiating
Take-home messages: Three things we need to do - Build a relationship Be prepared to discuss your epistemological assumptions Have the difficult conversations
Thank you Kim.grey@pmc.gov.au