Using evaluations to Inform recommendations for reform CEF SEMINAR Using evaluations to Inform recommendations for reform THURSDAY 20 APRIL 2017 #ipaacef

Pierre Skorich Chair CEF Facilitation Group #ipaacef

Evaluation – a valuable source of evidence Jenny Gordon Principal Adviser Research Productivity Commission Canberra Evaluation Forum 20 April 2017

Recommendation That [insert responsible party here] should undertake a rigorous cost-benefit analysis before they [insert investment or policy here]

The evidence hierarchy Meta analysis of evaluations Ex-post evaluations Randomised controlled trials Regression discontinuities Impact assessments Parameter estimates for models ‘First’ principle approaches Theoretically robust Empirically consistent

The application of different types of evaluations by the Commission Process evaluations Management information Implementation challenges Effectiveness evaluations Testing causation Cost-effectiveness evaluations Cost-benefit evaluation

The conceptual framework for policy development is cost-benefit

The insights from evaluations Effectiveness – what has worked Case studies – RoGS, OID Applicability – will it work Scalability Transferability Best option test modelling

But care is needed Extrapolation of trends and case studies Correlation versus causality Reporting bias Statistical significance versus empirical significance (magnitude) Functional form and omitted variable bias Time and place effects

Some simple things to make evaluations more useful Make them public Follow the cost-benefit framework approach (inputs, outputs, outcomes, impacts) Use a logframe approach for ex ante evaluations (make the assumptions about context clear) Document properly Adopt common definitions Don’t let perfect get in the way of good

Common language would help (A framework for measuring costs – from Deep and persistent disadvantage)

QUESTION AND ANSWER #ipaacef

Pierre Skorich Chair CEF Facilitation Group #ipaacef

Thank you for attending CEF SEMINAR Thank you for attending THURSDAY 20 APRIL 2017 #ipaacef