1. Using evaluations to Inform recommendations for reform
CEF SEMINAR
Using evaluations to
Inform recommendations
for reform
THURSDAY 20 APRIL 2017
#ipaacef

2. Pierre Skorich Chair CEF Facilitation Group
#ipaacef

3. Evaluation – a valuable source of evidence
Jenny Gordon
Principal Adviser Research Productivity Commission
Canberra Evaluation Forum 20 April 2017

4. Recommendation
That [insert responsible party here] should undertake a rigorous cost-benefit analysis before they [insert investment or policy here]

5. The evidence hierarchy
Meta analysis of evaluations
Ex-post evaluations
Randomised controlled trials
Regression discontinuities
Impact assessments
Parameter estimates for models
‘First’ principle approaches
Theoretically robust
Empirically consistent

6. The application of different types of evaluations by the Commission
Process evaluations
Management information
Implementation challenges
Effectiveness evaluations
Testing causation
Cost-effectiveness evaluations
Cost-benefit evaluation

7. The conceptual framework for policy development is cost-benefit

8. The insights from evaluations
Effectiveness – what has worked
Case studies – RoGS, OID
Applicability – will it work
Scalability
Transferability
Best option test
modelling

9. But care is needed
Extrapolation of trends and case studies
Correlation versus causality
Reporting bias
Statistical significance versus empirical significance (magnitude)
Functional form and omitted variable bias
Time and place effects

10. Some simple things to make evaluations more useful
Make them public
Follow the cost-benefit framework approach (inputs, outputs, outcomes, impacts)
Use a logframe approach for ex ante evaluations (make the assumptions about context clear)
Document properly
Adopt common definitions
Don’t let perfect get in the way of good

11. Common language would help (A framework for measuring costs – from Deep and persistent disadvantage)

12. QUESTION AND ANSWER
#ipaacef

13. Pierre Skorich Chair CEF Facilitation Group
#ipaacef

14. Thank you for attending
CEF SEMINAR
Thank you for attending
THURSDAY 20 APRIL 2017
#ipaacef