1. Office of the Auditor General of Canada Modernizing Accountability A need for evaluation Presentation to the CES 2003 Annual Conference Vancouver John Mayne john.mayne@oag-bvg.gc.ca June 4, 2003

2. Office of the Auditor General2 Outline Modernizing accountability The accountability process Keys to effective accountability The many roles for evaluation

3. Office of the Auditor General3 Traditional accountability Accountability is not a simple concept –Its meaning & how it works often disputed –Traditionally tied to hierarchy Traditionally, there has been accountability for: –Resource inputs, activities and outputs –Adherence to rules and procedures –Actions taken – or not: “blame culture”

4. Office of the Auditor General4 Modernizing accountability Pressures for change in public sector management and governance –Focus on results – distinguishing outcomes from outputs –Increased use of partnering arrangements –Increased discretion, flexibility and innovation (risk-taking)

5. Office of the Auditor General5 Purposes of accountability Control against the abuse of power Provide assurance that activities are carried out as intended, with due regard for fairness, propriety and good stewardship Encourage improved performance by reporting on & learning from what works – or does not

6. Office of the Auditor General6 Accountability revisited Accountability is a relationship based on obligations to demonstrate, review, and take responsibility for performance, both the results achieved in light of agreed expectations and the means used.

7. Office of the Auditor General7 Principles of effective accountability Clear roles and responsibilities –Parties understand and agree upon them Clear performance expectations –Objectives, expected results & constraints A balance of expectations with capacities –Linked to each party’s capacity to deliver Credible reporting –Demonstrating achievements & learning –Balanced: both good news and bad – not just good Reasonable review of performance –with adjustments, corrections & consequences for individuals (rewards & sanctions)

8. Office of the Auditor General8 The accountability process

9. Office of the Auditor General9 Accountability framework Clear roles and responsibilities Clear and realistic expected performance Credible reporting requirements Effective mechanisms for review & adjustment In the context of transparency & public service values

10. Office of the Auditor General10 Holding to account How are those responsible being held to account? Reporting − credible, understandable and timely –Financial and non-financial results –Means used to achieve those results (fairness, propriety & good stewardship) –Both good news and bad Review & adjustment –Review & analysis of performance –Necessary adjustments: Something has to happen

11. Office of the Auditor General11 Keys to effective accountability Fostering a climate for valuing results Designing practical yet challenging accountability frameworks Reliable measurement of results, and means Learning not to blame; results-based learning Setting outcome performance expectations Reasonable accountability for outcomes Implementing accountability practices

12. Office of the Auditor General12 Setting expectations essential for accountability traditionally involves specifying a few clear and concrete output/outcome performance levels would be better to –present and discuss the results chain –agree on some clear and concrete outputs and outcomes within the results chain to be accomplished performance expectations results chains

13. Office of the Auditor General13 Accountability for outcomes Need to credibly demonstrate: 1.The extent to which the expected results were achieved 2.The contribution made by activities and outputs of the program to the outcomes 3.The learning and change that have resulted, and 4.The soundness and propriety of the means used.

14. Office of the Auditor General14 Outcomes and attribution Outcome results-focus requires credible assessment of program’s contribution Contribution analysis - not an exact science Logic and empirical evidence can provide a reasonable basis for attribution Looking for plausible association

15. Office of the Auditor General15 Evaluating accountability Is there an appropriate framework? –Roles and responsibilities –Expected performance –Reporting requirements –Mechanisms for review and adjustment Is the arrangement being held effectively to account? –Credible reporting –Closing the loop (reviewing and adjusting)

16. Office of the Auditor General16 Conclusion Move beyond blame Hold those responsible for public policy effectively to account –For the results – and improvement through learning –For the means used to achieve the results All based on sensible measurement (evaluation and monitoring) of results and contribution

17. Office of the Auditor General17 OAG references Modernizing Accountability in the Public Sector. Report of the Auditor General of Canada to the House of Commons, Dec 2002, Chapter 9. http://www.oag- bvg.gc.ca/domino/reports.nsf/html/20021209ce.html/$file/20021209ce.pdf Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly. OAG Discussion Paper, 1999 http://www.oag-bvg.gc.ca/domino/other.nsf/html/99dp1_e.html Also in Canadian Journal of Program Evaluation, Vol. 16, No. 1, 1-24. Reporting Performance to Parliament: Progress Too Slow. Report of the Auditor General of Canada to the House of Commons, 2000, Chapter 19. http://www.oag- bvg.gc.ca/domino/reports.nsf/html/0019ce.html/$file/0019ce.pdf Reporting on Outcomes: Setting Performance Expectations and Telling performance Stories. OAG Discussion Paper, 2003. http://www.oag- bvg.gc.ca/domino/other.nsf/html/200305dp1_e.html/$file/200305dp 1_e.pdf