1. Approach of the Office of the C&AG to Examining Value for Money Seamus McCarthy Deputy Director of Audit Office of the Comptroller and Auditor General Irish Evaluation Network Friday 14 December 2007

2. Objectives Place VFM examination work by OCAG in the context of accountability and evaluation structures in Ireland Outline how evaluation challenges are dealt with in VFM examination work Briefly survey some possible future developments for OCAG assessment of VFM

3. C&AG’s VFM reporting mandate C&AG may examine and report on –the economy and efficiency achieved by audited bodies in the acquisition, use and disposal of resources –the adequacy of the systems, procedures and practices used by audited bodies to evaluate the effectiveness of their operations C&AG may make such comparisons as he considers approporate (including comparisons of systems, procedures and practices)

4. Evolution of the mandate C&AG role is part of the Westminster model of government Original focus on financial audit –Appropriation accounts system –State agencies’ accounts Progressively more reports on cases of loss, waste or extravagance e.g. –Project cost overruns –Fishery harbours –Prison planning Project audits: 1990-1993 Comptroller and Auditor General (Amendment) Act, 1993

5. The accountability cycle Comptroller and Auditor General Committee of Public Accounts Departments/ agencies Report Dáil Éireann Report Information Public hearing Minister for Finance Information Minute

6. Public auditing principles Independence Objectivity/freedom from bias Wide scope of mandate Freedom to report publicly Authority

7. Certification audit vs Reporting Certification audit Statutory/obligatory Output is a certificate/pro forma Accounts submitted for audit Numbers based Standards set Set audit procedures Answers are clear Ongoing relationship with auditees’ staff Reporting Discretionary Output is a report Nothing formal to ‘audit’ Narrative/analysis based Standards must be developed No fixed procedures No ‘right’ answers New relationships with auditees’ staff

8. Organisation of OCAG C&AG Director of Audit Secretary and Director of Audit Secretary and Director of Audit Vote audits Vote audits Semi State audits Semi State audits VFM exams VFM exams Corporate Services Certification audit Certification audit Financial management reporting Value for money reporting

9. Strategic reporting issues Reporting governed by ethical/professional standards Quality assurance –Expert input –Internal review/audit of working papers –Challenge/clearance process –‘Cold’ review Resources –Internal staffing Developing capacity –Consultants Ensuring independence/objectivity Understanding of requirements for VFM reporting

10. Stages in the examination process Selection of examinations Planning Investigation Reporting Follow-up

11. Selection of examinations C&AG exercises discretion Strategic themes/priorities e.g. –Public sector management reforms –PPPs –Health sector Matters arising (from audits, controversies, etc) Consultation with PAC Review of potential reporting issues based on –Risks to value for money in the body under review (performance risks) –Potential added value of a C&AG report –Feasibility

12. Some recent reports Schools PPP Waste management in hospitals Rent supplement The Financial Regulator Follow up review of examinations carried out 1997- 2000 eGovernment Drug addiction treatment and rehabilitation

13. Examination planning Scope setting - bound by statutory mandate –Issues to be examined – may include financial management issues –Time frame –Financial scope Analytic framework development Liaison with auditee(s) Tests of data availability Time and budget planning Formal review of plan by C&AG

14. Socio/economic statistics Business targets Output measures Activity management Activity measures Capacity management Measures of resource use Budgeting Financial results Public sector business model Financial outturn Inputs (resources) Activities/processes Outputs Identify social needs Set objectives Budget Planned inputs Planned activities Planned outputs Planned impacts Planned results Impacts Results

15. Analytic framework Core VFM concepts (economy, efficiency, effectiveness) are very high level Risk-based prioritisation of key performance areas For each performance area, identify –Performance standard in terms of 3 Es (+ equity) –Performance measure(s) –Target(s) Focus on performance of relevant bodies – things they control or influence, rather than context indicators Framework set at outset, but may evolve in light of information

16. Engagement with auditees Objectives –Brief management about plans, and get their views –Identify data sources/data holders –Agree working procedures –Resolve difficulties –Inform management about progress –Share findings (no surprises principle) –Get the management perspective

17. Investigation Statutory access rights Data sources –Official documents –Literature search –Administrative data e.g. databases –Semi-structured interviews (with managers, staff, clients) –Questionnaires/surveys –Sampling of cases (e.g. for process review) Data handling procedures

18. Analysis Data checking/validation incl. system review Forms of analysis used –Exploratory data analysis –Descriptive statistics –Trend analysis –Regression –DCF-based financial models –Activity-based costing Review/audit of results Analysis of professional judgements results

19. Reporting Target audiences: members of the Dáil; the public; public service managers Style: non-specialist; concise but comprehensive; clear Conclude on examination questions Include –recommendations for improvement –examples of transferrable good practice Formal clearance of drafts –two (or more) rounds of clearance

20. Follow-up Publication of report Briefing/update for C&AG for appearance at PAC meeting PAC hearing Follow-up review or re-audit Potential impacts (during or post examination) –Expenditure savings –Greater efficiency –Improved service –Better systems for evaluating effectiveness

21. Possible future developments Review of C&AG’s legislative mandate Reporting of transferable good practice Ad hoc reference groups for specific evaluations Development of performance reporting (e.g. Output Statements, VFM reports) Changing structure of public sector – new agencies, inspectors, regulators, PPPs, community/voluntary sector, etc. –Revised performance models/accountability regimes –New issues e.g. gaps in service provision; boundary disconnection; contract/service level management

22. Discussion