Federal Policy and Program Effectiveness Information: who can deliver? A presentation to the Canadian Evaluation Society June 3, 2003 Vancouver Prepared by Michael Obrecht Canadian Institutes for Health Research

Key Points of the Presentation 1.There is lack of clarity world-wide about the nature of effectiveness information 2.Canadian federal departments and agencies tend to focus on the effectiveness of relatively narrow program entities 3.An arms-length yet participatory effectiveness assessment process should be considered

Background: points advanced last year No major arms-length determination of effectiveness of Canadian federal programming Auditor General’s mandate requires a focus on economy and efficiency -AG looks for presence of systems for measuring effectiveness but does not assess their products Waste and inefficiency are captured, but not necessarily relative effectiveness

Effectiveness can be missed when Economy and Efficiency are the focus Effectiveness model may be overlooked Effectiveness may be overshadowed Ineffectiveness may be overlooked Bad policy or program revealed Economy and efficiency good bad Effectiveness

Background (continued) AG has independence, authority, power and resources But even if mandated for effectiveness assessment may be handicapped by: –Fault-finding orientation –Focus on spending rather than outcomes –Top-down, non-participative approach –Positivistic view of government business

Background (concluded) Last year (2002) we advanced the idea of a federal Evaluator General focused on generating effectiveness information on major policies and programs Have since then further investigated the relationship between evaluation and audit in theory and in practice in governments around the world

Point 1: Worldwide lack of clarity about the nature of effectiveness information In many jurisdictions, ongoing performance information and long-term effectiveness information are not clearly differentiated

Differentiating Effectiveness and Performance Information Effectiveness InformationOngoing Performance Information Complex, multi-dimensionalSimple, uni-dimensional WholisticParticulate Difficult to generateRelatively easy to collect Rich in meaningContext-dependent meaning Supports varying interpretationsLimited range of interpretation Decision-relevantDescriptive information Covers alternativesCovers just the entity

Examples of Focus on Performance Indicators United States: Government Performance and Results Act (GPRA) Canada: Improved Estimates Process – Annual Performance Reporting (DPR)

Drawbacks of Focus on Performance Measures One-way information flow Selective information Questionable targets Devalued process General shallowness/ unsuitability for decision-support

Unanswered Questions What kind of performance information do Parliamentarians want? Is ongoing performance information enough? Are there federal jurisdictions in which effectiveness information is generated and used?

Point 2: Internal evaluations by departments and agencies are of limited use externally As spenders of funds, departments and agencies have some motivation to understand the effectiveness of their discrete program mechanisms or activities As receivers of funds, they have little motivation to evaluate major programs or policies that affect their funding

Unanswered Question To what extent are cross-cutting evaluations of horizontal programs providing Parliament with decision-relevant effectiveness information?

Point 3: A participatory yet arms length effectiveness assessment process should be considered Effectiveness information is value-laden Stakeholders value systems must be taken into account when generating effectiveness information if it is to be useful Public representatives and public interest groups must be involved in the design of effectiveness studies on programs or policies in which they are interested

Arms-length Management Required Managing multi-stakeholder effectiveness studies requires –Dispassionate interest – objectivity –High credibility –Deep commitment – resources Should it be the Auditor General?

AG Management of Effectiveness Studies – the points in favour High credibility Arms-length from government World-wide trend of increased interest of auditors in effectiveness assessment Recruitment of senior evaluation experts by AG Intensive training in evaluative methodologies is feasible – example: US General Accounting Office (GAO) Long history of federal audit function

AG Management of Effectiveness Studies – the points against Past focus on fault-finding – and current power resides in fault-finding approach Authoritarian rather than participative Pervading positivist view Parliament has in the past decided against mandating AG for effectiveness assessment Federal separation of audit and evaluation

Idea of an Evaluator General for Canada Still alive - no clear world evidence of in- depth effectiveness information from federal audit functions Innovative and would fit the Canadian situation Would provide balance and oversight Best empowered through legislation