1 Transport Canada Transports Canada Presentation for CES - Conference 2000 Presented by Jennifer Birch-Jones, Evaluation Manager Gail Young, Evaluation Manager Program Evaluation Transport Canada May 16, 2000 INTEGRATING PERFORMANCE MEASUREMENT AND POLICY/PROGRAM EVALUATION IN TRANSPORT CANADA Program Evaluation, Transport Canada May, 2000

2 Share Transport Canada’s (TC’s) evolving approach to integrating performance measurement and policy and program evaluation – spheres of influence – attribution – on-going performance monitoring – research on performance challenges ObjectiveObjective Page 2

3 Not just staying within budgets and meeting revenue targets Not just being very busy, all the time Not just producing a lot of high quality goods and services Not just having satisfied “customers” but making a difference in the real world...changing attitudes and behaviours, social and economic conditions, the state of the environment, the sustainability of natural resources, the state of the world Transport Canada’s Approach: Performance is… Page 3

4 Our evaluation unit is part of the policy shop - distinct from audit and consulting functions that reside in corporate services We look at program and policy impacts We have been designated as the centre of functional expertise for performance measurement, as well as evaluation Our strategic objectives include positioning ourselves to contribute to strategic decision making Transport Canada’s Approach: Add Value Page 4

5 When it comes to performance, managers and evaluators both want to When it comes to performance, managers and evaluators both want to Understand the policy or program logic Keep track of what’s happening and make changes where needed Figure out how policy/program interventions are making things happen - cause and effect Report performance - monthly, yearly, every 5 years... Transport Canada’s Approach: Work with Managers Page 5

6 Operational > internal operations > the “how” > activities & outputs Direct Control Behavioral Change > Changes in individuals and organizations in and organizations in - awareness - awareness - acceptance - acceptance - ability/capacity - ability/capacity - action/adoption - action/adoption > the “what” and the “who” > direct outcomes Direct Influence Operational > internal operations > the “how” > activities & outputs Direct Control State > the vision - the ideal world > benefits to the target population, the Canadian population, the Canadian public, the industrial public, the industrial sector, the resource sector, the resource > the “why” > long-term impacts Indirect Influence Behavioral Change > Changes in individuals and organizations in and organizations in - awareness - awareness - acceptance - acceptance - ability/capacity - ability/capacity - action/adoption - action/adoption > the “what” and the “who” > direct outcomes Direct Influence Operational > internal operations > the “how” > activities & outputs Direct Control Performance Story Logic - Spheres of Influence With many thanks to Steve Montague, PMN Page 6

7 Operation > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate Direct Control Operation > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate Direct Control Behavioral Change Target Audience > Rail Industry > General Public > NGOs > Passengers > OGDs > TSB Results > Awareness > Acceptance > Ability > Adoption / Action Direct Influence Operation > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate Direct Control State Vision > Best possible rail transportation system Safety Results > Protection of life, property and environment > Public confidence enhanced Indirect Influence Operation > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate Direct Control Behavioral Change Target Audience > Rail Industry > General Public > NGOs > Passengers > OGDs > TSB Results > Awareness > Acceptance > Ability > Adoption / Action Direct Influence Operation > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate Direct Control Rail Safety’s Performance Story Page 7

8 Safety Performance Story for Rail Crossings for Rail Crossings Safety Performance Story for Rail Crossings for Rail Crossings State Indirect Influence Direct Influence Operation Direct Control > Regulate > Educate > Incentives > Enforce Target Audience > Rail Industry > Vehicle Drivers > NGOs > Other Road Users > Provinces / Municipalities Results > Increased Road User Alertness (Awareness ) > Improved Crossing Infrastructure (Ability) > Adoption of Safe Practices by Rail Industry and Road Users (Action) Safety Benefits > Reduction in Rail Crossing > Reduction in Rail Crossing Accidents / Fatalities / Injuries Accidents / Fatalities / Injuries > Public Confidence Enhanced Natural Environment Media Social / Economic Factors Page 8

9 Safety Performance Story for Rail Crossing Contributions for Rail Crossing Contributions Safety Performance Story for Rail Crossing Contributions for Rail Crossing Contributions State Indirect Influence Direct Influence Operation Direct Control > Financial Contribution Contribution Target Audience > Provinces / Municipalities Results > Improved Crossing Infrastructure (Ability) Safety Benefits > Reduction in Rail Crossing > Reduction in Rail Crossing Accidents / Fatalities / Injuries Accidents / Fatalities / Injuries > Public Confidence Enhanced Natural Environment Media Social / Economic Factors Page 9

10 Examples of Measures for Rail Safety (See Attachment 1 for a more complete list) Page 10

11 Application to Policy A similar approach can be used to tell the performance story for major policies Can lead to some interesting observations about actual and potential government intervention to achieve policy objectives For example…mapping the spheres of influence for government’s economic oversight of the air navigation system (ANS) raises questions about government’s role in influencing economic variables Page 11

12 TC Monitoring ? CTAuser-initiatedappealsCTAuser-initiatedappeals TC Minister > board appts > approvals TC Minister > board appts > approvals Sector Behaviour CTAuser-initiatedappealsCTAuser-initiatedappeals TC Minister > board appts > approvals TC Minister > board appts > approvals TC Monitoring ? NAV CANADA > financially viable > compliant with leg/regs leg/regs - respects int’l - respects int’l agreements agreements - respects charging - respects charging principles principles - respects level of - respects level of service conditions service conditions > responsive to users on > responsive to users on - price & service issues - price & service issues - capital investment in - capital investment in new technology new technology > transparent > transparent USERS > satisfied with ANS price & service price & service > satisfied with openness of openness of NAV CANADA NAV CANADA State of Transportation NAV CANADA > financially viable > compliant with leg/regs leg/regs - respects int’l - respects int’l agreements agreements - respects charging - respects charging principles principles - respects level of - respects level of service conditions service conditions > responsive to users on > responsive to users on - price & service issues - price & service issues - capital investment in - capital investment in new technology new technology > transparent > transparent USERS > satisfied with ANS price & service price & service > satisfied with openness of openness of NAV CANADA NAV CANADA CTAuser-initiatedappealsCTAuser-initiatedappeals TC Minister > board appts > approvals TC Minister > board appts > approvals TC Monitoring ? > ANS proportion of air transportation costs minimized TC Contributes to Natural Environment Media Social / Economic Factors TC Influences TC Controls Spheres of Influence for ANS Economic Oversight Page 12

13 Measurement by Sphere of Influence STATE OPERATION BEHAVIOURAL Is the intervention still appropriate? (eval) (See Attachment 2 for more detail) efficiency ( PM) productivity (PM) quality (PM) timeliness (PM) Changes in variables: What happened? (PM) Effectiveness and success: Did we make a difference? Was it the expected difference? (eval) Cost-effectiveness and alternatives: Is there a better way? (eval) Changes in variables: What happened? (PM) Success: What was our contribution to changes at the state level? (eval) Indirect Influence Direct Influence Direct Control Page 13

14 Shows realistic accountabilities: managers are relieved to finally see it acknowledged that they don’t control or even directly influence everything - they readily accept the logic of spheres of influence Responds to an immediate management need to monitor, review, improve, report Creates a meaningful link between performance management and reporting to central agencies - no longer just feeding the beast Why it’s Working Page 14

15 Helps managers do the job they know they should be doing: monitoring the performance of their own programs and policies Lets evaluators do the job they do best – helping managers set up performance measurement systems – applying research expertise to the tough questions such as attribution Builds a community of practice – learning – changing – sharing Why it’s Working Continued Page 15

16 Objectivity, independence, co-option…need we say more? Hiring and training the “new” evaluator - we need facilitators with people skills who can apply shared leadership in a team setting…and be solid analysts as well Manager has to be ready and evaluator has to be flexible and adaptable - may not always get the perfect framework, measures or data Research to determine attribution is a methodological challenge that cannot always be met, given the resources available to evaluators ChallengesChallenges Page 16