Evaluating the Performance of PBOs Helaina Gaspard Institute of Fiscal Studies and Democracy (IFSD) June 6-10, 2016 World Bank GN-PBO Annual Meeting Washington, DC USA

Why evaluate PBOs? 1.Support to parliament  Parliament’s scrutiny function derives from the Magna Carta Legitimacy and survivability of the office  Credibility of outputs; perceptions of stakeholders; and organizational factors can influence a PBO’s operation

Why a common evaluation framework?  PBOs are emerging globally – What works? What doesn’t?  An evaluation framework can work in two ways: 1.To assess the work of an existing PBO 2.To establish the parameters for a new PBO  A common evaluation framework can be applied based on context, inputs, outputs and outcomes.

An Evaluation Framework for PBOs CONTEXTINPUTSOUTPUTSOUTCOMES  Legislation  Sources of accountability  Defining stakeholders  Budget  Human resources  Access to data  Independence  Peer review of reports (scientific and methodological soundness)  Transparency gains  Assessment against counterfactuals  Assessment against peers  Stakeholder perceptions (via surveys)  Media impact analysis  Expert interviews

Case study - the 2014 UK OBR review  First review of the UK OBR  Context, input, output, outcome frame  Defined performance – better decision support and enhanced transparency, rather than being “right”  Provided advice to support organizational sustainability  Helped to build legitimacy or the fledgling institution

Context  Constitutional and political ecosystem.  Founding legislation; parameters of the PBO.  Legislation guarantees a minimum.

Points of Departure Civil Society Media Legislators Parliament Cabinet Civil Service Legislation Implementation PBO Sustainability Imperative (t 0)

Inputs  Resources: both human and monetary that enable a PBO to fulfill their mandate.  Access to data: can a PBO get what it needs from the executive branch to do its work?  Independence: is a PBO free from political influences to select its staff, head and fulfill its mandate?  Don’t let the numbers fool you – you can do a lot with a little.

Inputs of OECD IFIs. Source: Review of the United Kingdom’s Office for Budget Responsibility (2014), p. 32.

Independence of the OBR. Source: Data sourced from the Review of the United Kingdom’s Office for Budget Responsibility (2014), p. 34.

Outputs  The products or reports prepared and released by the PBO.  Evaluating outputs is a technical assessment of a PBO’s products.  Are they credible?  Are they methodologically sound?  Are they transparent?  Are they peer-reviewed?  Are they comparable to those of peers?

Process Defined Transparent Disciplined Economy Indicators assessed over multiple years Errors analysed by level, growth rate, contributions to growth and probabilistic estimates Public finances Details of receipts and spending assessed over 3 year period Changes due to economic and fiscal forecast, policy and classification assessed on annual basis Lessons learned Transparency drives agenda for forecast improvement Forecast evaluation transparency gains. Source: Review of the United Kingdom’s Office for Budget Responsibility (2014), p. 61.

Outcomes  Perceptions and impacts of a PBO matter for sustainability  Identify a PBO’s key stakeholders (from legislation and convention)  Survey them, interview them and track media results  A PBO can exist on paper but if it cannot communicate with stakeholders, it may become irrelevant  Assessing outcomes:  How do parliamentarians perceive the PBO? Do they leverage their outputs?  Does the PBO resonate in the media?

Factiva analysis of OBR media citations and mentions ( ). Source: Review of the United Kingdom’s Office for Budget Responsibility (2014), p. 72.

 Note: Parliamentary survey results from the OBR evaluation were unusable due to a lack of response rates  Only 52 responses out of a potential 1,427 (650 Members of Parliament and 777 Peers (House of Lords))  5 from MPs, 47 responses from Peers  A survey of Canadian parliamentarians (February 2015-August 2015) yielded a 20% response rate (considered statistically relevant)

Establishing a PBO with the evaluation framework  “Flip the framework on its head” and build evaluation into your design  It can imbue your office with credibility, motivation and institutional sustainability  Identify your desired outcomes; consider your context:  What inputs and outputs are required to fulfill your goals? ContextInputsOutputsOutcomes

Why a common framework?  Baselines  Define a basic common standard of institutional organization and products  What are the fundamental common attributes of PBOs?  Benchmarks  Establish peer groups for assessment and comparison  You know you have succeeded when…  Aspirational institutions  Models of successful PBOs can be useful examples for new institutions

Conclusion  Cement dries quickly – you must act to set precedents of practices  No legislation is perfect. Use your inputs and outputs to maximize your mandate and your outcomes for stakeholders  With strong leadership, you can maximize mandate by ensuring that the office’s products are relevant, their content authoritative and their release timely and foster sustainability

Thank you Institute of Fiscal Studies and Democracy University of Ottawa