Can Feds KISS? Creating a Simple and Smart Strategic Execution and Evaluation “System” Despite Complex Mandates and Requirements Environmental Evaluator’s Network Concurrent Sessions June 23, 2011, 4-5 PM Room 310 Liz Davenport and Thanh Vo Dinh National Oceanic and Atmospheric Administration 1

 How a SMART planning, budgeting, execution, and evaluation system can better manage a complex portfolio  How better integration of performance monitoring, program evaluation, and assessing strategic progress can improve performance management  How Feds can KISS to ensure GPRA Modernization Act improves agency and government-wide performance 2

 NOAA manages about 12,800 employees, aided by about 4,000 contractors  NOAA has 6 Line Offices and 12 Staff Offices  There are 127 mandates and authorities applicable to NOAA’s mission  NOAA’s responsibilities span all 50 states, Guam, Puerto Rico, and the Pacific Island Region 3

4

 Here is SEE…under its first cycle of implementation... 5

6 1 Focus Annual Guidance Memorandum 2 Assess Corporate Portfolio 3 Request Budget 4 Reassess Corporate Portfolio 5 Execute Annual Operating Plans 6 Evaluate Progress Request Budget to Achieve Intended Investment Outcomes Evaluate Progress to Plan, Performance, Programs, and Strategic Priorities Assess Portfolio to Plan Investments Reassess Portfolio to Optimize Performance in Operational Plans Execute Operating Plans to Achieve Priority Outcomes Align Strategic Plan Implementation Plans Strategy Execution & Evaluation Framework Focus Attention to Achieve Priority Outcomes

Compliance with performance evaluation requirements Compliance and some performance data used to improve management Compliance and sporadic use of requirements to “message” and “manage” performance to improve outcomes Managers consistently use program evaluation requirements to define long- term outcomes, improve short- term success and achieve objectives Agency success is strongly aided by assimilated program evaluation and a performance management framework that makes optimal use of performance data for management 7

8 GPRA Modernization Act Authorizes Implementation to:  Foster coordinated, cross-cutting, government-wide planning ◦ Government-wide strategic goals with annual government-wide (Federal) performance plan  Engage Congress to identify management/performance concerns ◦ Agency strategic plans include Federal performance goals and timed for release with changes in Administration  Improve accountability for results and address management weaknesses ◦ Agency and separate Federal performance plans aligned; looks at past 5 years of performance data ◦ Performance information better utilized due to greater Administration and Congressional engagement ◦ Quarterly reviews reported at

 Adopt Standard, Federal Terms and Definitions  Ensure Agency Clarity Regarding: ◦ Program ◦ Performance Measurement ◦ Performance Measures ◦ Program Evaluation  Comply with New Standard: ◦ Government-wide/cross-agency performance plan, with agreed upon outcomes and accountability within a 2-year time frame 9

10  How can we define “program” for agency performance planning and reporting and for Federal performance planning and reporting, given the need to link to and show progress in achieving Federal goals?  How can we do evaluations and use them to provide feedback to our Agency for these new requirements?

 GAO Defined Types of Federal Program Evaluations Process (or Implementation) Evaluation – assesses extent to which a program is operating as intended…assesses program activities’ conformance to statutory and regulatory requirements, program design, and professional standards or customer expectations Outcome Evaluation – assesses extent to which a program achieves its outcome- oriented objectives…focuses on outputs and outcomes (including unintended effects) to judge program effectiveness but may also assess program process to understand how outcomes are produced Impact Evaluation – (type of outcome evaluation) assesses net effect of a program by comparing program outcomes with an estimate of what would have happened in absence of the program…employed when external factors are known to influence program’s outcomes…to isolate program’s contribution to achievement of objectives Cost-Benefit and Cost-Effectiveness Analyses – compare program’s outputs or outcomes with costs (resources expended) to produce them…when applied to existing programs, considered a program evaluation…cost-effectiveness analysis assesses the cost of meeting a single goal or objective and can be used to identify the least costly alternative for meeting that goal…aims to identify all relevant costs and benefits, usually expressed in dollar terms Performance Measurement and Evaluation, Definitions and Relationships, GAO SP 11

 Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?)  Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation? 12

 Theory of changes: Logic model for government-wide goals from collective actions  Performance accountability:  Conflict resolution and coordination of capabilities between partners  Strategic evaluation in concert with performance measurement and program evaluation 13

 Thanks for letting us share SEE and speculate about GPRAMA impacts on evaluation  Thoughts, experiences, best practices on…  How can we define program for agency performance planning and reporting and for Federal performance planning and reporting given the need to link to and show progress in achieving Federal goals?  How can we do evaluations and use them to provide feedback to our Agency for these new requirements?  Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?)  Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation? 14

15

 What are your thoughts, experiences, best practices on… 1. GPRAMA requirements on your Agency’s evaluation and performance management? 2. Communicating evaluation results, using evaluation to facilitate, increase relationships/interaction with Congress and stakeholders? 3. Implementing GPRAMA requirements on evaluation within the current budget environment? 4. How are you building evaluation capacity? (Linking evaluation with performance management and evaluating success/failure of strategic goals/plans)? 16

Effective compliance with evaluation requirements Learning culture through evaluation feedback Robust program management Better investments through adaptive management Valuable decision-making support tool Transparent performance, outcomes, and results Monitor progress toward achieving strategic goals SMART performance targets Improved accountability 17

Former Process 1.Revised at least every 3 years 2.Covered at least a 5-year period 3.Consulted with Congress 4.Involved other stakeholders 5.Submitted to OMB and Congress Amended Process 1.Revision every 4 years, approximately 1 year after a new Presidential term begins 2.Consultations with relevant Congressional committees at least every 2 years 3.Publication online and President and Congress notified Amended Content 1.Relationship to Federal government priority goals 2.Interagency coordination and collaboration 3.Identification of agency priority goals 4.Description of incorporation of Congressional input 18 Former Content 1.Mission statement 2.Strategic goals 3.Strategies and resources 4.Relationship to performance goals 5.External factors that could significantly affect the goals 6.Program evaluations

Former Process 1.Upcoming fiscal year 2.No set time frame by OMB 3.Submitted to OMB Amended Process 1.Covers 2 fiscal years (current/upcoming) 2.Concurrent with President’s Budget 3.Available online; President and Congress notified 19 Former Content 1.Performance goals in objective, quantifiable, measurable form 2.Cover all program activities 3.Strategies and resources 4.Performance measures 5.Basis for comparing actual results with performance goals 6.Means to verify/validate data Amended Content 1.Relationship to agency strategic and priority goals, and Federal government performance goals 2.Coordination and collaboration 3.Contributing programs and activities 4.Milestones 5.Goal leaders 6.Balanced set of measures 7.Data accuracy and reliability 8.Major management challenges 9.Low-priority programs

GPRA Modernization Act…Agency Quarterly Priority Progress Reviews ◦ At least quarterly, agency head, COO, and PIC review with goal leaders progress toward each priority goal ◦ Review involves contributors in and out of Agency:  Assess how programs and activities contribute to goal  Categorize goals by risk of not being achieved  For those at risk, identify strategies to improve performance 20

GPRA Modernization Act…Requires High Performance Goals with Expanded Performance Reporting ◦ Set quarterly progress reviews ◦ Require Congressional consultation and engagement in performance management ◦ Require a Performance Portal ◦ Reduce duplicative performance reporting by 10% 21

GPRA Modernization Act Senate Committee Suggests: ◦ Agency, OMB, and Relevant Congressional Committee Review of Agency Meeting Goals ◦ If Goals Are Not Met:  After 1 st FY, require a Goal Performance Improvement Plan  If not met after 2 nd FY, submit actions to improve performance (statutory changes, resource transfer, etc.)  If not met after 3 rd consecutive FY, must act within 60 days or face statutory changes or possible termination or reduction possible 22

23 1. Strategic Plan must include how program evaluations were used to establish or revise strategic goals and a schedule for future evaluations 2. Performance reports must include summary findings of program evaluations completed during the fiscal year covered by the report 3. Evaluation will benefit performance planning and goal setting as well as quarterly reviews of priority goals