Performance Measures for Clean Air 9th Annual Environmental Performance Summit October 3-5, 2007 Mike Hadrick, US EPA

2 Outline of Presentation A Matter of Perspective A Matter of Perspective Measure Types and Uses Measure Types and Uses Analytical Frameworks Analytical Frameworks Measures for Ground-level Ozone Measures for Ground-level Ozone Challenges Challenges Improvement Efforts Improvement Efforts Conclusions Conclusions Recommended Reading Recommended Reading

3 A Matter of Perspective

7 Measure Types and Uses

8 Measures for Lower Management  Output measures track activities and monitor progress and accomplishments in the short run.  Think of driving two days to a vacation spot and hitting milestones on the way...

9 Short Run Measures for Vacation Are we on schedule or running late? Are we where we planned to be, or did we hit some detours? Are we on schedule or running late? Are we where we planned to be, or did we hit some detours? What about our final destination? Will we still have the time there to do the things we planned? What about our final destination? Will we still have the time there to do the things we planned? Based on the above, do we need to adjust anything to have a nice vacation once we get there? Based on the above, do we need to adjust anything to have a nice vacation once we get there?

10 Measures for Upper Management Outcome measures track bigger picture or long- term trends – such as goal achievement. Back to the vacation...you arrived a couple days ago... Outcome measures track bigger picture or long- term trends – such as goal achievement. Back to the vacation...you arrived a couple days ago... How’s it going? Are we having fun yet? Was this a good idea? Or is it more like “well, we’re not doing this again!”? How’s it going? Are we having fun yet? Was this a good idea? Or is it more like “well, we’re not doing this again!”? There’s not much you can do about it now other then lessons learned for next year’s vacation. There’s not much you can do about it now other then lessons learned for next year’s vacation.

11 Simple Logic Model for Any Type of Program InputsOutputsIntermediateOutcomesEndOutcomes Hierarchy of Indicators – Specific to Environmental Programs Actions by Regulators Responses of Regulated Community or Society Changes in Pressure or Stressor Quantities Changes in Ambient Conditions Changes in Exposure, Body Burden, or Uptake Changes in Human Health or Ecological Condition Analytical Frameworks

12 Measures for Ground-level Ozone

13 Program Purpose and Role of Air Office Protect human health and the environment from the potential effects of air pollution associated with ozone. Protect human health and the environment from the potential effects of air pollution associated with ozone. Set health-based standards that protect with human health and the environment and assists states in implementing the standards. Set health-based standards that protect with human health and the environment and assists states in implementing the standards.

14 Simple Logic Model for Ozone EPA Inputs EPA Outputs IntermediateOutcomesEndOutcomes $$$FTE Establish air quality standards Develop technical info, methods, & guidance to implement Conduct oversight Implement federally if States do not Reduced emissions of air pollutants Lower ambient concentrations Fewer days with unhealthy air Reduced atmospheric deposition Reduced acidity of lakes and streams Reduction in prevalence and incidence of air pollution-related health effects Improved ecosystem condition

15 Hierarchy of Indicators for Ozone Actions by Regulators Response of Regulated Community or Society Changes in Pressure or Stressor Quantities Changes in Ambient Conditions Changes in Exposure, Body Burden, or Uptake Changes in Human Health or Ecological Condition Establish air quality standards Develop technical info, methods, guidance to implement Conduct oversight Implement federally if States do not Implement programs Compliance Take voluntary actions Implement alternatives Air pollutant or precursor emissions from various source sector categories Air pollutant concentrations Deposition of air pollutants Lake and stream acidity Prevalence and incidence of air pollution- related health effects Ecosystemcondition

16 Short Run Measures Timeliness of rulemakings, etc. (Deputy) Timeliness of rulemakings, etc. (Deputy) Status of State Implementation Plan development by states or review/processing by Regional Office (program office) Status of State Implementation Plan development by states or review/processing by Regional Office (program office) % of permits reviewed or issued within target timeframe (PART) % of permits reviewed or issued within target timeframe (PART) NO X reductions from electric utilities (program office) NO X reductions from electric utilities (program office)

17

18 Longer-term Indicators Mobile source emissions of ozone precursors (PART) Mobile source emissions of ozone precursors (PART) Emissions of ozone precursors from all source categories (ROE) Emissions of ozone precursors from all source categories (ROE) % reduction in population-weighted ambient concentration of ozone in monitored counties (PART) % reduction in population-weighted ambient concentration of ozone in monitored counties (PART) Ambient ozone concentrations (ROE) Ambient ozone concentrations (ROE)

19

20

21

22

23

24

25 End Outcome Indicators Asthma prevalence and mortality (ROE) Asthma prevalence and mortality (ROE) Childhood asthma prevalence and mortality (ROE) Childhood asthma prevalence and mortality (ROE) Chronic obstructive pulmonary disease (COPD) mortality (ROE) Chronic obstructive pulmonary disease (COPD) mortality (ROE) Cardiovascular disease (CVD) mortality (ROE) Cardiovascular disease (CVD) mortality (ROE)

26 Challenges Requirements of statutes, regulations, and court orders Requirements of statutes, regulations, and court orders Data not available in time for official reporting cycles Data not available in time for official reporting cycles Time between activity or expenditure and outcome Time between activity or expenditure and outcome

27 Challenges Who’s performance does measure actually capture? Who’s performance does measure actually capture? Linking activities to outcomes (causality) Linking activities to outcomes (causality) Measures overload Measures overload

28 Improvement Efforts 2004 NRC Recommendations: Air Quality Management in the United States 2004 NRC Recommendations: Air Quality Management in the United States EPA Air and R&D offices working together on Air Accountability Framework per NRC EPA Air and R&D offices working together on Air Accountability Framework per NRC EPA Indicators Initiative EPA Indicators Initiative Air office’s Measures Improvement Effort Air office’s Measures Improvement Effort

29 Conclusion Be clear on the program’s logic Be clear on the program’s logic Develop measures for each phase of program Develop measures for each phase of program Select measures for program evaluation or reporting based on intended audience and use Select measures for program evaluation or reporting based on intended audience and use

30 Resolution & Zoom

31 Recommended Reading Performance-Based Management Handbook Performance-Based Management Handbook DOE – Oak Ridge Institute for Science and Education Performance Measurement Challenges and Strategies Performance Measurement Challenges and Strategies U.S. Office of Management and Budget

32 Additional Air Info

33 EPA Report on the Environment Describes national environmental conditions and trends using existing data and indicators Describes national environmental conditions and trends using existing data and indicators IDs data gaps and research needs IDs data gaps and research needs Discusses challenges in filling those gaps Discusses challenges in filling those gapswww.epa.gov/roe

34