Session 853 Ex tending Organizational Capacity and Capability to Evaluate Federal Environmental Research Programs Board of Scientific Counselors Program Reviews of the U.S. Environmental Protection Agency’s Office of Research and Development Lorelei Kowalski Designated Federal Official for the BOSC Office of Research and Development U.S. Environmental Protection Agency October 29, Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Lorelei Kowalski Designated Federal Official for the BOSC Office of Research and Development U.S. Environmental Protection Agency October 29, Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

Federal Advisory Committee Act The 1972 Federal Advisory Committee Act (FACA), 5 U.S.C. App. 2, created an orderly procedure by which Federal agencies may seek advice and assistance from citizens and experts.The 1972 Federal Advisory Committee Act (FACA), 5 U.S.C. App. 2, created an orderly procedure by which Federal agencies may seek advice and assistance from citizens and experts. Enacted to promote meaningful public participation in government decisions and to ensure that no particular interest groups have unfair access to policy makers.Enacted to promote meaningful public participation in government decisions and to ensure that no particular interest groups have unfair access to policy makers. Maximizes public access to advisory committee deliberations and requires balanced committee membership.Maximizes public access to advisory committee deliberations and requires balanced committee membership. More than 950 advisory committees with more than 45,000 members advise the President and the Executive Branch on a variety of issues.More than 950 advisory committees with more than 45,000 members advise the President and the Executive Branch on a variety of issues.

Federal Advisory Committee Act Requirements A Designated Federal Officer (DFO) (government employee) must attend each meeting.A Designated Federal Officer (DFO) (government employee) must attend each meeting. Meetings must be announced in advance and open to the public, unless designated as closed for national security, or other appropriate reasons. The public is given an opportunity to provide comments during each meeting.Meetings must be announced in advance and open to the public, unless designated as closed for national security, or other appropriate reasons. The public is given an opportunity to provide comments during each meeting. Any committee documents, including reports, minutes, agenda, etc. must be made available to the public.Any committee documents, including reports, minutes, agenda, etc. must be made available to the public. The committees must be “fairly balanced” in the points of view represented and bring a diversity of expertise and perspectives.The committees must be “fairly balanced” in the points of view represented and bring a diversity of expertise and perspectives.

Federal Advisory Committee Act at EPA EPA manages approximately 24 committees and 37 subcommittees.EPA manages approximately 24 committees and 37 subcommittees. Federal advisory committees are an important tool within EPA for building consensus and providing ideas and recommendations from EPA’s diverse customers, partners, and stakeholders.Federal advisory committees are an important tool within EPA for building consensus and providing ideas and recommendations from EPA’s diverse customers, partners, and stakeholders. Although not required under FACA, EPA requires subcommittees to adhere to the same public involvement requirements as Executive Committees.Although not required under FACA, EPA requires subcommittees to adhere to the same public involvement requirements as Executive Committees.

The Board of Scientific Counselors (BOSC) The BOSC is a Federal Advisory Committee, established by EPA to provide advice, information, and recommendations about the ORD research program. Includes up to 15 membersIncludes up to 15 members Meets three to five times each yearMeets three to five times each year Has an approved charterHas an approved charter Announces its meetings in the Federal RegisterAnnounces its meetings in the Federal Register Opens its meetings to the publicOpens its meetings to the public Has an internet websiteHas an internet website

The Board of Scientific Counselors Mission Provide ORD advice and recommendations on: Science and engineering research, programs and plans, laboratories and centers, products, and management practicesScience and engineering research, programs and plans, laboratories and centers, products, and management practices Peer review policies, and utilization of peer reviewPeer review policies, and utilization of peer review Program development and progress, research planning process, and research program balanceProgram development and progress, research planning process, and research program balance Human resources planning, and the appropriate scope and design of training programs for environmental research professionalsHuman resources planning, and the appropriate scope and design of training programs for environmental research professionals

The Board of Scientific Counselors Membership Members constitute a distinguished body of scientists and engineers, drawn from academia, industry, and environmental communities, who are recognized experts in their respective fields. Members are Special Government Employees (SGEs), who are invited to serve on the Committee for their individual expertise. The BOSC can establish subcommittees, who must report their recommendations and advice to the BOSC Executive Committee for full deliberation and discussion. Subcommittees have no authority to make decisions on behalf of the BOSC, nor can they report directly to EPA. In FY05, the BOSC had eight active subcommittees.

Office of Research and Development Program Reviews The National Academy of Sciences has recommended independent expert review for evaluating federal research programsThe National Academy of Sciences has recommended independent expert review for evaluating federal research programs The Office of Management and Budget (OMB) highlights the value of recommendations from independent expert review in guidance to federal agencies.The Office of Management and Budget (OMB) highlights the value of recommendations from independent expert review in guidance to federal agencies. ORD chose to use the existing BOSC Committee to conduct the expert review of ORD research programs.ORD chose to use the existing BOSC Committee to conduct the expert review of ORD research programs.

Program Review Purpose The program reviews provide prospective and retrospective evaluation of ORD research at the program level. Strengthen the research program and accountabilityStrengthen the research program and accountability Communicate research progress and resultsCommunicate research progress and results Verify clients application of research to environmental decisionsVerify clients application of research to environmental decisions Help managers decide research investments/disinvestmentsHelp managers decide research investments/disinvestments Provide input to external evaluations and reports:Provide input to external evaluations and reports:  OMB’s Program Assessment Rating Tool  Performance and accountability reports to Congress under the Government Performance and Results Act (GPRA)

Evaluation Criteria OMB/OSTP* Investment Criteria for R&D: relevance, quality, and performance—at the program levelOMB/OSTP* Investment Criteria for R&D: relevance, quality, and performance—at the program level R&D investments must be planned to be relevant to national priorities, agency missions, and customer needsR&D investments must be planned to be relevant to national priorities, agency missions, and customer needs Programs must maximize the quality of the research they invest inPrograms must maximize the quality of the research they invest in R&D programs must demonstrate performance by setting annual and long-term goals and demonstrating progress toward outcomesR&D programs must demonstrate performance by setting annual and long-term goals and demonstrating progress toward outcomes OMB and OSTP encourage federal research managers to characterize the program’s scientific leadership as wellOMB and OSTP encourage federal research managers to characterize the program’s scientific leadership as well *OSTP = Office of Science and Technology Policy

Program Reviews Conducted In FY05, the BOSC Executive Committee conducted program reviews of six ORD research programs by establishing the following subcommittees: Endocrine disrupting chemicalsEndocrine disrupting chemicals Human healthHuman health Ecological researchEcological research Particulate matter/ozoneParticulate matter/ozone Drinking waterDrinking water Global changeGlobal change

Program Review Process Each program review was coordinated by a DFO and Subcommittee ChairEach program review was coordinated by a DFO and Subcommittee Chair Process steps included formation of the subcommittee membership; coordination of a face-to-face meeting; and conference calls before and after that meetingProcess steps included formation of the subcommittee membership; coordination of a face-to-face meeting; and conference calls before and after that meeting Review steps included development of charge questions; participation in conference calls and face-to-face meeting; development of a draft report; vetting of the draft report through the BOSC Executive Committee; finalization of the report; and transmission of the report to ORD.Review steps included development of charge questions; participation in conference calls and face-to-face meeting; development of a draft report; vetting of the draft report through the BOSC Executive Committee; finalization of the report; and transmission of the report to ORD. The BOSC completed 18 conference calls and 6 face-to-face meetings in FY05 for the six subcommittees.The BOSC completed 18 conference calls and 6 face-to-face meetings in FY05 for the six subcommittees.

Program Review Results Final program review reports transmitted to ORD in FY05:Final program review reports transmitted to ORD in FY05:  Endocrine disrupting chemicals: April 2005  Human health: August 2005  Particulate matter/ozone: August 2005  Ecological Research: August 2005 A total of 72 recommendations were made in these reportsA total of 72 recommendations were made in these reports ORD has already prepared responses to recommendations from two of these reports, and is working on responses for the other two.ORD has already prepared responses to recommendations from two of these reports, and is working on responses for the other two.

Program Reviews Next Steps Lessons learned in implementing program reviews from the past year were discussed at a BOSC Executive Committee meeting in September 2005, including perspectives from both the Committee and ORD.Lessons learned in implementing program reviews from the past year were discussed at a BOSC Executive Committee meeting in September 2005, including perspectives from both the Committee and ORD. Measures to improve and streamline the process were suggested and are being developed/implemented.Measures to improve and streamline the process were suggested and are being developed/implemented. All agreed the effort was a worthwhile activity, and the BOSC will continue to conduct program reviews for ORD.All agreed the effort was a worthwhile activity, and the BOSC will continue to conduct program reviews for ORD. In FY06, the BOSC is planning to complete program reviews for the Global Change and Drinking Water research programs, and initiate program reviews for the Land (Hazardous Waste), Water Quality, and Sustainability research programs.In FY06, the BOSC is planning to complete program reviews for the Global Change and Drinking Water research programs, and initiate program reviews for the Land (Hazardous Waste), Water Quality, and Sustainability research programs.