Planning and Oversight of NSFs Large Facility Projects Mark Coles Deputy Director for Large Facility Projects Office of Budget, Finance and Award Management

2 Introduction National Academies study Setting Priorities for Large Research Facility Projects (Brinkman report) makes specific recommendations for prioritizing, planning, and overseeing the construction and operation of large facilities supported by NSF. The report cites the need to make greater resources available for pre- award planning of a projects workscope, and Implement rigorous post-award oversight through periodic external review National Science Board and National Science Foundation joint response embraces this recommendation. Purpose of this presentation is to discuss how this should be implemented and solicit input

3 Summary of the NAS Study Recommendations Development of a year facility roadmap Three levels of criteria for ranking: –Scientific and technical merit –Agency Strategic Criteria –National Criteria New starts ranked in annual budget request using clear rationale based on roadmap Enhanced project pre-approval planning and budgeting Greater independent oversight and review needed Review effectiveness of Deputy Director for Large Facilities in two years OSTP should coordinates roadmaps across agencies NSF Leadership and NSB should pay careful attention to implementation of proposed reforms

4 NAS Study Findings There is a lack of funding for disciplines to conduct idea-generating and project-ranking activities and, once ideas have some level of approval, a lack of funding for conceptual development, planning, engineering, and designinformation needed when judging whether a project is ready for funding in light of its ranking and for preparing a project for funding if it is selected.

5 NSB defines a new process for approving readiness of a project: Project plans are judged to be construction ready by the Large Facility Deputy, The budget for construction and for operations costs has been justified to the satisfaction of the Chief Financial Officer From: Attachment 5 to NSB-04-92, NSB What are the criteria for ascertaining readiness of plans and budget?

6 Readiness certification process needs to be defined Goal: NSB approves projects that are well defined and that have thorough cost estimates Present situation: –Project baseline, including well scrutinized bottoms-up contingency estimate, is established after construction. –Many projects are de-scoped during construction (examples: LIGO, Gemini, IceCube, likely ALMA) Development of a project baseline (planning, costing, enabling R&D, PMCS, risk assessment, staffing) requires ~5-10% of total project cost.

7 Recommendation: Develop project baseline prior to approval of construction award Project baseline: –Detailed technical workscope flowdown from science requirements –Bottoms-up budget estimate –Resource loaded schedule with realistic budget profile and critical path –Thorough risk assessment and item-by-item contingency estimate –Robust project execution plan for project management, and staffing commitments for key work package managers

8 Benefits of defining project baseline prior to construction award Development of a robust project baseline reduces uncertainty in project workscope. Need to make an earlier commitment of significant development funding to a project, with expectation that construction approval is likely if pre-defined development criteria are satisfied. Internal Management Plan (aka Project Development Plan) could lay out predefined criteria that must be satisfied. For example: –Completion and successful review of preparatory research –Satisfactory demonstration of enabling technology –Develop of detailed bottoms-up cost and contingency estimates, resource loaded schedule, recruitment of key personnel, Project Management Plan, and other components of Project Baseline, followed by successful external baseline review.

9 More planning resources will: Reduce uncertainties in construction and operations budgets Reduce uncertainty in construction schedule Reduce likelihood of de-scoping Thorough pre-construction risk assessment will create a more robust Project Execution Plan Create an appropriate framework for cooperation for activities involving international and inter-agency partnerships as an integral part of in- depth planning. Make project oversight during construction more straightforward, add more definition to the proposed workscope and decision points. Allow more in-depth consideration of transition to operations and greater certainty in predicted O&M costs. Avoid a funding hiatus. Overlap some development activities beyond NSB approval and inclusion in annual NSF budget submission.

10 Other factors to consider Greater involvement by NSF OD, NSB, OMB, Congress, in the early stage of a project - prior to entry into Readiness Stage. Need to provide visible, but not risk-free, level of commitment to a project during the readiness phase so that a project can recruit engineering and managerial staff and undertake enabling R&D, detailed plans, cost estimates. Need to set an expectation that a project will go forward into construction from the readiness phase if it meets the criteria established in its Internal Management Plan (and that if can fall out, or terminate, if unsuccessful)

11 Suggested Implementation Prior to entering Readiness phase a project must have: –Scientific merit satisfying the first two NAS criteria – scientific merit and NSF priority –Preliminary Project Plan: Top down estimates of technical scope, budget, contingency, Detailed plan for enabling R&D, planning, technical demonstrations – Project Development Plan Detailed schedule with identified decision point and plan for determining construction readiness. Prioritization of readiness activities with window for construction approval, to provide latitude for timing of construction approval –Approval by Facilities Panel of Project Development Plan –Approval by MREFC Panel of overall activity, timing, decision points, etc.

12 Implementation: Requirements to enter Readiness Phase Review and approval of: –Preliminary technical scope, –Top-down budget and contingency estimates, –Project development plan –Project development schedule, including proposed decision point window Presentation and approval to NSF OD and NSB of project for entry into Readiness phase Determination by NSF OD and NSB that project meets third level NAS criteria – national importance Inform OMB and Congress of: –Projects potential construction scope –Development schedule –Window for construction approval (and allow for tuning to meet future budget envelope expectations)

13 While in the Readiness Phase Regular status reports to NSF and NSB Periodic updates to OMB and Congress of project status External reviews of development milestones as appropriate Baseline review to establish construction readiness and budget readiness Request to NSF OD and NSB to exit Readiness Phase upon successful completion of baseline review Inform OMB and Congress of successful completion of baseline review and request inclusion of projects construction budget within window Projects that fail to meet development milestones may be delayed or terminated, as appropriate.

14 Exiting Readiness Phase Construction activity begins Construction oversight by NSF begins: –Monthly status reports –Periodic external reviews of construction activity Development activities conclude Periodic updates to inform NSF OD, NSB, OMB, Congress of project status

15 Summary Impacts on NSF Greater commitment of R&RA funds for extensive planning Involvement of NSF Director, OMB, Congress from beginning of the intensive planning stage, rather than at the end Implementation of formally approved project development plans Greater responsibility for evaluation of project technical plans and budget by CFO and Deputy Director for Large Facilities

16 Oversight during construction Practices for post-award oversight vary greatly across foundation External review by slowly varying panels with a great breadth of relevant experience are used mostly in Physics. Other disciplines rely on self-assessment, or narrowly constituted advisory panels from the target research field How can we promote implementation of periodic external review by a panel of scientific, engineering, management, and technical experts? (As recommended by NAS report)

17 Need for further detailed advice NSFs Facilities Management and Oversight Guide will be revised as part of NSF/NSB response to NAS study Other findings from NAS study also present opportunities to improve NSFs capabilities: –Post award oversight –Full life-cycle cost accounting Need for guidance on how to develop and implement these policies, particularly in areas where there has not been a tradition of: –Construction of large facilities –Use of power tools for technical and financial reporting –Periodic post-award external review

18 Facilities Subcommittee With approval of Business and Operations Committee, propose to establish Facilities Subcommittee as a subsidiary of the Business and Operations Committee to provide specific guidance on these policy issues Facilities Subcommittee will report its findings to the full Business and Operations Committee Subcommittee will meet twice per year Plan to have five members of subcommittee, all with significant experience with management of large scientific facilities

19 Facilities Subcommittee Charter The Facilities Subcommittee will examine the business practices the NSF applies to planning and executing major research facility projects, including: –NSF policies and procedures for pre-construction planning and evaluation of proposed large facility projects, –Non-technical aspects of the oversight practices applied by NSF during construction of large facilities, and –Practices used to monitor business operations aspects of the facilities. The Subcommittee will make recommendations to the Business and Operations Advisory Committee as to the adequacy of current practices, and will make recommendations for ways that the NSF can improve current practices to enhance their effectiveness.

20 Proposed Facilities Subcommittee Membership The following individuals have agreed to serve on the Subcommittee if approved by the Business and Operations Committee: –Tom Kirk (Chair of Subcommittee and member of B&O Committee) Associate Laboratory Director of High Energy and Nuclear Physics, Brookhaven National Laboratory –Jim Yeck Project Director, IceCube Project, Univ. of Wisconsin, Madison –Janos Kirz Acting Director, Advanced Light Source, Lawrence Berkeley National Laboratory –Paul Gilbert Chairman Emeritus, Parsons Brinckerhoff Quade & Douglas, Inc. –Larry Ramsey Professor of Astronomy and Astrophysics at Penn State, and Project Scientist on Hobby-Eberly Large Telescope

21 How the B&O Committee can help Recommend business processes for ascertaining readiness of budget and construction plans Recommend how best to promote the implementation of periodic external review during construction Approve charter and membership of Facilities Subcommittee

Backup materials

23 NSF/NSB Response: The National Academies Report properly calls attention to the necessity for considerable pre-approval funding for planning and development when it questions whether there is sufficient NSF support for this "bottom up" process. NSF endorses the Reports recommendations to provide researchers access to funding sufficient to develop compelling research agendas, to refine and prioritize their facility requirements, and to complete research and development on facility designs and needed technologies. The level and form of funding for planning and development will be reviewed, and an evaluation will be made of how project funds are best invested to attain robust plans and schedules with better cost projections, so that only well-defined and thoroughly-costed projects are brought forward for consideration by the Board.

24 More NSF/NSB Response : The Director and the Board recognize the need to strengthen oversight of the implementation of large facility projects, which will require increased investments of NSF staff time and travel funds. The Report emphasizes the importance of initial planning and definition of technical scope, budget, and schedule, followed by periodic post-award status reviews held on-site by external experts, with implementation of a transparent process for management of changes to a projects implementation plan.

25 And more NSF/NSB response … On the basis of merit-reviewed proposals, NSF will fund these planning and project development efforts. In accepting the Facility Plan the NSB will concur that each of the Readiness List projects has attained that status by an appropriate process