US Particle Accelerator School Review of Scientific User Facilities Division Committee of Visitors Presented by W. A. Barletta, Chair

US Particle Accelerator School Top level charge  Assess processes used during FY2010, 2011, & to solicit, review, recommend, & document proposal actions 2. to monitor active projects and programs.  Program elements to assess 1. Light Sources including the Accelerator & Detector Research Program 2. Neutron Sources 3. Nanoscale Science Research Centers & Electron Beam Micro- Characterization Centers 4. Construction Projects & MIEs  Report to the Summer 2013 BESAC meeting

US Particle Accelerator School Charge specifics  (1) For scientific user facilities including the accelerator & detector program,  Assess the efficacy  Assess the quality of processes used to: (a) Solicit, review, recommend, & document proposal actions (b) Monitor active projects, programs & facilities  (2) Within boundaries set by DOE missions & funding,  Comment on how the award process has affected: (a) Breadth & depth of portfolio elements (b) National & international standing of portfolio elements

US Particle Accelerator School BES SUFD Committee of Visitors Dr. William Barletta (MIT/USPAS), Chair  Construction Projects  James Krupnick (LBNL, Ret.)  Angus Bampton (PNNL)  Jeff Hoy (Trident Service LLC)  Maria Dikeakos (DOE PPPL)  Nano-Science & E-Beams  Prof. Donald Tennant (Cornell)  Prof. Beatriz Roldan Cuenya (UCF)  Dr. Ernie Hall (GE)  Dr. James Liddle (NIST)  Light Sources, Accelerator & Detector Research  Dr. Simon Bare (UOP)  Prof. Nora Berrah (WMU)  Dr. Gene Ice (ORNL)  Dr. Joel Ullom (NIST)  Dr. David Robin (LBNL)  Neutron Facilities  Prof. Sunil Sinha (UCSD)  Dr. Robert Dimeo (NIST)  Prof. Thomas Russell (UMass)  Dr. John Tranquada (BNL) Chairman in Red BESAC Member in Italics Chairman in Red BESAC Member in Italics

US Particle Accelerator School Overarching conclusions  We commend SUFD for effective use of its available funding to construct & operate a system of facilities which deliver world-leading science  The efficacy of SUFD’s processes to review, recommend & document proposal actions are excellent  The efficacy of the processes to monitor & review active projects, programs & facilities are also excellent  SUFD staff members are to be commended for rigorous & effective program management Comment: BES growth over the decade is impressive

US Particle Accelerator School Overarching conclusions  Within the scope of DOE missions & available funding, SUFD’s award processes continue to enhance  the breadth & depth of portfolio elements as well as  their national and international standing  International competition in scientific user facilities is fierce; maintaining scientific leadership will require increased investments  for the user facilities  for user support

US Particle Accelerator School SUFD-wide findings & recommendations - 1  A. Implementation of previous COV recommendations  FINDING: SUFD has effectively addressed the majority of the recommendations made in the previous COV report  FINDING: SUFD travel budget continues to be incommensurate with most effective oversight of its program  RECOMMENDATION: Enhance the effectiveness of program oversight by increasing the flexibility of SUFD managers to interact with facility managers to communicate with the facilities staff via increased on-site presence

US Particle Accelerator School SUFD-wide findings & recommendations - 2  B. Assessment of COV process effectiveness  FINDING: Providing all data on computers to each reviewer is highly responsive to previous recommendations & greatly improves the COV process. The electronic documentation was thorough and well organized  RECOMMENDATION: The move toward a full, searchable database is commendable. It should be available to the next COV

US Particle Accelerator School SUFD-wide findings & recommendations - 3  C. Facility review process description & effectiveness  FINDING: The 3-year reviews of the facilities are well organized and well executed with appropriate review teams  FINDING: A uniform definition of high impact publications has been established for light sources & neutron sources.  RECOMMENDATION: Finalize the set of uniform definitions of high impact publications for the nanoscience centers

US Particle Accelerator School SUFD-wide findings & recommendations - 4  D. General Issues  FINDING: The quality of the scientists at the facilities is the critical asset that ensures excellence & success.  RECOMMENDATION: Place added emphasis on career development as well as on maintaining state-of-the-art experimental apparatus, sample environment & software at all facilities to maximize scientific productivity  FINDING: Different types of facilities serve different scientific communities. They are all needed & important.  RECOMMENDATION: New metrics that account for scientific impact should apply to all the types of scientific user facilities

US Particle Accelerator School SUFD-wide findings & recommendations - 5  Recommendations: If an increased fraction of industrial users is desired, this fact should be clearly communicated to the management of facilities  Proposal evaluation should include criteria that value factors such as economic & technological impact  Thorough review of user agreements should be undertaken with a view to removing barriers to industry users

US Particle Accelerator School Light sources and Accelerator & Detector Research

US Particle Accelerator School Light source findings - 1  The review process is comprehensive, balanced, fair, and transparent to all facilities  They are all reviewed using the same major criteria.  The number and breadth of the reviewers for each facility was appropriate  Reviews provided important comments & specific actionable recommendations where appropriate.  The reviews were uniformly forthright, detailed, & contained a summary section at the beginning of the review  The detail in review comments indicated a deep appreciation of important issues that went beyond the documentation solicited.  This indicates the importance of the on-site review.

US Particle Accelerator School Light source findings - 2  Review follow-up was not documented between reviews  RECOMMENDATION: A formal yearly follow up of facility recommendations should be documented each year  Ideally this could be a short response saying all issues had been previously addressed when appropriate  Recommendations from the prior reviews were not sent to the facility reviewers  RECOMMENDATION: Make previous review recommendations & facility response available to the facility reviewers at the outset  to allow the review committee to assess how the facility has responded to the prior recommendations.

US Particle Accelerator School Accelerator & Detector Research - Findings  Resources available are so limited ~$10M (~1% of total SUFD) that the breadth is necessarily small (focusing primarily on FELs)  This figure is too small to maintain scientific leadership  Projects are generally excellent & matched to SUFD needs  At present the portfolio does not include x-ray optics

US Particle Accelerator School Accelerator & Detector Research - Recommendations  Increase the ADR portfolio to $20M to $30M per year (2-3% of SUFD budget)  Consider the concept of a HUB to advance ADO technology in support of its scientific mission  Add X-ray optics to the ADR portfolio  As part of increasing the portfolio, formalize the proposal solicitations  Continue using workshop reports to guide research initiatives & to shape investment priorities  Develop topic-specific metrics to assess / characterize US capabilities in accelerators, detectors & optics

US Particle Accelerator School Neutron sources

US Particle Accelerator School Neutron sources  Finding : Few instruments in ORNL’s Neutron Sciences Directorate operate in the steward-partner model.  RECOMMENDATION: Strongly encourage neutron scattering facilities to explore forming partnerships on instruments with potential partners from other agencies in the cooperative stewardship model  Exploit neutron scattering capabilities to benefit the broadest possible scientific community  Finding: It is unreasonable to frontload facilities with understaffed instrumentation or beamlines  Recommendation: When an MIE is being considered, a the facility should have well-designed plan to ensure its robust, long-term operation for users

US Particle Accelerator School Neutron sources & neutron science  Finding: Trends in the science enabled by neutrons have evolved significantly since last studies (1993, 2002) on the future of neutron science in the US  Recommendation: We recommend that BES join with other agencies, such as DOC, NSF, & NIH, to assess current status & future directions for U.S. neutron science  Include factors, such as neutron measurement capacity & capabilities and present & future needs of the U.S. scientific community

US Particle Accelerator School Nanoscience Research Centers & E-beam Microcharacterization Centers

US Particle Accelerator School NSRCs & EBCs - General Observations  Finding: NSRCs are entering a post-ramp up phase in which they face issues of high demand for facilities & instruments & over subscribed staff  Recommendation: Provide guidance to NSRCs to plan for expansion of facilities or expanded operating hours  Alternatively adopt higher rejection rates of user proposals

US Particle Accelerator School  Finding: SUFD program managers have done a commendable job at establishing thorough & transparent processes for evaluating ongoing projects  The 3-year review cycle might be excessive for timely corrective actions  Recommendation: Increase face-to-face to time between DOE officials & management, scientific staff, & user community of the NSRCs &EBCs, including regular (yearly) on-site visits  Methods of assessing NSRCs have not evolved since the centers were first commissioned  Recommendation: NSRCs (& EBCs) are sufficiently different from light sources to warrant tailored metrics & assessment methods (user surveys) NSRCs & EBCs - General Observations

US Particle Accelerator School NSRCs & EBCs - General Observations  BES recommended merger of EBMCs & NSRCs in response to a triennial review finding that EBMCs were understaffed & underfunded to carry out their mission.  This recommendation focuses on improvements in synergy & operational efficiency.  Recommendation: SUFD should ensure that facility merger plans focus clearly on these improvements

US Particle Accelerator School High Utilization Instruments  Finding: The unique, world-leading instruments associated with the EBMCs are in high demand but are not utilized optimally  Staff funding is currently for 40 hour week, yet the labs are open more than 8 hours per day.  Recommendation: Merger plans should include expanded staffing (>8 hr/day) on select tools

US Particle Accelerator School Industry Participation  NSRCs have not attracted a significant number of industry users  Recommendations: If an increase in the fraction of industrial users is desired, this fact should be clearly communicated to the NSRCs  Proposal evaluation should include criteria that value potential technological or economic impact  Thorough review of user agreements should be undertaken with a view to removing legal barriers to industry users

US Particle Accelerator School CONSTRUCTION PROJECTS

US Particle Accelerator School Findings regarding project management  Comment: PM processes are robust & successful in delivering projects on schedule, within budget & meeting baseline technical performance parameters  SUFD met, and often exceeded, BES goal of remaining within + / - 10% for cost/schedule performance  Project management performance metrics have become even more visible indicators of DOE performance for external stakeholders  Recommendation: Ensure that CD4 requirements are reasonable, broadly understood by all stakeholders, & fully achievable within the project budget  Effort should be made to manage & align expectations for what constitutes successful initial scientific operations

US Particle Accelerator School Impact of travel reduction  Comment: SUFD management should determine the correct level of field presence for Program Managers to provide adequate Federal oversight, operational awareness, & fosters strong and open communication between field and HQ elements  On-site presence, graded to project risk, should be appropriately balanced with use of remote communication tools to live within constrained budgets  Recommendation: Mitigate negative impacts of reduced travel funds  Balance field presence with communication technology to maintain robust communication between program managers & the on-site members of IPTs

US Particle Accelerator School The COV thanks Jim Murphy and the entire SUFD staff for their candor & responsiveness Special thanks to Linda Cerrone for her help in making this review easier and more pleasant