1. OFFICE OF SCIENCE Dark Energy Survey (DES) Project at the Fermi National Accelerator Laboratory June 22-23, 2010 Department of Energy/ National Science Foundation Review Committee for the Daniel R. Lehman, Chair DOE/SC Review Committee Office of Science, U.S. Department of Energy http://www.science.doe.gov/opa/

2. OFFICE OF SCIENCE 2

3. Review Committee Participants OFFICE OF SCIENCE Daniel R. Lehman, DOE/SC, Review Chairperson

4. 4 Charge Memorandum Charge Questions 1.Technical: Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately? 2.Cost and Schedule: Are the current cost, schedule and contingency projections consistent with the approved baseline for all the projects? 3.Management: Is the management structure adequate and appropriate to successfully execute the projects within specifications, budget, and schedule? Are major risks being addressed effectively? 4.ES&H: Are ES&H aspects being properly addressed and are Integrated Safety Management Principles being followed? 5.Operations: Are the preliminary plans for the operations phase appropriate for this stage of the project? 6.Previous Review: Has the project responded satisfactorily to recommendations from the previous review? OFFICE OF SCIENCE

5. 5 Report Outline/ Writing Assignments OFFICE OF SCIENCE

6. 6 2.1.1 DECam Optics and Opti-Mech, CFIP Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE 1.Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately? Yes, through the assembly and hand-off of the DECam to CTIO. the major components of the Opto-mechanical system are either finished and ready for assembly or are progressing satisfactorily. 2. Has the project responded satisfactorily to recommendations from the previous reviews? Yes. The recommendation regarding oversight of the optics polishing, installation and delivery continues to be valid until commissioning. Recommendations regarding management issues were reviewed by SC4.

7. Findings The major opto-mechanical subsystems have been procured or are in the late stages of procurement. These includes DECam itself, the Telescope Simulator, the F/8 Changer, and other handling fixtures. The effort is now shifting to the assembly and installation of DECam in the Telecope Simulator and testing of the assembly under computer control with dummy optics. Testing of the F/8 Changer at FNAL is nearing completion and the assembly will be ready to ship to CTIO in the near future. Shipping of other elements not required for TS testing will follow shortly. The Telescope Simulator allows both the DECam project and the CTIO personnel who will be working on the telescope installation to gain experience with the assembly of the final DECam system. This will greatly speed up acceptance testing. 7 2.1.1 DECam Optics and Opti-Mech, CFIP Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE

8. Findings (continued) The corrector lenses have nearly finished polishing at SESO and are due to arrive at UCL in a staggered fashion over the next three months. The first two elements have been received. CTIO appears to be nearly ready for the arrival of DECam. The major work items remaining are the completion of the Telescope Control System upgrade (scheduled for completion in the fall of 2010) and the installation of the cooling system cabling. 8 2.1.1 DECam Optics and Opti-Mech, CFIP Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE

9. Comments Although several risks have been retired, significant risks to the project will unavoidably remain in this section of the project. In particular, there are no spare corrector lenses or filters, and the damage to any one element means schedule delays and potential cost overruns on for the project. This is particularly important for optical element 5, which also serves as the Dewar window. Because this element is destined to be handled much more during integration and testing, it is the most at risk for damage. No full-scale filters have yet been made available to DES from Asahi, although their delivery date is scheduled for about three months. More troubling is the potential effect of breaking a filter, given its importance in DES observations and the lack of any spares. On a positive note, the vendor has several spare blanks, which would reduce the time of recovery for a broken filter. 9 2.1.1 DECam Optics and Opti-Mech, CFIP Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE

10. Comments (continued) UCL has developed the concept for an end-to-end on-axis optical test of the corrector assembly using a test sphere and computer generated hologram (CGH). The are both benefits and risks associated with implementing such a test. Funds (~$5K) are available. SESO has notified UCL that the coating lab may shut down for renovation before the full complement of corrector lenses are coated. SESO is considering out-sourcing the coatings. This represents potential delays and added technical risk. 10 2.1.1 DECam Optics and Opti-Mech, CFIP Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE

11. Recommendations 2.1.1.1 Investigate the purchase of an additional z’ filter if contingency allows, to mitigate the risk to the Dark Energy Survey due to loss of a filter. 2.1.1.2 Consult with UCL regarding the advisability as a risk reduction measure of a full-aperture on-axis null test for the corrector assembly in the next ~2 months. 2.1.1.3 Discuss with UCL the risks associated with coating DECam lenses by third-party vendor and develop a recommendation on whether such a plan is acceptable to the project immediately. 11 2.1.1 DECam Optics and Opti-Mech, CFIP Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE

12. 12 2.1.2 DECam CCDs, SISPI and Electronics Roger Smith, Caltech / Paul Padley, Rice University OFFICE OF SCIENCE 1.Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately? Yes 6.Has the project responded satisfactorily to recommendations from the previous reviews? Yes, … clearly and explicitly.

13. 13 2.1.2 DECam CCDs, SISPI and Electronics Roger Smith, Caltech / Paul Padley, Rice University OFFICE OF SCIENCE Findings: Production of DECam Electronics is well advanced Boards produced in Barcelona are behind schedule, but this should not impact overall schedule SISPI software is also well advanced and is actively being used providing useful user feedback to the developers. 91 science grade CCDs have been produced, significantly more than the 62+10(spares) needed for DECam. This is consistent with yield projections. There have been two separate “full well degradation” events, occurring only in the MCCDTV, affecting different groups of engineering grade CCDs. The reason for this degradation is not understood. There have been modifications to the electronics to mitigate undesirable transients to address potential risks, though stress testing does not support a causal relationship. None of the 91 science grade CCDs have been exposed to risk. Funding for the early commissioning phase (in FY 2011) is uncertain.

14. 14 2.1.2 DECam CCDs, SISPI and Electronics Roger Smith, Caltech / Paul Padley, Rice University OFFICE OF SCIENCE Comments: The reviewers are impressed by the excellent team of scientists and engineers and the outstanding progress, and consequent low technical risk. The project should have no difficulty reaching its CD4 milestone The degradation of CCD full well is a source of serious concern: The reviewers are confident that the DECam CCD team has pursued every conceivable avenue to understand this problem but to date neither the physics of the CCD behavior nor the cause are understood. Performance before and after the “events” has been well documented and shown to be stable, suggesting that risk exposure is intermittent. While the uncertain funding of the FY11 commissioning activities is not a threat to the CD4 milestone, it is a serious risk to the timely transition to scientific operations. Lack of funding for travel for the university groups is a related issue.

15. 15 2.1.2 DECam CCDs, SISPI and Electronics Roger Smith, Caltech / Paul Padley, Rice University OFFICE OF SCIENCE Recommendations: Continue to monitor engineering grade CCDs for degradation events, to investigate the cause, and to identify the full well degradation mechanism within the CCDs. Identify the key DECam personnel that should be available for travel to CTIO, or at CTIO for the commissioning phase. Work with the participating institutions to secure funding for the FY11 commissioning activities. We urge NOAO to provide commensurate travel support for technical staff at CTIO to participate in system testing on the telescope simulator at SiDet. Evaluate simple illumination options to support *daytime* testing of DECam during commissioning, on the telescope.

16. 16 2.1.3 DECam Integration & Test Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE 1.Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately? Yes, for the part of the work related to integration and testing on the telescope simulator at Fermilab. For the part relating to the integration in Chile, the plans are not yet complete. 2. Has the project responded satisfactorily to recommendations from the previous reviews? Yes, although the CTIO installation plan that was created in response to our recommendations is now obsolete and should be redone very soon.

17.  Findings The integration of optical elements at Fermilab is proceeding smoothly. The telescope simulator has been working, and testing of components beginning with the F8 assembly is underway. The lens integration at UCL is slightly behind schedule due to fabrication issues with the mirror cells. At the moment, this is not critical to the schedule. There is currently no detailed plan describing the installation procedures for DECam at CTIO, only drafts. At the moment, this is still acceptable because the experience with the telescope simulator will modify these plans. The current schedule has a detailed plan in place by early 2011. 17 2.1.3 DECam Integration & Test Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University

18.  Comments The planning and funding commitments for DECam installation and commissioning at CTIO need to be in place very soon, as installation is currently planned in ~15 months. The Fermilab DECam team and CTIO need to jointly develop a plan that encompasses the scope of work, management structure, schedule and funding requirements for submission to their respective administrations. Key decision points should be identified. The timing of the installation faces critical scheduling decisions that have to be taken at about the same time as the planned arrival of DECam at CTIO in June 2011. Removal of the current Blanco top-end marks a “point of no return” for the telescope, when no other science programs can be scheduled at least until the end of the first phase of commissioning (when the F8 secondary system is commissioned.) At the moment, this is scheduled to begin as soon as DECam has successfully been powered on in the Coude’ room at CTIO. This phase of the project must be managed carefully. 18 2.1.3 DECam Integration & Test Matt Johns, Carnegie Institution Ian Dell’antonio, Brown University

19.  Recommendations 2.1.3.1 Complete the plan for installation of DECam on the telescope jointly with CTIO and prepare a funding proposal to support work by DECam staff to assist with installation and commissioning as per the management section recommendation. 19 2.1.3 DECam Integration & Test Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University

20. 20 2.2 DESDM Julian Borrill, LBNL / Rob Cameron, SLAC OFFICE OF SCIENCE 1.Are the accomplishments to date and planned future activities consistent with the baseline objectives and is the project progressing adequately? Yes, to meet the minimal objectives; meeting the full objectives will be challenging in the remaining time. 6.Has the project responded satisfactorily to recommendations from the previous reviews? For the most part, yes, although restructuring the project leadership is still in progress.

21. 21 2.2 DESDM Julian Borrill, LBNL / Rob Cameron, SLAC OFFICE OF SCIENCE Findings Project leadership has been restructured, with a new PI, project manager, and data challenge coordinator. DC5 has been partially completed, demonstrating reasonable data quality and effective use of node reservation. DC5a/b also exposed human errors, software bugs, algorithmic limitations, task management challenges, computation, IO and database bottlenecks, and hardware reliability issues; these are being addressed. Analysis of MOSAIC data illustrated real data issues not present in simulations. DC6a (Community Pipeline testing), DC6b (DESDM System acceptance testing) and DC7 (stress testing) are being planned for the next 13 months. Data processing has moved to NCSA’s Abe system, augmented by smaller clusters in Louisiana and Texas; NCSA’s Blue Waters system, scheduled to come online at the end of the DESDM development phase, is central to operating phase plans. CP requirements have been drafted but are yet to be agreed. Scientific analysis computing requirements are being gathered from SWGs and institutions. The current budget only extends through FY11, ending before commissioning.

22. 22 2.2 DESDM Julian Borrill, LBNL / Rob Cameron, SLAC OFFICE OF SCIENCE Comments The combination of the Project Scientist’s relocation to Europe, the new PI only having an 8% time-commitment, and the delay in appointing a Project Manager, has been detrimental but the situation should now improve. The number and complexity of the remaining data challenges has increased, resulting in a very tight, success-oriented, schedule. Although a formal deliverable, no commissioning plan exists for the CP; in the current schedule, CP will be the first element exercised in spring 2012. The need to reprocess the data coupled with the frequency of Abe’s node failures made DC5 unexpectedly time- and energy-consuming. Some DC5a error flags were either unset or unnoticed, and slow write-intensive jobs have required throttling, introducing a serial bottleneck. Analysis of real data alongside simulations is very valuable; preCAM is an obvious candidate. Stress-testing is not currently included in the DESDM acceptance tests, but only occurs immediately before DECam deployment. Feedback from the SWGs, as DESDM customers, would be extremely useful in future reviews.

23. 23 2.2 DESDM Julian Borrill, LBNL / Rob Cameron, SLAC OFFICE OF SCIENCE Recommendations Revise the risk management plan in the light of DC5 issues, particularly including extra-project hardware problems and transition costs. Include stress-testing in the DESDM acceptance plan. Develop a timely commissioning plan for the Community Pipeline upon completion of its specification. Consider including requests for additional resources where those are found to be necessary to meet project objectives on time when submitting the supplementary proposal to NSF to fund DESDM activities through at least the DECam/CP commissioning phase. Schedule a post-DC7, pre-commissioning, external review of the readiness of both the DESDM System and the Community Pipeline.

24. 24 3. Operations Planning, Calibration Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University OFFICE OF SCIENCE 1.Are the preliminary plans for the operations phase appropriate for this stage of the project? The planning for commissioning and operations is adequate for now, but key decisions need to be made in the next 6-9 months. 2. Has the project responded satisfactorily to recommendations from the previous reviews? Yes, a commissioning plan has been produced.

25.  Findings There is now a detailed plan from CTIO describing the activities to be carried out during commissioning. The current schedule has commissioning of DECam at CTIO completing in February, 2012. This is right at the end of the observing season for the DES. Any schedule slip would mean that observations of the DES would begin in September 2012. It is important that even if the schedule slips to commissioning beyond March 2012 the DES team obtain some data during the science verification stage to test the DESDM pipeline. The survey operations plan is adequate at the moment, although no definite funding to support operations has yet been approved. The DES calibration plan is at a relatively advanced stage, with the first phase of the calibration, the PreCam survey, beginning observations in September 2010. 25 3. Operations Planning, Calibration Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University

26.  Comments There are still concerns about control of the camera during the commissioning phase. While the MOU states that ownership of the commissioning process rests with the CTIO director, there will likely be circumstances that require work by DECam staff to resolve problems. Funding the work related to commissioning the camera in FY 2011 and FY 2012 is a crucial need for the DES project. PreCam, the precursor calibration survey to be carried out at the Schmidt telescope at CTIO, has gone from conception to a critical part of the calibration plan for the DES in less than a year. This is significant progress. The spectrophotometric calibration system has been successfully demonstrated at Las Campanas. The calibration hardware is on track for arrival at CTIO well prior to DECam. 26 3. Operations Planning, Calibration Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University

27. Recommendations: 3.1 Develop a detailed plan including budget requirements for DES operations to submit to the relevant funding agencies as soon as possible. 27 3. Operations Planning, Calibration Matt Johns, Carnegie Institute/ Ian Dell’antonio, Brown University

28. 28 4. Cost Kurt Fisher, DOE/SC Mark Reichanadter, SLAC OFFICE OF SCIENCE 2.Are the current cost, schedule and contingency projections consistent with the approved baseline for all the projects? DECAM available cost contingency ($1.7M) is low. CFIP is probably OK. Significant uncertainty in DES DM schedule projection. 6.Has the project responded satisfactorily to recommendations from the previous reviews? Yes, it appears that the project has been responsive to the cost recommendations.

29. 29 4. Cost Kurt Fisher, DOE/SC Mark Reichanadter, SLAC OFFICE OF SCIENCE Findings  The DECam Total Project Cost (TPC) is $35.15M  As of May 31, 2010, project is 78% complete  ~$26.0M progress is complete. $7.5M of work remains  Existing DECam contingency/remaining work has dropped from ~30% to ~22% since last review. Dollar value reserve is $1.7M.  Installation/commissioning are not included in project.  A bottoms-up cost to complete has been completed since the last review.  The project is not carrying any scope contingency  Currently the project has 11 medium to high level risks identified  The risks identified in the risk register are not correlated to dollars

30. 30 4. Cost Kurt Fisher, DOE/SC Mark Reichanadter, SLAC OFFICE OF SCIENCE Comments DECam has made substantial progress since the last review. Most procurements are in hand or under contract. Remaining work is primarily testing and assembly activities (mostly labor). After evaluating the remaining risks to DECam, cost contingency appears low. DECam should work to identify scope items that could be deferred should contingency demand remain high.

31. 31 4. Cost Kurt Fisher, DOE/SC Mark Reichanadter, SLAC OFFICE OF SCIENCE Recommendations 1.Prepare (for DECam) a potential list of scope reductions to adequately address lean contingency reserves. Present to the JOG by August 1, 2010. 2.Monitor contingency on a monthly basis and report usage in the DECam Monthly Report.

32. 32 5. Schedule and Funding Kurt Fisher, DOE/SC Mark Reichanadter, SLAC OFFICE OF SCIENCE 2.Are the current cost, schedule and contingency projections consistent with the approved baseline for all the projects? DECAM available cost contingency ($1.7M) is low. CFIP is probably OK. Significant uncertainty in DES DM schedule projection. 6.Has the project responded satisfactorily to recommendations from the previous reviews? Mostly, the integrated schedule between the three subprojects has not fully developed. With the hire of the DESDM Project Manager on July 1, 2010, the project will have responded to a key committee recommendation.

33. 33 5. Schedule and Funding Kurt Fisher, DOE/SC Mark Reichanadter, SLAC OFFICE OF SCIENCE Findings Critical-Decision 4 (CD-4) for DECAM is scheduled for September 2012. The current schedule has 12 months schedule contingency. A chronological list of major activities between the 3 subprojects has been developed. The property transfer MOU between DOE and NSF has not been ratified.

34. 34 5. Schedule and Funding Kurt Fisher, DOE/SC Mark Reichanadter, SLAC OFFICE OF SCIENCE Comments Today, the committee finds that it is likely that the DECam will be ready in about a year, CTIO will be prepared to accept the DECam, but DES DM will be late. The lack of an MOU between DOE and NSF could impact the first shipment of materials to CTIO scheduled for July, 2010. Obtain input from all subproject and develop and maintain an integrated project schedule between the three DES subprojects. Recommendations Execute the property transfer MOU immediately so as not to impact the first shipment of materials to CTIO. Obtain input from all subproject and develop and maintain an integrated project schedule between the three DES subprojects.

35. 35 6. Management Mark Reichanadter, SLAC Henry Heetderks, LBNL OFFICE OF SCIENCE 3.Is the management structure adequate and appropriate to successfully execute the projects within specifications, budget, and schedule? Are major risks being addressed effectively? Yes, for DECAM. Probably for the CFIP. Uncertain for the DESDM. 6.Has the project responded satisfactorily to recommendations from the previous reviews? All recommendations have been adequately addressed with the exception of a DES DM Project Manager, who is planned to start on July 1, 2010.

36. 36 6. Management Mark Reichanadter, SLAC Henry Heetderks, LBNL OFFICE OF SCIENCE Findings The DES PEP includes a detailed list of deliverables required to achieve CD-4 for the DECam. The DECam relies on ~$6M of in-kind contributions and is tracking their progress. Lens polishing is the most critical outstanding issue. A set of tests have been developed and agreed upon by DECam and CTIO to formally accept the DECam hardware as complete and begin the final installation process. A new Project Director has assumed the leadership of the DES Project in March 2010. Local PI for DESDM has been identified and is working at 8% time. A full time DESDM PM has been hired and will start on 1 July, 2010

37. 37 6. Management Mark Reichanadter, SLAC Henry Heetderks, LBNL OFFICE OF SCIENCE Findings DECam PM indicated that projected FY11 funding is inadequate to support cost of DECam installation at CTIO. In addition, DOE-supported operations activities in FY12 are not adequately funded. MOU between DOE and NSF necessary to get favorable tax treatment in Chile for the DECam is not yet signed.

38. 38 6. Management Mark Reichanadter, SLAC Henry Heetderks, LBNL OFFICE OF SCIENCE Comments  DECam – Substantial progress, 78% complete. Team has added critical engineering resources. Configuration control / risk processes functioning adequately. Remaining cost contingency is a concern.  FY11 funding to support DECam installation and commissioning has not yet been identified. This should be addressed very soon to avoid a work interruption and delay to DES science.  DES DM – Experiencing significant technical difficulties and schedule delays, related to insufficient resources in DES DM leadership. DES DM PM starts on July 1. Will need substantial support by the JOG, DES Council, Fermilab and NCSA to avoid delays to DES ‘ready for commissioning’.  CFIP – Some schedule delays but adequate float. Expected to be ready for the DECam next year.

39. 39 6. Management Mark Reichanadter, SLAC Henry Heetderks, LBNL OFFICE OF SCIENCE Comments Summarizing, the committee found challenges in DES DM leadership, FY11/12 funding needs, and property transfer require immediate attention. A sufficiently detailed action plan for each of the three elements should be prepared and presented to the DES Council and JOG for approval (see recommendation). DOE and Fermilab would be well served by developing a DES “End Game Plan” to fully develop the strategy and plans for completing project activities and transition into operations and science. Fermilab should give due consideration to its role in DES operations and the science program. Include the JOG as appropriate.

40. 40 6. Management Mark Reichanadter, SLAC Henry Heetderks, LBNL OFFICE OF SCIENCE Recommendations 1.The JOG, in collaboration with NCSA, Fermilab and the DES team, should work to address the following issues: a)Resolve any remaining issues with the DOE / NSF property transfer by August 1. b)Develop a plan and supporting budget for FY11 (installation & commissioning) and FY12 (operations) for resolution by August 1, 2010. c)Develop a DES DM Action Plan that addresses the tight schedule and emerging technical challenges by August 1, 2010. This should include a credible set milestones and adequate resources.

41. 7. ES&H Dennie Parzyck, DOE/FSO  Are ESH aspects being properly addressed and are Integrated Safety Management (ISM) principles being followed? Yes –DECam has maintained a strong ISM ESH program. –Technical Design Report fully describes the ISM Program. –An effective DOE/Fermilab ESH oversight program has been maintained. –Hazard Assessment (13) and Readiness Clearance ( 4 ) programs have been implemented. –The DECam Safety Assessment Document is in draft stage. –Communication with Chris Smith demonstrates a transfer of ISM culture to CTIO.  Has the project responded satisfactorily to recommendations from the previous reviews? Yes –2009 No previous recommendations. –2008 Previous recommendation on safety documentation has been addressed.  Recommendations: –Completion of remaining safety documentation. –Continued coordination with CTIO safety program plans. 41 OFFICE OF SCIENCE