John Peoples October 3, The Dark Energy Survey Structure, Management, and Oversight A presentation to the Directors of Fermilab, NCSA and NOAO

John Peoples October 3, Outline of the presentation Background for the expectations of DES and NOAO The structure of DES as a project The management of DES The oversight of DES The way the draft MOU among DES, Fermilab, NCSA, & NOAO enables the management and oversight Outstanding issues for management and oversight

John Peoples October 3, Key elements of the NOAO Announcement of Opportunity A partnership opportunity to develop a major new instrument for the Blanco telescope –A special opportunity to exploit the wide field capability of the prime focus Up to 30% of the Blanco telescope for 5 years commencing on 2007 or 2008 for the science project NOAO will contribute the operation of the telescope and an upgraded control system NOAO would expect to partner with the successful proposer in developing a data management system, which is compatible with the National Virtual Observatory (NVO)

John Peoples October 3, The Dark Energy Survey: The science project The goal of the DES Collaboration is to take the next big step in unraveling the mystery of dark energy. –We plan to measure w and w a through four independent methods: galaxy clusters, weak lensing tomography, galaxy angular clustering, and supernova luminosity distances. –We proposed a 5000 sq. deg. Optical & near IR survey to a depth of ~24 th magnitude in g,r,i, &z bands with the NOAO Blanco. –We are building DECam, a wide field CCD camera, and DESDM, a Data Management System to achieve this goal. DES has prepared a Science Requirements and Technical Specifications Document to define and control the performance baseline of the Survey

John Peoples October 3, NOAO BIRP Review NOAO External Review of the DES proposal by the BIRP: Blanco Instrumentation Review Panel –Was impressed by the thorough scientific and technical description of the project. –Thought the instrument would be an asset to the national observing system, and the survey archive would enable an enormous amount of science.

John Peoples October 3, BIRP recommendations BIRP recommended NOAO to proceed with the partnership, provided that: –The camera, the data reduction pipelines and the DES data archive be deliverables. –A formal set of written acceptance testing plans and performance metrics be developed. –There should be an external oversight committee. –DECam should not preclude use of the F/8 focus. –Filter changes be should be possible in normal operations and NOAO should supplement the DES griz filters. –NOAO should commit adequate resources to enable community science.

John Peoples October 3, NOAO expectations of the DES Project US astronomical community access to a major new instrument that can exploit the still powerful capabilities of the Blanco. –DECam will be a community instrument. A majority of the remaining time on the Blanco after accounting for the time allocated to DES could be awarded to peer reviewed proposals from the astronomical community. US astronomical community access to the public archive created from the data obtained with DECam. –NOAO plans to place the DECam data and the processed data in a public, NVO compatible archive, under its stewardship. NOAO has prepared a Community Needs Document to help define the scope of DECam and the DESDM from the community perspective.

John Peoples October 3, DES Project Structure: The DECam Project The DECam project consists of: –The instrument and the data acquisition system –Interfaces to the observer and the upgraded control system –On mountain data reduction for quality assurance –Connection to the NOAO data transport system –Simulated data for the DESDM data challenges The DECam project will deliver to CTIO –DECam with the required documentation –Specialized spares –The observing plan and an operations & maintenance plan –Personnel & support to help the integration and commissioning of DECam on the Blanco

John Peoples October 3, The Dark Energy Survey Camera: DECam 1.1 Management 1.2 Focal Plane Detectors 1.3 Front End Electronics 1.4 Optics 1.5 Opto-Mechanics 1.6 Survey Image Processing System (SISPI) 1.7 Survey Planning 1.8 CTIO Integration DECam will replace the prime focus cage DECam Project Structure

John Peoples October 3, Survey Image System Process Integration (SISPI) WBS 1.6 CTIO will upgrade the Telescope Control System (TCS) Data Management (DM): U. Illinois-Astro/NCSA U Illinois-HEP (J. Thaler) is leading the SISPI development - similar to HEP-DAQ systems

John Peoples October 3, DES Project Structure: DES Data Management Project The DES Data Management System consists of: –Processing pipelines with built in QA testing –A distributed archive to support automated data processing and calibration within a grid computing environment –A catalog archive database to support science analyses –Web portals for control, monitoring and scientific analyses –Hardware platforms required for operations DESDM will produce and distribute science ready DES archived data products to the Collaboration

John Peoples October 3, System Overview

John Peoples October 3, DES Project Structure: DES Data Management Project DESDM project will deliver to NOAO DPP –processing pipelines that remove instrument signature with built in QA testing suitable for integration into E2E –The documentation for the pipelines –The support to integrate the pipelines into E2E and commission their operation During the operations phase DES will deliver the raw data and the processed data with the instrument signature removed to NSA via the E2E Data Transport System

John Peoples October 3, DES Data Management Project requirements from NOAO The DECam pipeline and delivery of raw and pipeline- processed images into the NOAO Science Archive (NSA) are a DES deliverable, along with DECam. In this context, pipeline-processed images refers to images that have been corrected for instrumental signature and astrometrically and photometrically calibrated. The pipeline used for non-DES observations may be a parallel pipeline to the actual DES pipeline.

John Peoples October 3, Data Challenge Schedule We are here base level system data quality, stress test deploy and test outside NCSA final validation and stress test

John Peoples October 3, DES Project Structure NOAO/CTIO NOAO/CTIO will provide –The Blanco telescope and operations support –An upgraded telescope control system –Infrastructure improvements to support DECam operations and maintenance –Leadership of the integration and commissioning of DECam on the Blanco –Interface to the NOAO/DPP Data Transport System DECam will be a facility instrument and NOAO will operate it for the community as it operates other instruments

John Peoples October 3, DECam & CTIO

John Peoples October 3, DES Project Structure NOAO/DPP NOAO/DPP will provide its E2E Data Management System to users of DECam to receive, process and distribute DECam data. It consists of: –Data capture and transport over NOAO network facilities to La Serena –Raw data storage at La Serena and subsequent distribution to other nodes, including NCSA, over NOAO network facilities –Archive and access services through the NOAO NVO portal –Pipeline processing for the community - DES must provide a well documented and E2E integrated pipeline to remove DECam instrument signatures for this purpose –Data management infrastructure E2E will also serve the public archive for DES DECam data products.

John Peoples October 3, The DES Collaboration Fermilab University of Illinois at Urbana-Champaign University of Chicago Berkeley Lab University of Michigan NOAO Spanish Consortium: Institut d'Estudis Espacials de Catalunya (IEEC/ICE): Institut de Fisica d'Altes Energies (IFAE),CIEMAT, Madrid: (UAM) United Kingdom Consortium: University College London; University of Cambridge; University of Edinburgh; University of Portsmouth Institute of Cosmology and Gravitation; University of Sussex 14 institutions and 81 participants

John Peoples October 3, One survey: four projects The Collaboration proposes to organize the DES as four projects to carry out the survey: Fermilab leads the construction of DECam NCSA/UIUC leads the development of the DES Data Management System NOAO CTIO leads the integration of DECam on the Blanco and the telescope improvements NOAO DPP provides DECam data to the astronomical community through the E2E system, which it develops with specific contributions from DES Each lead institution has extensive experience in the project that it leads. All participants plan to engage in the science and contribute to the deliverables

John Peoples October 3, Dark Energy Survey Oversight: a collaboration perspective CTIO Director A. Walker Management Committee J. Peoples, Chair NOAO DPP C. Smith DES Data Management Project J. Mohr, PM

John Peoples October 3, PPD is host division for DECam and provides most of the technical resources and management support The Experimental Astro-physics Group in CD has the astrophysics experience and are also involved in SDSS and SNAP Science and Technical Requirements come from the DES collaboration through the MC and the Project Scientist

John Peoples October 3, DECam Work Breakdown Structure Level 2 Managers: bring L2 subsystem into operation on budget and sched. prepare monthly reports and schedule updates coordinate with other L2 Managers

John Peoples October 3, Data Management Project DM System Requirements –Reliably transfer ~300GB/night for 500 nights from CTIO to NCSA –Automatically process data with built-in quality assurance –Archive data products that meet science requirements and serve to collaboration and community Deliverables –DM System ( for HPC platforms and workstations ) Pipeline middleware Astronomy modules Catalog database Image Archive –Archived science ready DES data DM Team –U Illinois/NCSA, Fermilab and NOAO –Additional DES collaborators Pursuing spiral development strategy ‘04-’09 U Illinois/NCSA DES DM Team

John Peoples October 3, DES Oversight I would like the Directors of the three lead institutions to collectively oversee the DES in order to assure that –The DES meets the expectations of Fermilab, NCSA and NOAO for the DES project –The performance of the project deliverables meets the DES Science Requirements and Technical Specifications –The responsibilities for meeting the Community Needs are shared equitably and are consistent with agency direction –The Collaboration obtains sufficient funds to complete the project –The Collaboration meets the cost and schedule baseline established by the funding agencies – The terms of the MOU among the Collaboration, Fermilab, NCSA, and NOAO are met

John Peoples October 3, Dark Energy Survey Communications and Oversight DES Collaboration Management Committee J. Peoples, Project Director DECam Project B. Flaugher, PM T. Abbott, DPM W. Merritt, DPM CTIO Director A. Walker Blanco Scientist T. Abbott NOAO DPP C. Smith DES Data Management Project J. Mohr, PL C. Beldica, PM Directors FNAL, NCSA, NOAO

John Peoples October 3, The DES-FNAL-NCSA-NOAO MOU It defines the deliverables and a plan for periodic reviews of the progress that the Collaboration is making on the deliverables. It acknowledges the community needs and a process to accommodate the needs. It makes provisions for the the community use of DECam and DESDM provided pipelines and community access to all DECam data and reduced data. It defines the processes for accepting the deliverables. It defines the responsibilities of the Collaboration, which include fundraising and periodic reporting to the Directors. It defines the responsibilities of Fermilab, NCSA, and NOAO. It defines a framework for oversight of the project activities by the Directors.

John Peoples October 3, Outstanding Issues The success of the DECam project depends on DOE funding the major part of the project. DOE has not announced whether it will fund the project. However, its key advisory panels recommended that DOE fund DES. Fermilab has submitted a DECAm project plan to DOE that is consistent with its announced goals. –PPARC and MEC have made commitments to funding the UCL and Barcelona & Madrid contributions respectively, but they are contingent on full US approval. Design and development efforts are underway and funded. –The DECam team is making good progress on all areas of development, in particular CCD development and testing, with DOE support for R&D.

John Peoples October 3, Outstanding Issues The success of the DESDM project depends on NSF awarding funding to Joe Mohr at UIUC. A proposal submitted a year ago for DESDM was not supported. The NSF program manager indicated that NSF would not fund the proposal if DESDM were part of the required deliverables to NOAO in the MOU. – This has complicated completing the MOU –Joe Mohr will submit a revised proposal to NSF in November –In the meantime work on Data Challenge 2 is proceeding as planned with NCSA and UIUC support

John Peoples October 3, Outstanding Issues The ability of NOAO to support the DES project may be impacted by the NSF Senior Review of its facilities for astronomy, which includes NOAO. The Senior Review, which has been in progress since July 2005, will report to the NSF Mathematical and Physical Sciences Advisory Committee on November 1-3. The report will define the landscape in which NOAO and DES must live. –The uncertainty has made it difficult to reach agreement on the definition of the DES contributions to E2E and the DESDM portions of the MOU.

John Peoples October 3, Resolution of the Issues Agreement on the definition of the DES contributions to E2E and the DESDM portions of the MOU. –A working group with membership from the DESDM team at NCSA and the NOAO DPP team and chaired by J Peoples will be set up to define the DESDM deliverables to NOAO DPP. Its report will be submitted to the Directors for their consideration. Review of the entire DES project and operations –I would like the Directors to commission a review of the entire project after the working group has reported and the MOU is ready for signature –In the past DOE and NSF program managers have asked to see the entire project, including operations

John Peoples October 3, DESDM Oversight at NCSA/UIUC Oversight Group of 3 senior NCSA developers: –Randy Butler, Mike Freemon and Jay Alamed Senior/Administration Oversight: –NCSA Director's Office – Thom Dunning and Danny Powell –College of Liberal Arts and Sciences – Assoc Dean James Kirkpatrick