DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008 DC3 Goals and Objectives Jeff Kantor DM System Manager Tim Axelrod DM System Scientist

DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008 DC3 Overall Goals and Objectives Extend DC2 Application Framework and Middleware to support Data Release Pipelines Expand DC2 Nightly Pipelines functionality to address LSST instrument signature removal and dayMOPS, and to improve quality of data outputs Provide Image Processing Pipeline for control and commissioning visualization requirements Develop first release of Data Release Pipelines (coaddition, detection, photometric calibration, astrometric calibration) Develop first release of Science Data Quality Analysis System (SDQAS) Continue scaled tests of data transfer, data processing, database ingest (to 15% of final LSST requirements) Conduct first scaled tests of data query (with map reduce/bigtable and DBMS SQL) Integrate new team members Answer key questions for PDR as documented in DM R&D Plan

DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008 DC3 Top-level Project Plan

DC2 Post-Mortem/DC3 Scoping February 5 - 6, Produce computing/storage, long-haul network acquisition plans 1/1/08 – 9/30/08 LLNL (mountain/base), NCSA (archive center), SDSC (data access center), NOAO (long-haul networks) 2.Produce cyber-security plan 1/1/08 – 9/30/08 NCSA (centers), LLNL (mountain/base) 3.Monitor and extrapolate computing, storage, and network architectures and price/performance trends; produce bi-annual report summarizing findings 9/1/05 – 9/30/10 LLNL (mountain /base), NCSA (centers), SDSC, NOAO 4.Model and prototype LSST infrastructure architecture and design, including scalability and reliability features. 6/1/06 – 9/30/10 LLNL (mountain/base), NCSA (archive center), SDSC (data access center), NOAO (long-haul networks) 5.Validate scalability of infrastructure via scaled performance loading tests of pipeline processing, data ingest, and data transfer (Data Challenge 3, 15% of LSST final requirements) 1/1/08 – 9/30/08 NCSA, LLNL, SDSC, NOAO For PDR - Monitor and evaluate computing, storage, and network resource price/performance trends and architectures

DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008 For PDR - Peta-Scale Database Architecture and Analysis 1.Test the performance of open source versions of Map/Reduce/Bigtable (hadoop/hdfs) on ingest and external queries. Characterize performance for spatial, temporal, and ad hoc meta-data based queries 1/1/08 – 9/30/08 SLAC, JHU, Google 2.Implement provenance ingest and re-creation of data products from raw data and provenance 1/1/08 – 9/30/08 SLAC, LLNL 3.Expand the implementation of persistence to include deep detection objects 1/1/08 – 9/30/08 SLAC 4.Evaluate relational database and map-reduce technologies for storage and query of various types of LSST data, including images, catalogs, and meta-data. Evaluate open source versus commercial technologies. 1/1/08 – 9/30/08 SLAC, Google R&D Plan contains 15 items with PDR significance