National Center for Supercomputing Applications Data Management for the Dark Energy Survey OSG Consortium All Hands Meeting SDSC, San Diego CA March 2007 Greg Daues National Center for Supercomputing Applications University of Illinois at Urbana-Champaign, IL, USA

National Center for Supercomputing Applications The Dark Energy Survey International Collaboration –Fermilab - Camera building, Survey Planning and Simulations –U Illinois - Data Management, Data Acquisition, SPT –U Chicago - SPT, Simulations, Optics –U Michigan- Optics –LBNL - Red Sensitive CCD Detectors –CTIO - Telescope & Camera Operations –Spanish Consortium- Front End Electronics Institut de Fisica d’Altes Energies, Barcelona Institut de Ciencias del Espacias, Barcelona CIEMAT, Madrid –UK Consortium- Optics UCL, Cambridge, Edinburgh, Portsmouth and Sussex –U Penn- science software, electronics –Brazilian Consortium- Simulations, CCDs, Data Management Observatorio Nacional Centro Brasileiro de Pesquisas Fisicas Universidade Federal do Rio de Janeiro Universidade Federal do Rio do Sul –Ohio State University- electronics –Argonne National Laboratory- electronics, CCDs

National Center for Supercomputing Applications DESDM Team Tanweer Alam 3 Emannuel Bertin 4 Joe Mohr 1,3 Jim Annis 2 Dora Cai 3 Choong Ngeow 1 Wayne Barkhouse 1 Greg Daues 3 Ray Plante 3 Cristina Beldica 3 Patrick Duda 3 Douglas Tucker 2 Huan Lin 2 1 U Illinois Astronomy 2 Fermilab 3 U Illinois NCSA 4 Terapix, IAP

National Center for Supercomputing Applications The Dark Energy Survey A study of the dark energy using four independent and complementary techniques –Galaxy cluster surveys –Weak lensing –Galaxy angular power spectrum –SN Ia distances Two linked, multiband optical surveys –5000 deg 2 gri and z (or Y+Z?) –Repeated observations of 40 deg 2 Construction and Science –Build new 3 deg 2 camera for Blanco 4m in Chile –Invest in upgrades for Blanco 4m –Build DM system to archive and process data at NCSA –Survey and Science: Image credit: Roger Smith/NOAO/AURA/NSF Blanco 4m on Cerro Tololo

National Center for Supercomputing Applications DES Instrument (DECam) 3556 mm 1575 mm Camera Filters Optical Lenses Scroll Shutter Elements: mosaic camera 5 element optical corrector 4 filters (g,r,i,z) Focal plane: 62 2K x 4K CCD modules (0.27''/pixel)

National Center for Supercomputing Applications DES Data Management Overview Data Management System must provide –Reliably transfer ~310GB/night produced on 525 nights over 5 years from CTIO to NCSA –Process data at NCSA, provide to DES team and community –Maintain DES archive over the long term ~2PB total, including 100TB database –Community access Raw and reduced data 1 yr after acquisition Co-add and catalogs at midpnt and end of survey Reduction pipelines released when camera deployed DESDM design guided by requirements for –Data –Processing –Storage and Distribution –Automation and Reliability –Hardware Single DECam exposure (500Mpix) consists of 62 individual CCD images

National Center for Supercomputing Applications DESDM Data Flow

National Center for Supercomputing Applications Pipeline Architecture Java Middleware layer with XML-described workflows Java Middleware layer with XML-described workflows NCSA workflow scripting packages : NCSA workflow scripting packages : OGRE [Apache Ant] OGRE [Apache Ant] OgreScript [ELF container] OgreScript [ELF container] Developed within LEAD, OGCE, and other projects Developed within LEAD, OGCE, and other projects Orchestration/Control and Management of Tasks Orchestration/Control and Management of Tasks Modular components with dependency rules Modular components with dependency rules Grid functionality: Grid functionality: Integrated with Trebuchet (jglobus) GridFTP library Integrated with Trebuchet (jglobus) GridFTP library Workflows publish/subscribe to Event Channel Workflows publish/subscribe to Event Channel (Interact with web services) (Interact with web services) Application Layer: Astronomy Modules Application Layer: Astronomy Modules Many authored by DES team in C Many authored by DES team in C Terapix software : SCAMP, SExtractor, SWarp Terapix software : SCAMP, SExtractor, SWarp

National Center for Supercomputing Applications Orchestration Pipeline Orchestration Pipeline Operates on a single DES server : Control and Management Operates on a single DES server : Control and Management Database interaction; Setup of workspaces, staging via Database interaction; Setup of workspaces, staging via GridFTP GridFTP Condor-G for data parallel job description/submission Condor-G for data parallel job description/submission to Globus gatekeepers to Globus gatekeepers Pipeline Server Orchestration Pipeline Catalog and Image Archive Database... TeraGrid Compute Pipelines Event Channel Pipeline Processing

National Center for Supercomputing Applications Image Processing Pipelines Image Processing Pipelines Executed on TeraGrid/HPC systems (NCSA Mercury) Executed on TeraGrid/HPC systems (NCSA Mercury) Data parallelism by CCD : 62 independent jobs Data parallelism by CCD : 62 independent jobs Data parallelism by exposure : 250 jobs per night Data parallelism by exposure : 250 jobs per night Monitoring and Quality Assurance through events system Monitoring and Quality Assurance through events system Pipeline Server Orchestration Pipeline Catalog and Image Archive Database... TeraGrid Compute Pipelines Event Channel Pipeline Processing

National Center for Supercomputing Applications DES Distributed Archive Database tracks image location and provenance Sites/Nodes within Replica Archive FNAL Enstore Mass Storage FNAL Enstore Mass Storage NCSA MSS NCSA MSS DES development platforms/workstations DES development platforms/workstations NCSA Mercury NCSA Mercury NCSA Cobalt & Tungsten, SDSC IA64, TACC Lonestar NCSA Cobalt & Tungsten, SDSC IA64, TACC Lonestar GPFS-WAN space GPFS-WAN space Replicate a fixed Filesystem Structure at each node Data from all Stages of Processing Data from all Stages of Processing Archive/ raw/* red/* coadd/*

National Center for Supercomputing Applications Data Access and Replication software Abstracts file transfer API to hide underlying middleware Abstracts file transfer API to hide underlying middleware Uniform interface to Uniform interface to - heterogeneous resources - heterogeneous resources - different transfer tools & strategies - different transfer tools & strategies Between TeraGrid sites, use NCSA Trebuchet library : Between TeraGrid sites, use NCSA Trebuchet library : Dataset optimizations: client_pool_size, tcp buffer Dataset optimizations: client_pool_size, tcp buffer ~ 300 MB/s for 3 clients ~ 300 MB/s for 3 clients NCSA MSS optimizations: stream mode, “target active” NCSA MSS optimizations: stream mode, “target active” ~ 60 MB/s ~ 60 MB/s For retrieval from Enstore/dCache, use For retrieval from Enstore/dCache, use Storage Resource Manager (SRM) developed at FNAL Storage Resource Manager (SRM) developed at FNAL Optimized Tape to Cache staging at FNAL Enstore Optimized Tape to Cache staging at FNAL Enstore DES Distributed Archive

National Center for Supercomputing Applications DES Data Challenges DESDM employs spiral development model with testing thru yearly Data Challenges Large scale DECam image simulation effort at Fermilab – Produce sky images over hundreds of deg 2 with realistic distribution of galaxies, stars (weak lensing shear from large scale structure still to come) – Realistic effects: cosmic rays, dead columns, gain variations, seeing variations, saturation, optical distortion, nightly extinction variations, etc Data challenges: DESDM team uses current system to process simulated images to the catalog level and compare catalogs to truth tables

National Center for Supercomputing Applications DES Data Challenges (2) DES DC1 Oct 05 - Jan 06 5 nights simulated data (ImSim1) 700 GB raw 5 nights simulated data (ImSim1) 700 GB raw Basic Orchestration, Condor-G job submission Basic Orchestration, Condor-G job submission First Astro modules: Image Reduction & Cataloguing; First Astro modules: Image Reduction & Cataloguing; Ingestion of objects to Database Ingestion of objects to Database TeraGrid usage: Data parallel computing on NCSA Mercury TeraGrid usage: Data parallel computing on NCSA Mercury Catalogued, calibrated and ingested 50 million objects Catalogued, calibrated and ingested 50 million objects DES DC2 Oct 06 - Jan nights simulated data (5TB, equiv to 20% of SDSS imaging dataset) 10 nights simulated data (5TB, equiv to 20% of SDSS imaging dataset) Astronomy algorithms: astrometry, co-addition, cataloguing Astronomy algorithms: astrometry, co-addition, cataloguing Distributed Archive; high performance GridFTP transfers Distributed Archive; high performance GridFTP transfers Archive Portal Archive Portal Teragrid usage: Production runs on NCSA Mercury; Teragrid usage: Production runs on NCSA Mercury; Pipelines successfully tested on NCSA Cobalt, SDSC IA64 Pipelines successfully tested on NCSA Cobalt, SDSC IA64 Catalogued, calibrated and ingested 250 million objects Catalogued, calibrated and ingested 250 million objects

National Center for Supercomputing Applications DESDM Partnership with OSG DESDM reflects underlying DOE-NSF partnership in DES – Camera construction supported by DOE (Fermilab) – Telescope built and operated by NSF (NOAO) – Data management development supported by NSF (NCSA) DESDM designed to support production, delivery and analysis of science ready data products to all collaboration sites Fermilab and NCSA both host massive HPC resources – Strong incentive for developing DM system that seamlessly integrates TG and OSG resources

National Center for Supercomputing Applications OSG Partnership Components Archiving – Distributed archive already includes node on Fermilab dcache system Simplifies transfer from simulated and reduced images among DES collaborators at NCSA and Fermilab – In operations phase Fermilab and NCSA will both host comprehensive DES data repositories Processing – Large scale reprocessing of DES data would benefit from access to more HPC resources – Orchestration layer already supports processing on multiple (similar) TG nodes – Working to add support for processing on Fermigrid platforms