1. National Center for Supercomputing Applications University of Illinois at Urbana-Champaign The Dark Energy Survey Middleware LSST Workflow Workshop 09/2010 Michelle Gower NCSA on behalf of the DES Data Management Team and DES Collaboration

2. DES Data Management Observing from 2011-2016: 200 TB raw data, 4 PB of products 100 TB object catalog 120+ CPU years/year of processing Processing includes: Removing the instrument signature Removing artifacts (e.g. planes) Calibrating/registering Feature detection Feature analysis Monitoring/quality assessment

3. Requirements Level of parallelism changes through processing Easily modified by scientists Local and remote clusters Work with project’s Archive system Monitoring of jobs while they’re running Less research/More production

4. Processing Framework Overview 4 Archive Nodes Target (HPC)‏ Machines Notification Service Database Pipelines of AppModules Event Monitor Archive Portal Orchestration DAF

5. Middleware Workflow Condor DAGman Job submission Condor-G to pre-WS GRAM for TeraGrid resources Condor (vanilla jobs) for local machines File transfer GridFTP using clients uberftp and globus-url-copy Runtime Monitoring Elf/Ogrescript

6. Example: Module Definition output_dir = ${archive_root}/${run_dir}/raw exec=${des_home}/bin/crosstalk.pl args= -archiveroot ${archive_root} -list ${crosstalk_list} -binpath ${des_home}/bin –outputpath ${output_dir} -detector ${detector} -crosstalk ${crosstalk_file} -photflag ${photflag} –verbose ${verbose} wall_mod = 30 basename crosstalk fileclass = src filetype = src query_fields = fileclass,filetype,project,nite min_num_per_job = 5 fileclass = cal filetype = xtalk query_fields = fileclass,filetype,project,detector

7. Workflow CrosstalkCreateCorImCorrect MaskingAstroRefine RemapPSFModelWeakLensing Make Bkgd

8. Run queries to get master input file lists Stage input images to target machine Generate Jobs: input lists, job descriptions, DAG Setup target machine Stage generated lists and files to target machine Make timestamp on target machine Target jobs run Ingestion of new or modified files Job Generation

9. Upcoming Orchestration Work Skipping modules Repeatedly using set of blocks (Campaign processing) Let condor manage vanilla jobs per machine threaded vs serial Finer grain restarts

10. Notification Events System AppModule (Elf/OgreScript) AppModule (Elf/OgreScript) Science Code ActiveMQ Message Bus ActiveMQ Message Bus Event Store Portal Monitor

11. DES Monitor Portal Current Technology Drupal CMS / PHP, Javascript/Ajax Tools for Viewing Active Processing Status High level Alert Monitoring Quick navigation to event logs Quality Assurance Profiles Histograms of QA metrics, outlier detection Processing Timing Summary Report Middleware level timing profile

12. Acknowledgements DESDM Team: Jim Myers, Terry McLaren Joe Mohr Bob Armstrong, Dora Cai, Ankit Chandra, Greg Daues, Shantanu Desai, Michelle Gower, Wayne Hoyenga, Chit Khin, Kailash Kotwani Past DESDM Team Members, DES Project Team and Collaboration Members National Science Foundation http://cosmology.illinois.edu/DES/