Presentation is loading. Please wait.

Presentation is loading. Please wait.

HTCondor in Astronomy at NCSA

Published byἈπφία Σαμαράς Modified over 5 years ago

Similar presentations

Presentation on theme: "HTCondor in Astronomy at NCSA"— Presentation transcript:

1 HTCondor in Astronomy at NCSA
Michael Johnson, Greg Daues, and Hsin-Fang Chiang HTCondor Week 2019

2 The Dark Energy Survey “The Dark Energy Survey (DES) is designed to probe the origin of the accelerating universe and help uncover the nature of dark energy by measuring the 14-billion-year history of cosmic expansion with high precision.” Reference:

3 400+ scientists from 25 institutions in 7 countries
DES Collaboration 400+ scientists from 25 institutions in 7 countries

4 DES: Mapping the Sky DES is a 5.5-year optical survey covering 5000 deg2 of the southern sky Each section of the “footprint” is observed 2x a year in 5 different filters Certain fields are revisited every ~10 nights to search for supernovae The project is very similar to a high-energy physics experiment. The goal is to construct a large and uniform statistical dataset based on observations and derived measurements which can be analyzed for physical phenomena. The manner in which we process and analyze data draws on the HEP/open science community, including HTC.

5 DES: Instrumentation Blanco 4m Telescope @ Sees 20x area of Full Moon!
Cerro Tololo Inter-American Observatory, La Serena, Chile Sees 20x area of Full Moon! 570-Mpix, 62-CCD camera observes in 5-6 filters Reference (left): Reference (right): Reference (bottom): Dark Energy Survey Collaboration

6 DES: Data Management Raw images streamed from Chile → Tucson → NCSA
Images cleaned and millions of stars and galaxies cataloged Over 18,000 images/night 1 TB raw data/night → 5 TB processed data/night Data are archived at NSCA and served to the collaboration for scientific analysis Reference (globe):

7 Data Processing Operational Modes
Run in several processing modes Nightly (within 24 hrs) Initial processing to assess data quality Feedback to mountaintop Annually Latest and greatest calibrated processing over all prior data Basis for internal and public data releases Difference imaging Always some level of processing (multiple pipelines) occurring at any given time. As we near survey’s end, we are running new value-added processing pipelines.

8 DESDM Data Management System
The DESDM system is based on a combined processing framework and data management framework Centralized job configuration and management Data movement orchestration Provenance and metadata collection (Open Provenance Model) Continual data annotation Data lifecycle management The DESDM system allows for configuring and managing the various and simultaneous processing “campaigns” For a given campaign, specify which data to process (via metadata query), which pipelines and configs to use, where to archive the data, where to process data, what provenance to collect Manage relative prioritization of campaigns Annotating outputs for identification of data used for downstream processes (e.g., QA, release prep, data management activities) Centralized data management, distributed computing

9 DES and Beyond DES is scheduled to end around 2021, but DECam is still a world- class instrument and will continue to be used for many more years. Want to leverage our data management system for future needs: Processing public DECam data sets to complement and expand DES (DECADE) On-sky DECam follow-up for optical MMA As future surveys come online can we use DECam as a follow-up instrument? Are there other programs/initiatives that can make use of our system and take advantage of the knowledge we’ve gained processing for DES?

10 DESDM Workflow & HTCondor Infrastructure
Python wrapping HTCondor DAGMan submits Nested DAG workflow for each Unit (Exposure, Tile, etc) Numerous DAGs, No Overarching Workflow Throttling Issues for PRE/POST Submit Side Infrastructure Separate Central Manager (collector, negotiator) Two largish Submit nodes (schedd) Multi-schedd process configuration (~OSG Login) File Staging/Transfer No-shared-filesystem processing Data staged in & out via Curl/webdav

11 DESDM HTCondor Infrastructure & Platforms
Illinois Campus Cluster Models: Investor, RCaaS, etc. DESDM as an investor: provisions ~ 32 nodes, 900 cores, CentOS7 Main ICCP has PBS scheduler, DESDM nodes managed separately DESDM Condor Pool - Partitionable Slots Compute jobs run on Local Scratch Disk Machine Ads for Processing Type/Campaign Jobs of a species sent to targeted nodes (e.g., avoid defrag issues) Best for ‘realtime’, quick turn around

12 DESDM HTCondor Infrastructure & Platforms
Blue Waters Computing System at NPCF DESDM works with Innovation and Exploration Allocation HTCondor Glide-ins submitted through PBS scheduler Glide-in setup a driver for RSIP solution (general workflows) HTCondor Execute directories on shared Lustre file system Scale constrained by metadata server FermiGrid HTCondorCE : JobRouter to DES Nodes DES Virtual Organization Software Stacks in CVMFS /cvmfs/des.opensciencegrid.org DESDM Software Services FHNW-Zurich

13 DESDM HTCondor Infrastructure & Platforms
DESDM setup for use of Open Science Grid DESDM as a OSG project Submit Node with Flocking setup FLOCK_TO = flock.opensciencegrid.org Data Origin for utilizing StashCache infrastructure K8s worker node - OSG pods on PRP Kubernetes Cluster Registered /cvmfs/desdm.osgstorage.org in DES VO DESDM setup for OSG prototype for other efforts at NCSA

14 DESDM HTCondor Infrastructure & Platforms
Testing DESDM with HTCondor, condor_annex on AWS Single Exposure test with DESDM framework on AWS EC2 instance, used Singularity Glide-in to ‘production pool’ Testing condor_annex in ‘personal condor’ Default HTCondor 8.6.x, Amazon Linux Customized AMI with HTCondor 8.8.x, Amazon Linux 2 Encryptfs issue Need to examine annex to ‘production pool’ / HTCondor as root

15 Large Synoptic Survey Telescope
A dedicated 10yr Deep-Wide-Fast Survey LSST v.s. DES: 2 times mirror size, 5 times pixels LSST can obtain “DES” in 1.5 months 4 times larger area Repeat the full sky every 3-4 nights Open data, open source Science operations starts in 2023 ~200,000 images per night Raw data ~20TB per night 60PB of raw image data 500PB of final image data

16 Large Synoptic Survey Telescope
Data Processing & Workflow management 11 Public Data Releases Proof-of-concept with DESDM system Customization to the DESDM system would be needed for LSST Proof-of-concept with HTCondor + Pegasus on AWS Exploration just started this month Plan to use HTCondor Annex Plan to use S3 storage Decision not finalized

Download ppt "HTCondor in Astronomy at NCSA"

Similar presentations

Overview of Wisconsin Campus Grid Dan Bradley Center for High-Throughput Computing.

Overview of Wisconsin Campus Grid Dan Bradley Center for High-Throughput Computing.
The Dark Energy Survey The Big Questions The Discovery of Dark Energy The Dark Energy Survey – The telescope – The camera – The science Expected Results.

The Dark Energy Survey The Big Questions The Discovery of Dark Energy The Dark Energy Survey – The telescope – The camera – The science Expected Results.
Astronomy of the Next Decade: From Photons to Petabytes R. Chris Smith AURA Observatory in Chile CTIO/Gemini/SOAR/LSST.

Astronomy of the Next Decade: From Photons to Petabytes R. Chris Smith AURA Observatory in Chile CTIO/Gemini/SOAR/LSST.
Introduction The Open Science Grid (OSG) is a consortium of more than 100 institutions including universities, national laboratories, and computing centers.

Introduction The Open Science Grid (OSG) is a consortium of more than 100 institutions including universities, national laboratories, and computing centers.
1 LSST: Dark Energy Tony Tyson Director, LSST Project University of California, Davis Tony Tyson Director, LSST Project University of California, Davis.

1 LSST: Dark Energy Tony Tyson Director, LSST Project University of California, Davis Tony Tyson Director, LSST Project University of California, Davis.
Building a Framework for Data Preservation of Large-Scale Astronomical Data ADASS London, UK September 23-26, 2007 Jeffrey Kantor (LSST Corporation), Ray.

Building a Framework for Data Preservation of Large-Scale Astronomical Data ADASS London, UK September 23-26, 2007 Jeffrey Kantor (LSST Corporation), Ray.
Astro-DISC: Astronomy and cosmology applications of distributed super computing.

Astro-DISC: Astronomy and cosmology applications of distributed super computing.
Aus-VO: Progress in the Australian Virtual Observatory Tara Murphy Australia Telescope National Facility.

Aus-VO: Progress in the Australian Virtual Observatory Tara Murphy Australia Telescope National Facility.
CREATING A MULTI-WAVELENGTH GALACTIC PLANE ATLAS WITH AMAZON WEB SERVICES G. Bruce Berriman, John Good IPAC, California Institute of Technolog y Ewa Deelman,

CREATING A MULTI-WAVELENGTH GALACTIC PLANE ATLAS WITH AMAZON WEB SERVICES G. Bruce Berriman, John Good IPAC, California Institute of Technolog y Ewa Deelman,
CONDOR DAGMan and Pegasus Selim Kalayci Florida International University 07/28/2009 Note: Slides are compiled from various TeraGrid Documentations.

CONDOR DAGMan and Pegasus Selim Kalayci Florida International University 07/28/2009 Note: Slides are compiled from various TeraGrid Documentations.
Open Science Grid For CI-Days Internet2: Fall Member Meeting, 2007 John McGee – OSG Engagement Manager Renaissance Computing Institute.

Open Science Grid For CI-Days Internet2: Fall Member Meeting, 2007 John McGee – OSG Engagement Manager Renaissance Computing Institute.
National Center for Supercomputing Applications Observational Astronomy NCSA projects radio astronomy: CARMA & SKA optical astronomy: DES & LSST access:

National Center for Supercomputing Applications Observational Astronomy NCSA projects radio astronomy: CARMA & SKA optical astronomy: DES & LSST access:
1 New Frontiers with LSST: leveraging world facilities Tony Tyson Director, LSST Project University of California, Davis Science with the 8-10 m telescopes.

1 New Frontiers with LSST: leveraging world facilities Tony Tyson Director, LSST Project University of California, Davis Science with the 8-10 m telescopes.
1 Radio Astronomy in the LSST Era – NRAO, Charlottesville, VA – May 6-8 th 2013. LSST Survey Data Products Mario Juric LSST Data Management Project Scientist.

1 Radio Astronomy in the LSST Era – NRAO, Charlottesville, VA – May 6-8 th LSST Survey Data Products Mario Juric LSST Data Management Project Scientist.
National Center for Supercomputing Applications University of Illinois at Urbana-Champaign The Dark Energy Survey Middleware LSST Workflow Workshop 09/2010.

National Center for Supercomputing Applications University of Illinois at Urbana-Champaign The Dark Energy Survey Middleware LSST Workflow Workshop 09/2010.
Introduction to Apache OODT Yang Li Mar 9, 2012. What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.

Introduction to Apache OODT Yang Li Mar 9, What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.
1 Evolution of OSG to support virtualization and multi-core applications (Perspective of a Condor Guy) Dan Bradley University of Wisconsin Workshop on.

1 Evolution of OSG to support virtualization and multi-core applications (Perspective of a Condor Guy) Dan Bradley University of Wisconsin Workshop on.
Open Science Grid For CI-Days Elizabeth City State University Jan-2008 John McGee – OSG Engagement Manager Manager, Cyberinfrastructure.

Open Science Grid For CI-Days Elizabeth City State University Jan-2008 John McGee – OSG Engagement Manager Manager, Cyberinfrastructure.
Sep 21, 20101/14 LSST Simulations on OSG Sep 21, 2010 Gabriele Garzoglio for the OSG Task Force on LSST Computing Division, Fermilab Overview OSG Engagement.

Sep 21, 20101/14 LSST Simulations on OSG Sep 21, 2010 Gabriele Garzoglio for the OSG Task Force on LSST Computing Division, Fermilab Overview OSG Engagement.
CRISP & SKA WP19 Status. Overview Staffing SKA Preconstruction phase Tiered Data Delivery Infrastructure Prototype deployment.

CRISP & SKA WP19 Status. Overview Staffing SKA Preconstruction phase Tiered Data Delivery Infrastructure Prototype deployment.

Similar presentations

Ads by Google