CREATING A MULTI-WAVELENGTH GALACTIC PLANE ATLAS WITH AMAZON WEB SERVICES G. Bruce Berriman, John Good IPAC, California Institute of Technolog y Ewa Deelman,

Slides:



Advertisements
Similar presentations
3 September 2004NVO Coordination Meeting1 Grid-Technologies NVO and the Grid Reagan W. Moore George Kremenek Leesa Brieger Ewa Deelman Roy Williams John.
Advertisements

Pegasus on the Virtual Grid: A Case Study of Workflow Planning over Captive Resources Yang-Suk Kee, Eun-Kyu Byun, Ewa Deelman, Kran Vahi, Jin-Soo Kim Oracle.
Natasha Pavlovikj, Kevin Begcy, Sairam Behera, Malachy Campbell, Harkamal Walia, Jitender S.Deogun University of Nebraska-Lincoln Evaluating Distributed.
The ADAMANT Project: Linking Scientific Workflows and Networks “Adaptive Data-Aware Multi-Domain Application Network Topologies” Ilia Baldine, Charles.
László Dobos 1,2, Tamás Budavári 2, Nolan Li 2, Alex Szalay 2, István Csabai 1 1 Eötvös Loránd University, Budapest,
Ewa Deelman, Integrating Existing Scientific Workflow Systems: The Kepler/Pegasus Example Nandita Mangal,
Solar and STP Physics with AstroGrid 1. Mullard Space Science Laboratory, University College London. 2. School of Physics and Astronomy, University of.
Authors: Thilina Gunarathne, Tak-Lon Wu, Judy Qiu, Geoffrey Fox Publish: HPDC'10, June 20–25, 2010, Chicago, Illinois, USA ACM Speaker: Jia Bao Lin.
e-MERLIN Key Project on Massive Star Formation
M ORPHOLOGY OF G ALACTIC O PEN C LUSTERS Thirty six open clusters were selected from the 2MASS database on the basis of the latest open cluster catalogue.
Science Applications of Montage: An Astronomical Image Mosaic Engine ESTO G. Bruce Berriman, John Good and Anastasia Laity.
The Montage Image Mosaic Service: Custom Mosaics on Demand ESTO John Good, Bruce Berriman Mihseh Kong and Anastasia Laity IPAC, Caltech
Montage: Experiences in Astronomical Image Mosaicking on the TeraGrid ESTO.
Nikolay Tomitov Technical Trainer SoftAcad.bg.  What are Amazon Web services (AWS) ?  What’s cool when developing with AWS ?  Architecture of AWS 
An Astronomical Image Mosaic Service for the National Virtual Observatory
A Grid-Enabled Engine for Delivering Custom Science- Grade Images on Demand
Astro-DISC: Astronomy and cosmology applications of distributed super computing.
An Astronomical Image Mosaic Service for the National Virtual Observatory / ESTO.
Ewa Deelman Using Grid Technologies to Support Large-Scale Astronomy Applications Ewa Deelman Center for Grid Technologies USC Information.
Cloud Computing: Is It Powerful, And Can It Lead To Bankruptcy? G. Bruce Berriman Infrared Processing.
Intermediate HTCondor: Workflows Monday pm Greg Thain Center For High Throughput Computing University of Wisconsin-Madison.
MATE-EC2: A Middleware for Processing Data with Amazon Web Services Tekin Bicer David Chiu* and Gagan Agrawal Department of Compute Science and Engineering.
Astronomical Image Mosaicking ESTO.
Ewa Deelman, Clouds: An Opportunity for Scientific Applications? Ewa Deelman USC Information Sciences.
Scientific Workflows in the Cloud Gideon Juve USC Information Sciences Institute
A Brief Overview by Aditya Dutt March 18 th ’ Aditya Inc.
Scientific Workflows in the Cloud Gideon Juve USC Information Sciences Institute
Star Formation Research Now & With ALMA Debra Shepherd National Radio Astronomy Observatory ALMA Specifications: Today’s (sub)millimeter interferometers.
CONDOR DAGMan and Pegasus Selim Kalayci Florida International University 07/28/2009 Note: Slides are compiled from various TeraGrid Documentations.
Authors: Weiwei Chen, Ewa Deelman 9th International Conference on Parallel Processing and Applied Mathmatics 1.
Pegasus A Framework for Workflow Planning on the Grid Ewa Deelman USC Information Sciences Institute Pegasus Acknowledgments: Carl Kesselman, Gaurang Mehta,
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
Why Build Image Mosaics for Wide Area Surveys? An All-Sky 2MASS Mosaic Constructed on the TeraGrid A. C. Laity, G. B. Berriman, J. C. Good (IPAC, Caltech);
The Grid is a complex, distributed and heterogeneous execution environment. Running applications requires the knowledge of many grid services: users need.
Accessing the Amazon Elastic Compute Cloud (EC2) Angadh Singh Jerome Braun.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Cloud Distributed Computing Platform 2 Content of this lecture is primarily from the book “Hadoop, The Definite Guide 2/e)
Running Scientific Workflow Applications on the Amazon EC2 Cloud Bruce Berriman NASA Exoplanet Science Institute, IPAC Gideon Juve, Ewa Deelman, Karan.
A Framework for Elastic Execution of Existing MPI Programs Aarthi Raveendran Tekin Bicer Gagan Agrawal 1.
 The workflow description modified to output a VDS DAX.  The workflow description toolkit developed allows any concrete workflow description to be migrated.
Experiences Using Cloud Computing for A Scientific Workflow Application Jens Vöckler, Gideon Juve, Ewa Deelman, Mats Rynge, G. Bruce Berriman Funded by.
A Framework for Elastic Execution of Existing MPI Programs Aarthi Raveendran Graduate Student Department Of CSE 1.
Dr. Ahmed Abdeen Hamed, Ph.D. University of Vermont, EPSCoR Research on Adaptation to Climate Change (RACC) Burlington Vermont USA MODELING THE IMPACTS.
Pegasus: Mapping Scientific Workflows onto the Grid Ewa Deelman Center for Grid Technologies USC Information Sciences Institute.
Ten Years of Software Sustainability at The Infrared Processing and Analysis Center G. Bruce Berriman and John Good NASA Exoplanet Science Institute, Infrared.
Combining the strengths of UMIST and The Victoria University of Manchester Adaptive Workflow Processing and Execution in Pegasus Kevin Lee School of Computer.
The New Zealand Institute for Plant & Food Research Limited Use of Cloud computing in impact assessment of climate change Kwang Soo Kim and Doug MacKenzie.
Pegasus: Running Large-Scale Scientific Workflows on the TeraGrid Ewa Deelman USC Information Sciences Institute
The infrared extinction law in various interstellar environments 1 Shu Wang 11, 30, 2012 Beijing Normal University mail.bnu.edu.cn.
CEOS Data Cube Open Source Software Status Brian Killough CEOS Systems Engineering Office (SEO) WGISS-40 Harwell, Oxfordshire, UK September 30, 2015 (remote.
Virtual Observatories, Astronomical Telescopes and Instrumentation (August 2002)1 An Architecture for Access To A Compute Intensive Image Mosaic Service.
Cloud Computing Paradigms for Pleasingly Parallel Biomedical Applications Thilina Gunarathne, Tak-Lon Wu Judy Qiu, Geoffrey Fox School of Informatics,
Parallel Applications And Tools For Cloud Computing Environments CloudCom 2010 Indianapolis, Indiana, USA Nov 30 – Dec 3, 2010.
Funded by the NSF OCI program grants OCI and OCI Mats Rynge, Gideon Juve, Karan Vahi, Gaurang Mehta, Ewa Deelman Information Sciences Institute,
1 USC Information Sciences InstituteYolanda Gil AAAI-08 Tutorial July 13, 2008 Part IV Workflow Mapping and Execution in Pegasus (Thanks.
STELLAR POSITIONS AT LWIR WAVELENGTHS NEW RESULTS USING MSX 6 Dec 00 Dr. Michael P. Egan Technical Advisor Space Infrared Technology CoE Air Force Research.
Observability of YSOs with the WISE and AKARI infrared observatories Sarolta Zahorecz Eötvös University, Budapest PhD student, 3. year Thesis advisor:
HPC In The Cloud Case Study: Proteomics Workflow
11. Looking Ahead.
Pegasus WMS Extends DAGMan to the grid world
AWS Integration in Distributed Computing
By Chris immanuel, Heym Kumar, Sai janani, Susmitha
Cloudy Skies: Astronomy and Utility Computing
Bolocam Galactic Plane Survey Herschel Hi-GAL Plane Survey
Cloud Distributed Computing Environment Hadoop
The “Milky Way”.
Pegasus Workflows on XSEDE
High Throughput Computing for Astronomers
Frieda meets Pegasus-WMS
Presentation transcript:

CREATING A MULTI-WAVELENGTH GALACTIC PLANE ATLAS WITH AMAZON WEB SERVICES G. Bruce Berriman, John Good IPAC, California Institute of Technolog y Ewa Deelman, Gideon Juve, Mats Rynge ISI, University of Southern California Jamie Kinney, Ann Merrihew Amazon Web Services

How Can Astronomers Use The Cloud? How well can we … Manage virtual machines and operate a virtual cluster for processing data. Abstract the technical details of processing from astronomers. Optimize processing and manage costs. Exploit cloud storage in operations. Create turnkey tools for astronomers to exploit cloud platforms.

A Multi-wavelength Image Atlas of the Galactic Plane Use open source tools to create a new data product on the Amazon Elastic Compute Cloud (EC2). 16 wavelengths from 1μm to 24 μm. Coverage l = 0° - 360°; b = ±20°. Processed at 1 arcsec spatial sampling, Cartesian projection, Galactic coordinates.

The Montage Image Mosaic Engine ANSI-C Toolkit for creating and managing image mosaics in FITS format. Portable and scalable – runs on all Unix platforms. BSD 3-clause license. Widely adopted by astronomy and IT communities. 5 Montage Workflow ReprojectionBackground Rectification Co-addition Output Input Three-color Spitzer mosaic of the LMC created from 300,000 frames. Seven degrees on a side. Blue μm; green- 8 μm red - 24 μm

The Pegasus Workflow Management System Pegasus is a workflow planner Mature and used in many fields. Takes abstract descriptions of workflows and sets up and runs a processing plan for an environment whose configuration is incorporated into workflow manager. Layered on HTCondor (scheduler) and DAGMan (submits jobs to HTCondor in the correct order). EC2 environment is already known to Pegasus. Abstract Workflows Directed Acyclical Graphs Identifies the computational flow and dependencies.

Galactic Plane Workflow 16 hierarchical workflows Each one with 1,001 sub- workflows Over 10M input files 45 TB output dataset

System Architecture Design

Be careful about your choices. Amazon offers a wide range of instance types and costs of running applications can vary widely. hi1.4xlarge (the one we used) Memory optimized, with 2 x SSD ephemeral drives. 318,000 core hours. Spot instance price: US $5,950. cc2.8xlarge (benchmarked) Compute cluster optimized, with 4 ephemeral drives (2 used.) 274,000 core hours. Spot instance price: US $2,200. Best practice: do a cost benefit analysis or benchmarking.

Multi-Color Mosaics Top to bottom: 2MASS, GLIMPSE, MSX, WISE

Three color mosaics near NGC 6537 Three color-infrared images of the Galactic Plane, extracted from the 16- wavelength Galactic Plane Atlas. Top left: 2MASS 1.2 μm, 1.6 μm, 2.2 μm Top right: GLIMPSE 4.5 μm, 5.8 μm and 8 μm Bottom left: 2MASS 1.2 μm, GLIMPSE 8.0 μm, MSX 21.3μm

WISE Mosaics Top to bottom: 3.4  m, 4.6  m, 12  m, 22  m.

What Have We Learned? Amazon EC2 is a powerful platform for astronomical computing. Open Source tools are invaluable for taking advantage of it. Optimizing operations requires human intervention. Be mindful of pay-as-you-go: need to do cost benefit analyses, benchmarking for optimization. High latency in recovering images from S3 storage.

Next Generation Montage: Data Cubes The structure of a molecular disk wind in HD , measured by ALMA (PI: M. Rawlings). Re-projection by Montage of a data cube of the star that covers multiple velocities relative to the center of the CO J=3- 2 line. Shown are images at four velocities.

Next Generation Montage: Sky Partitioning Schemes Transform HEALPix images, standard in cosmic background missions, for multi-wavelength image analysis and visualization. 353 GHz all-sky map measured by Planck. Re-project images to TOAST partition scheme and view in immersive platforms such as World Wide Telescope Re-project images to TOAST partition scheme and view in immersive platforms such as World Wide Telescope Re-project images to TOAST partition scheme and view in immersive platforms such as World Wide Telescope Re-project images to TOAST partition scheme and view in immersive platforms such as World Wide Telescope Re-project images to TOAST partition scheme and view in immersive platforms such as World Wide Telescope Re-project images to TOAST partition scheme and view in immersive platforms such as World Wide Telescope Re-project images to TOAST and visualize in World Wide Telescope