Presentation is loading. Please wait.

Presentation is loading. Please wait.

Grid Astronomy with Image Federation Roy Williams Michael Feldmann California Institute of Technology.

Published byBret Jacka Modified over 10 years ago

Similar presentations

Presentation on theme: "Grid Astronomy with Image Federation Roy Williams Michael Feldmann California Institute of Technology."— Presentation transcript:

1 Grid Astronomy with Image Federation Roy Williams Michael Feldmann California Institute of Technology

2 This Presentation Why (Roy Williams) –Image Federation –Standard protocols, standard data types –Grid Services: eg. Simple Image Access (SIA) –Image Mosaicking with Atlasmaker How (Mike Feldmann) –Condor –Queues –SRB –????

3 1. NVO Image Service: Federation Needs Standards

4 NVO Image Protocol SIAP Specify box by position and size SIAP server returns relevant images Footprint Logical Name URL Can choose: standard URL: http://....... SRB URL srb://nvo.npaci.edu/…..

5 Simple Image Access Service Query is sky region May query on image type, image geometry Response is VOTable of images Each has WCS (geometry) parameters Plus a URL to fetch the image Designed for Set of pointed observations (eg Hubble) Wide-area survey (eg Sloan) Image service –Mosaicking –Reprojection

6 2. Atlasmaker: Grid-based Image Federation

7 Image Federation

8 Multispectral Imagery Crab Nebula. 3 channels: X-ray in blue, optical in green, and radio in red. Moffet Field California. 224 channels from 400 nm to 2500 nm

9 Image Federation detection Stacking allows detection of faint sources. A 1-sigma detection in each of many bands becomes a 3- sigma detection. Images of the same galaxy taken several days apart are automatically subtracted from one another, and remaining bright spots may be supernova candidates. (NEAT project) Image subtraction allows detection of narrow-line features that are not also wide-band (eg Hα but not R- band)

10 Multi-Wavelength Image Morphology DPOSS-2MASS Image Mosaics J F N J H K Galaxy identifcation, galaxy clusters Pattern matching with shape AND color

11 Galaxy cluster in Xray, DPOSS, Sloan This is M77 & has Xray and Radio too Coma Virtual Sky http://virtualsky.org

12 Virtual Sky: Image Federation Xray (ROSAT) theme Change scale Change theme http://virtualsky.org/ from Caltech CACR Caltech Astronomy Microsoft Research Optical (DPOSS) Coma cluster Virtual Sky has 14,000,000 tiles 140 Gbyte

13 Image Federation

14 Virtual Sky Roy Williams, Caltech CACR Alex Szalay, Johns Hopkins University Ashish Mahabal, Caltech Astronomy Jim Gray, Microsoft Research George Djorgovski, Caltech Astronomy Julian Bunn, Caltech CACR Robert Brunner, Caltech Astronomy

15 Multichannel Images L is 23 cm wavelength C is 10 cm wavelength H is horizontal polarization V is vertical polarization A color image is 3 channels Principle components – Information concentration

16 Principle Components SDSS (5 channel) SDSS+2MASS (8 channel)

17 Mosaicking and Federation Every Astronomical image has a different projection different pointing of the telescope We want to mosaic different images We want to federate different information Compute intensive: flux in each pixel is carefully distributed into a new pixel grid Mosaicking Federation Infrared map Xray map today Xray map last year

18 Atlasmaker Uses Montage, Yoursky Project Estimate & correct Background Co-Add Data Chart David Hockney Pearblossom Highway 1986

19 Background Correction UncorrectedCorrected

20 Montage Background Correction Project pixels to output chart Fit ramps on overlap regions Fit ramps on projected images Subtract from Pixel values

21 Images and Charts Image Big data Chart Map: sphere plane FITS-WCS header small data An atlas is a collection of charts Hyperatlas is an attempt to standardize atlases

22 Hyperatlas Standard naming for atlases and vcharts TM-5-SIN-20 Vchart TM-5-SIN-20-1589 Standard Scales: scale s means 2 20-s arcseconds per pixel SIN projection TAN projection TM-5 layout HV-4 layout Standard Projections Standard Layout

23 Charts and Pages 1. Chart – a metadata entity specifying a map from sphere to plane 2. Page – a sized chart chosen from a standard set – an Atlas The virtual disk is 400,000 pixels wide SIN projection 3. A Tiling of a Page

24 Over to Mike

25 Atlasmaker DAG (tile level) An Atlasmaker tile is some region of the sky. Can retrieve needed images for a given page with SIAP services. Images returned by SIAP server for a page query establish dependency graph for the construction of this page. A tile is simply a subset of this page. Atlasmaker tile Survey images Neighboring Atlasmaker tile Survey images overlap multiple tiles

26 Atlasmaker DAG (tile to page) An Atlasmaker page is made up of many smaller tiles. The page defines the projection (Montage) A tile is a simple unit of work to be completed (easy to deal with computationally) Page Many tiles

27 Atlasmaker DAG (tile level) Tile A Tile B survey images Raw images on page border must be projected twice. Images on the border of two tiles of the same page do not need to be projected twice (this efficiency requires image product archival and good bookkeeping).

28 Atlasmaker DAG (page and up) coarse resolution page many levels resolution... high integrity imagescompressed images Web based navigation tool (page-level DAG)

29 Atlasmaker DAG Coarse resolution Pages Tiles Projected Images Raw Images Web based navigation tool Raw images archived Large dataset management (SRB) SIAP web service retrieval establishes dependencies Image compression Montage Projection defined Reduce resolution

30 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

31 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Probe top compressed Image

32 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

33 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

34 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

35 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

36 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

37 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

38 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

39 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

40 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

41 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

42 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

43 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Probe all compressed Images

44 Atlasmaker DAG Construction Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Complete!!!

45 Something breaks!!! Some survey images might have been flawed A valid state might have been set by a failed computation New/better images now exist We do not want to recalculate entire atlas!

46 Oops! Why Virtual Data?

47 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Discover an invalid state!!

48 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Propagate invalid state

49 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Fix invalid raw image

50 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Probe all compressed Images that are not in a valid state

51 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

52 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

53 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

54 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

55 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

56 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

57 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

58 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state

59 Atlasmaker DAG Maintenance Course resolution Pages Tiles Projected Images Raw Images Valid state Working state Initial state Invalid state Fixed!!!

60 Computational Approaches Course resolution Pages Tiles Projected Images Raw Images Web based navigation tool Promising Bag of Tasks locations. MW-based parallelism is very easy!!!

61 Serial Version First stab at Atlasmaker Allowed us to build a base of python modules used throughout this project We ran small sections of sky on single machine Assumed common filesystem We did our own bookkeeping Can run on any machine Slow!!!

62 MPI Version Take advantage of simple MW parallelism Machine must have MPI Assumed common filesystem Typical run on tightly-coupled machines MPI jobs needed to be small enough to avoid being brittle (hardware failures?) Generally limited to doing only small part of DAG

63 PBS/MPI Version Typical run on tightly-coupled machines Many smaller MPI jobs submitted to PBS Job-Manager parallelism Assumed common filesystem Requires fair amount of bookkeeping Tile is smallest unit of work Typical to submit a Tile or Page as single MPI job

64 PBS/Serial Version Not dependent on MPI Assumed common filesystem Just as (even more) efficient as MPI/PBS parallelism Can make a single image project the smallest unit of work (*still requires tile assembly*) Typically make a Tile a unit of work Roughly the same amount of bookkeeping as PBS/MPI version Many small PBS jobs often get picked up quickly System Administrators feel uncomfortable with many small jobs from a single user Queue policy often set against this usage

65 Condor/Serial Version Just like PBS/Serial version but submit all jobs to Condor job-manager instead of PBS job-manager Need machines with Condor Requires learning Condor Does not assume common filesystem Condor can leverage huge number of idle cycles (we are doing our part to keep very loosely-coupled applications off over- burdened tightly-coupled machines )

66 DAGMAN/Condor Version Requires using DAGMAN/Condor Need machines with DAGMAN/Condor We are using Condor discovered cycles Requires learning DAGMAN/Condor Does not assume common filesystem All the bookkeeping is done for us!

67 Pros and Cons MethodProsCons Serial Simple to codeSlow Assumes local filesystem MPI ParallelLimit to job submission size Limit to MPI supported machines Uses tightly-coupled resource Assumes common filesystem PBS/Serial Simple to code Job manager parallelism Bookkeeping Interactions with PBS not natural Assumes common filesystem Uses tightly-coupled resource PBS/MPI Roughly the same as PBS/Serial Bookkeeping Limit to MPI supported machines Assumes common filesystem Uses tightly-coupled resource Condor/Serial Condor discovered cpu cycles Does not assume common filesystem Bookkeeping Limit to Condor supported machines Condor/DAGMAN Almost no bookkeeping Condor discovered cpu cycles Does not assume common filesystem Limit to Condor supported machines

68 Atlasmaker to-do list Complete/test of Condor version Write DAGMAN version Continued refinement of data retrieval Establish useful meta-data for previously projected images and image products Get all surveys in reliable/redundant locations Construction entire atlas

69 Atlasmaker wish list Wider Condor/DAGMAN usage 100% reliable low latency access to raw data and constructed image products Resource broker to tell me where to submit my bag of tasks (in spirit of PBS/serial and MPI/serial versions)

70 Conclusions Computational approach depends on resources you can use Atlasmaker is still a work in progress

Download ppt "Grid Astronomy with Image Federation Roy Williams Michael Feldmann California Institute of Technology."

Similar presentations

Trying to Use Databases for Science Jim Gray Microsoft Research

Trying to Use Databases for Science Jim Gray Microsoft Research
Online Science -- The World-Wide Telescope Archetype

Online Science -- The World-Wide Telescope Archetype
World Wide Telescope mining the Sky using Web Services Information At Your Fingertips for astronomers Jim Gray Microsoft Research Alex Szalay Johns Hopkins.

World Wide Telescope mining the Sky using Web Services Information At Your Fingertips for astronomers Jim Gray Microsoft Research Alex Szalay Johns Hopkins.
1 Online Science -- The World-Wide Telescope as an Archetype Jim Gray Microsoft Research Collaborating with: Alex Szalay, Peter Kunszt, Ani

1 Online Science -- The World-Wide Telescope as an Archetype Jim Gray Microsoft Research Collaborating with: Alex Szalay, Peter Kunszt, Ani
1 Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

1 Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research
Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research
3 September 2004NVO Coordination Meeting1 Grid-Technologies NVO and the Grid Reagan W. Moore George Kremenek Leesa Brieger Ewa Deelman Roy Williams John.

3 September 2004NVO Coordination Meeting1 Grid-Technologies NVO and the Grid Reagan W. Moore George Kremenek Leesa Brieger Ewa Deelman Roy Williams John.
Aug 3-4, 2003 Victoria1 Montage, Pegasus and ROME G. B. Berriman, J.C. Good, M. Kong, A. Laity IPAC/Caltech J. Jacob, A. Bergou, D. S. Katz JPL R. Williams.

Aug 3-4, 2003 Victoria1 Montage, Pegasus and ROME G. B. Berriman, J.C. Good, M. Kong, A. Laity IPAC/Caltech J. Jacob, A. Bergou, D. S. Katz JPL R. Williams.
The NVO Data Discovery Portal Tom McGlynn NASA/GSFC.

The NVO Data Discovery Portal Tom McGlynn NASA/GSFC.
NVO Summer School, Aspen Sept 14 2004 1 Data Access Layer Working Group Image and Spectral Access Doug Tody National Radio Astronomy Observatory National.

NVO Summer School, Aspen Sept Data Access Layer Working Group Image and Spectral Access Doug Tody National Radio Astronomy Observatory National.
John Cunniffe Dunsink Observatory Dublin Institute for Advanced Studies Evert Meurs (Dunsink Observatory) Aaron Golden (NUI Galway) Aus VO 18/11/03 Efficient.

John Cunniffe Dunsink Observatory Dublin Institute for Advanced Studies Evert Meurs (Dunsink Observatory) Aaron Golden (NUI Galway) Aus VO 18/11/03 Efficient.
What does LOFAR have to do with the Virtual Observatory (VO)? LOFAR Science Day 16 December 2003 Melbourne David Barnes The University of Melbourne.

What does LOFAR have to do with the Virtual Observatory (VO)? LOFAR Science Day 16 December 2003 Melbourne David Barnes The University of Melbourne.
WEB DESIGN TABLES, PAGE LAYOUT AND FORMS. Page Layout Page Layout is an important part of web design Why do you think your page layout is important?

WEB DESIGN TABLES, PAGE LAYOUT AND FORMS. Page Layout Page Layout is an important part of web design Why do you think your page layout is important?
The zooniverse.org real science online. The Zooniverse is a collection of websites where members of the public are asked to look at data and interpret.

The zooniverse.org real science online. The Zooniverse is a collection of websites where members of the public are asked to look at data and interpret.
Jeroen Stil Department of Physics & Astronomy University of Calgary Stacking of Radio Surveys.

Jeroen Stil Department of Physics & Astronomy University of Calgary Stacking of Radio Surveys.
Condor and GridShell How to Execute 1 Million Jobs on the Teragrid Jeffrey P. Gardner - PSC Edward Walker - TACC Miron Livney - U. Wisconsin Todd Tannenbaum.

Condor and GridShell How to Execute 1 Million Jobs on the Teragrid Jeffrey P. Gardner - PSC Edward Walker - TACC Miron Livney - U. Wisconsin Todd Tannenbaum.
October 12, 2003ADASS NVO Tutorial1 How-To Implement Cone and SIA Services Gretchen Greene Space Telescope Science.

October 12, 2003ADASS NVO Tutorial1 How-To Implement Cone and SIA Services Gretchen Greene Space Telescope Science.
Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 1 Zhilin Li & Kourosh Khoshelham Dept of Land Surveying & Geo-Informatics.

Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 1 Zhilin Li & Kourosh Khoshelham Dept of Land Surveying & Geo-Informatics.
Morris LeBlanc.  Why Image Retrieval is Hard?  Problems with Image Retrieval  Support Vector Machines  Active Learning  Image Processing ◦ Texture.

Morris LeBlanc.  Why Image Retrieval is Hard?  Problems with Image Retrieval  Support Vector Machines  Active Learning  Image Processing ◦ Texture.
Grist: Grid Data Mining for Astronomy Joseph Jacob, Daniel Katz, Craig Miller, and Harshpreet Walia (JPL) Roy Williams, Matthew Graham, and Michael Aivazis.

Grist: Grid Data Mining for Astronomy Joseph Jacob, Daniel Katz, Craig Miller, and Harshpreet Walia (JPL) Roy Williams, Matthew Graham, and Michael Aivazis.

Similar presentations

Ads by Google