Semantic Grid + Data Federation US National Virtual Observatory Roy Williams California Institute of Technology NVO co-director

What is NVO? –Standard protocols, standard data types XML transfer protocol (VOTable) Resource description (VOResource etc) Publish/discover to federated registry (OAI) Semantic Types (UCD) Services: Cone search, Simple Image Access –Computing with big data on the Grid Database Crossmatch Image Federation: Atlases

First NVO Discovery

Database Fuzzy Join 2MASS versus SDSS cross- identification with - j_m as 2MASS magnitude and - I_mtotn as SDSS magnitude 2MASS : j_m,+ 15 SDSS: I_mtotn <= 18 Billion Source Cross-Identification: A Computational Challenge SDSS unmatched 2MASS matched SDSS matched 2MASS unmatched

Crossmatch Services SDSS database 2MASS database query Crossmatch service query scientific knowledge! NVO protocols

First NVO Discovery Database crossmatch of two massive databases creates new science “The sum is greater than the parts”

Semantic Grid

Cone Search First VO standard service Input: RA, DEC, SR must be present –decimal degrees J2000 Output: VOTable of sky-located data records –must have columns with UCDs: POS_EQ_RA_MAIN, POS_EQ_DEC_MAIN, ID_MAIN RA=300 DEC=25 SR=0.1 IDRADECxyz Request Response

Cone Search Registry POS_EQ_RA_MAIN POS_EQ_DEC_MAIN POS_EQ ID URLbase RA=200&DEC=20&SR=2 Request: HTTPget of shape: Response: VOTable of shape: A collection of services that have the same shape

Cone Search + Density Probe Cone Search Density Probe baseURL Spacing Search radius interoperating NVO-compliant services! Federation of Multiple Services

NVO Image Protocol SIAP Specify box by position and size SIAP server returns relevant images Footprint Logical Name URL Can choose: standard URL: SRB URL srb://nvo.npaci.edu/…..

Simple Image Access Service Query is sky region May query on image type, image geometry Response is VOTable of images Each has WCS (geometry) parameters Plus a URL to fetch the image Designed for Set of pointed observations (eg Hubble) Wide-area survey (eg Sloan) Image service –Mosaicking –Reprojection

Data Inventory Service What data covers a position in the sky? Registry OAIPublish Registry OAIQuery Registry OAIPublish DIS Caltech NCSA JHU/StSci Goddard

Data Inventory Service Request is a cone on the sky

Data Inventory Service Relevant Images and Catalogs NVSS Image ROSAT catalog

Image Federation

VO Registry Schemas & Service Types VOResourceID ivo://me.com/file123 Query service R R Portals Tools & Services Databases Grid Virtual Data md server for ivo:// VOView Fill-in forms Visualization Reports Publishing OAI Publish service Aladin OASIS DIS

What is in the Registry? Answer: “Entities” It has a global identifier ivo://……. –Must be resolved by authority It has “VOViews” –Queries return these …..and that’s all!

3 Views of an Entitiy Zoo-keeper metadata: carrots yes strong Transportation metadata: 4000 kg no carrots heavy Zoo-manager metadata: per day carrots “entity”

VOResource A mandatory form plus other supporting forms

Schemas and Service Types VOResource –Entity description form Organzation, project, data collection, service Has ivo:// identifier VORegion –sky coverage form (α/δ/λ) VOTable –star catalog, image list, other tables OAI –Registry harvesting –Distributed virtual registry CONE –Request-response for catalog SIAP –Request-response for images When can I publish my own schema to VO?

Dublin Core Metadata Title A name given to the resource. Creator An entity primarily responsible for making the content of the resource. Subject A topic of the content of the resource. Description An account of the content of the resource. Publisher An entity responsible for making the resource available Contributor An entity responsible for making contributions to the content of the resource. Date A date of an event in the lifecycle of the resource. Type The nature or genre of the content of the resource. Format The physical or digital manifestation of the resource. Identifier An unambiguous reference to the resource within a given context. Source A Reference to a resource from which the present resource is derived. Language A language of the intellectual content of the resource. Relation A reference to a related resource. Coverage The extent or scope of the content of the resource. Rights Information about rights held in and over the resource. Curation data for “any human creation”

Dublin Core Dublin Core is how the VO will interoperate with libraries of the world A global metadata standard

Prototype Registry Organization Data Collection Project Service SIA service

VOViews VOResource view Dublin Core view

OAI: Open Archives Initiative Harvesting Protocol OAI is popular –Ask your University librarian Distributed Comprehensive Registry –Harvesting Different views for different purposes –Six blind men and the elephant

OAI Harvesting Protocol 6 magic verbs of OAI

VO Identifiers ivo://mydomain.com / mySkySurvey # file00037.fits URI form Still in flux Authority ID Registered with IVOA Must correspond to a registry Resource ID Created by Authority Resolved by registry Record ID Not known to registry delimiter

Image Federation

Multispectral Imagery Crab Nebula. 3 channels: X-ray in blue, optical in green, and radio in red. Moffet Field California. 224 channels from 400 nm to 2500 nm

Image Federation detection Stacking allows detection of faint sources. A 1-sigma detection in each of many bands becomes a 3- sigma detection. Images of the same galaxy taken several days apart are automatically subtracted from one another, and remaining bright spots may be supernova candidates. (NEAT project) Image subtraction allows detection of narrow-line features that are not also wide-band (eg Hα but not R- band)

Principle Components SDSS (5 channel) SDSS+2MASS (8 channel)

Mosaicking and Federation Every Astronomical image has a different projection different pointing of the telescope We want to mosaic different images We want to federate different information Compute intensive: flux in each pixel is carefully distributed into a new pixel grid Mosaicking Federation Infrared map Xray map today Xray map last year

Atlasmaker Uses Montage, Yoursky Project Estimate & correct Background Co-Add Data Chart David Hockney Pearblossom Highway 1986

Images and Charts Image Big data Chart Map: sphere → plane FITS-WCS header small data An atlas is a collection of charts Hyperatlas is an attempt to standardize atlases

Hyperatlas Standard naming for atlases and vcharts TM-5-SIN-20 Vchart TM-5-SIN Standard Scales: scale s means 2 20-s arcseconds per pixel SIN projection TAN projection TM-5 layout HV-4 layout Standard Projections Standard Layout

Parallel Atlasmaker MPI Parallellism ~2% serial work (Amdahl) Projection is parallel All nodes share filespace Making a single Image  Making an Atlas of 1736 Images Teragrid Distributed Federated Scheduling wanted SRB as Virtual Data Catalog

Atlasmaker Architecture NVO/IVO NED Sloan DPOSS FIRST [2MASS] NVO Protocol making atlas pages scale reproject compress sky index Virtual Data System YourSky VirtualSky Oasis VIEW Bus federation data mining Hyperatlas service SIAP services

Atlasmaker Virtual Data System Metadata repositories Federated by OAI Data repositories Federated by SRB Compute resources Federated by TG/IPG Mosaicked data is on file 2a. Mosaicked data is not on file 2d: Store result & return result 2c: Compute on TG/IPG User request Request manager 2b. Get raw data from NVO resources

Atlasmaker stack Mosaicking (executables) Atlasmaker (script) Hyperatlas (service) NVO Image Access (service) SRB (service) web MontageYourSky Virtual Data System -- Chimera?

Charts and Pages Chart – a frame for specific data Page – an organization for data The virtual disk is 400,000 pixels wide SIN projection

Background Correction UncorrectedCorrected

Montage Background Correction Project pixels to output chart Fit ramps on overlap regions Fit ramps on projected images Subtract from Pixel values