Web Services for the National Virtual Observatory Tamás Budavári Johns Hopkins University

3/31/2005Tamás Budavári (JHU)2 Outline Motivations and trends How will the VO work? The NVO (hi)story Sample VO services Where we are going

3/31/2005Tamás Budavári (JHU)3 The World is Exponential Astrophysical data is growing exponentially Doubling every year (Moore ’ s Law+): both data sizes and number of data sets Computational resources scale the same way Constant $$$ will keep up with the data Main problem is the software component Currently components are not reused Software costs are increasingly larger fraction Aggregate costs are growing exponentially

3/31/2005Tamás Budavári (JHU)4 Making Discoveries When and where are discoveries made? Always at the edges and boundaries Going deeper, using more colors … Metcalfe ’ s law Utility of computer networks grows as the number of possible connections: O(N 2 ) VO: Federation of N archives Possibilities for new discoveries grow as O(N 2 ) Current sky surveys have proven this Very early discoveries from SDSS, 2MASS, DPOSS

3/31/2005Tamás Budavári (JHU)5 Publishing Data Roles Authors Publishers Curators Consumers Traditional Scientists Journals Libraries Scientists Emerging Collaborations Project website Bigger archives Scientists

3/31/2005Tamás Budavári (JHU)6 Changing Roles Exponential growth: data never centralized A project last 3-5 years (linear rate but new projects) Data sent upwards only at the end of the project More responsibility on projects Becoming Publishers and Curators Larger fraction of budget spent on software More standards are needed Easier data interchange, fewer tools More templates are needed Recycling software components

3/31/2005Tamás Budavári (JHU)7 Emerging New Concepts Standardizing distributed data Web Services, supported on all platforms Custom configure remote data dynamically XML: Extensible Markup Language SOAP: Simple Object Access Protocol WSDL: Web Services Description Language Standardizing distributed computing Grid Services Custom configure remote computing dynamically Build your own remote computer (and discard) Virtual Data: new data sets on demand

3/31/2005Tamás Budavári (JHU)8 Shielding Users Users do not want to deal with … XML – they just want their data Configuring grid computing – want results Web services Data appears in user memory, XML is invisible SOAP: just a remote procedure call

3/31/2005Tamás Budavári (JHU)9 NVO: How Will It Work? Define commonly used ‘ core ’ services Build higher level toolboxes/portals on top We do not build ‘ everything for everybody ’ Use the rule: Define the standards and interfaces Build the framework Build the 10% of services that are used by 90% Let the users build the rest from the components

3/31/2005Tamás Budavári (JHU)10 Atomic Services Metadata information about resources Wavelength: optical, ultraviolet, etc. Sky coverage: where on the sky? Translation of names to universal dictionary (UCD) Simple search patterns on the resources Cone Search Image mosaic Unit conversions Simple filtering, statistics, histograms On-the-fly recalibrations

3/31/2005Tamás Budavári (JHU)11 Higher Level Services Built on basic Core Services Perform more complex tasks, e.g. Automated resource discovery Cross-identifications Photometric redshifts Outlier detections Visualization facilities Expectation: Build custom services in days from existing building blocks (like today in IRAF or IDL)

3/31/2005Tamás Budavári (JHU)12 Early Standards VOTable Universal container for tables (in XML) First VO standard (from the DTD era) ConeSearch Simple catalog access based on location First VO standard interface (http get) Many implemented them!

3/31/2005Tamás Budavári (JHU)13 VO Registry Searchable (fast) Various resources OAI harvesting Web site and Web services 5,459 resources

3/31/2005Tamás Budavári (JHU)14 Sloan Digital Sky Survey The cosmic genome project Map the northern sky ~7000 sq.deg Sophisticated science archive (SQL) Many ways to query and filter data Various data types: Images, spectra, catalogs

3/31/2005Tamás Budavári (JHU)15 Courtesy of Robert Lupton

3/31/2005Tamás Budavári (JHU)16 SDSS Image Cutout Build JPEG mosaics For given (RA, Dec) Arbitrary zoom levels Overplot catalog data Overplot geometry MapQuest like navigation List of objects Web services JPEG returned in DIME or byte array in XML Kudos to Maria Nieto-Santisteban

3/31/2005Tamás Budavári (JHU)17 Spectrum Services Public repository 500,000 spectra (2B row) Spatial search and more Register & submit yours Web site On-the-fly plotting Building composites Web services More options and tools Kudos to László Dobos

3/31/2005Tamás Budavári (JHU)18 Catalog Access ConeSearch Very limited but in place Many resources available Need simple queries SQL-like queries, e.g. SELECT FROM WHERE SELECT FROM WHERE What else we need? To specify area on sky Could be very complicated Standard interface to DBs XML Web Services for interop Tools to federate archives Dynamically for flexibility

3/31/2005Tamás Budavári (JHU)19 Catalog Access: New Standard Virtual Observatory Query Language Level 3. Semantically rich XML language (TBD) Level 2. Portals combining astro archives e.g. SkyQuery and Open SkyQuery Level 1. Query a single database Astronomical Data Query Language  VOTable

3/31/2005Tamás Budavári (JHU)20 Open SkyQuery SkyNode interface to archives Implements ADQL returns VOTable Basic node understands “ REGION ” Full node understands “ XMATCH ” SkyQuery portal Knows the SkyNodes from Registry Understands federated query

3/31/2005Tamás Budavári (JHU)21 Architecture Image cutout SkyNode SDSS SkyNode 2MASS SkyNode FIRST SkyQuery Web App

3/31/2005Tamás Budavári (JHU)22 SkyNode Metadata functions Tables, Columns, Functions Uptime, Footprint, etc … Query functions PerformQuery (ADQL q) Simple query interface QueryCost (ADQL q) Cost estimate for query ExecutePlan (XPlan plan) Recursive probabilistic join

3/31/2005Tamás Budavári (JHU)23 Data flow SkyQuery query SkyNode 1 SkyNode 2 SkyNode 3 Registry

3/31/2005Tamás Budavári (JHU)24 SkyQuery portal Hides complexity Simple web services Methods Get to metadata of nodes Find location of nodes Send query to one node Send federated query Create execution plan Submit execution plan

3/31/2005Tamás Budavári (JHU)25

3/31/2005Tamás Budavári (JHU)26 Other VO Services Density map plotting Cosmic distance calc. Footprint intersection Wrappers for legacy TCP services, e.g. NED Analysis tools & libraries e.g. CFITSIO (in C#)

3/31/2005Tamás Budavári (JHU)27 NVO Summer School Aspen, Colorado September 2004 Full house ~40 “ students ”

3/31/2005Tamás Budavári (JHU)28 Questionnaire Very good scores !!! Needs to be longer (10d) Great software package ZIP file on the NVO twiki Proceedings On the Web H ERE H ERE NVO Summer School All possible OS Real test of interoperability Challenging to teach WS learning curve Depending on background Winning projects C. Miller & S. Krughoff Extinction map service R. Lucas and friends VLA science project

3/31/2005Tamás Budavári (JHU)29 NVO Comes of Age Special NVO session at AAS ‘ 05 Featuring ~10 science talks using NVO technology First refereed astronomy publications Using the NVO framework Next Summer School is being organized NASA+NSF: Long Term NVO Facility Starting in 2006

3/31/2005Tamás Budavári (JHU)30 NVO is here!

3/31/2005Tamás Budavári (JHU)31 Where to go… Links in the talk