The National Virtual Observatory When and where are discoveries made? Always at the edges and boundaries Going deeper, using more colors…. Metcalfes law Utility of computer networks grows as the number of possible connections: O(N 2 ) VO: Federation of N archives Possibilities for new discoveries grow as O(N 2 ) Current sky surveys have proven this amplification Very early discoveries from SDSS, 2MASS Exponential growth Data will be never centralized More responsibility on projects Becoming Publishers and Curators Larger fraction of budget spent on software Lot of development duplicated, wasted More standards are needed Easier data interchange, fewer tools More templates are needed Develop less software on your own Astrophysical data is growing exponentially Doubling every year (Moores Law): both data sizes and number of data sets Computational resources scale the same way Constant $$$ will keep up with the data Main problem is the software component Currently components are not reused Software costs are increasingly larger fraction Aggregate costs are growing exponentially Roles Authors Publishers Curators Consumers Traditional Scientists Journals Libraries Scientists Emerging Collaborations Project www site Bigger Archives Scientists Trends Discoveries Data Publishing Changing Roles Astrophysics has good track record FITS: universally used to share low level data Individual images, tables, files But: new industry standards emerging XML, SOAP Required by modern data exchange More dynamic (streams, queries) Merging heterogeneous sources Good time to adopt… Accessing remote data: WWW, FTP Data formatted in certain ways (HTML, FITS) Accessing remote computing: Hard configured local area clusters Remote supercomputers Need to move data to the computing Resources do not always match problem Standardizing distributed data Web Services, supported on all platforms Custom configure remote data dynamically XML: Extensible Markup Language SOAP: Simple Object Access Protocol WSDL: Web Services Description Language Standardizing distributed computing Grid Services Configure remote computing dynamically Build your own remote computer, and discard Virtual Data: new data sets on demand Standards Accessing Data Today Distributed Services Total area of 3m+ telescopes in the world in m 2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels. The data sets in astrophysics today span the whole electromagnetic spectrum, with a large sky coverage. The total volume of data today is about 100TB, in over 50 collections. New surveys, like GALEX and PRIME will dramatically improve the all-sky, all-wavelength coverage of the Universe. Next generation astronomical instruments have always enabled discoveries beyond their original goals. For example, HSTs new ACS camera sees 3000 galaxies where previously only a few hundred were known. NVO can capitalize on the richness of these new multiwavelength surveys. Building the Framework for the NVO Funded by NSF ITR (2001), $10M over 5 years Collaboration of 20 organizations Astronomy data centers National observatories Supercomputer centers University departments Computer science/IT specialists PI/Project Director: Alex Szalay (JHU) CoPI: Roy Williams (Caltech/CACR) Major milestones associated with simple, intermediate and complex science demos First science demonstrations planned for Jan 2003 External Advisory Committee Executive Committee Working Groups Project Teams Education and Outreach Initiatives Astrophysical Virtual Observatory funded by European Commission (3.3 million, three years) AstroGrid, funded by UK e-science program (£5 million, three years) Canadian Virtual Observatory National initiatives in Germany, Russia, Australia, Japan, China, India, … International VO roadmap developed International VO Alliance has been formed Metadata information about resources - Waveband - Sky coverage - Translation of names to universal dictionary Simple search patterns on the resources - Cone Search - Image mosaic - Unit conversions Simple filtering, counting, histogramming On-the-fly recalibrations Built on Atomic Services Perform more complex tasks - Automated resource discovery - Cross-identifications - Photometric redshifts - Outlier detections - Visualization facilities Expectation: Build custom portals in days from existing building blocks(like in IRAF or IDL) Define commonly used `atomic services Build higher level toolboxes/portals on top We do not build `everything for everybody Use the rule: Define the standards and interfaces Build the framework Build the 10% that are used by 90% Let users build the rest from components The NVO: How Will it Work? Atomic Services Higher Level Services NSF ITR NVO project is one of three major and numerous other small VO-related initiatives now underway world-wideNSF ITR NVO project is one of three major and numerous other small VO-related initiatives now underway world-wide NVO is adopting, adapting, or developing necessary technology as derived from science requirementsNVO is adopting, adapting, or developing necessary technology as derived from science requirements NVO project is dealing with many of the management challenges that will face the ultimate VO organizationNVO project is dealing with many of the management challenges that will face the ultimate VO organization NVO is inevitablethe next logical step in the evolution of the astronomical research environmentNVO is inevitablethe next logical step in the evolution of the astronomical research environment Summary International Collaboration Schedule Organization The NVO Collaboration Sponsored by the National Science Foundations Information Technology Research Program USNO Data Mining Find unusual patterns and features