Virtual Observatory: A Quick Overview, and Some Lessons Learned S. George Djorgovski Caltech ESIP Workshop, UCSB, July 2009

Astronomy Has Become Very Data-Rich Typical digital sky survey now generates ~ TB, plus a comparable amount of derived data products –PB-scale data sets are on the horizon Astronomy today has ~ PB of archived data, and generates a few TB/day –Both data volumes and data rates grow exponentially, with a doubling time ~ 1.5 years –Even more important is the growth of data complexity For comparison: Human memory ~ a few hundred MB Human Genome < 1 GB 1 TB ~ 2 million books Library of Congress (print only) ~ 30 TB

Exponential Growth in Data Volumes and Complexity Visible + X-ray CrabStar forming complex Radio + IR Understanding of complex phenomena requires complex data! Multi- data fusion leads to a more complete, less biased picture (also: multi-scale, multi-epoch, …) Numerical simulations are also producing many TB’s of very complex “data” Data + Theory = Understanding doubling t ≈ 1.5 yrs TB’s to PB’s of data, sources, param./source

The Archive Archipelago As the data sets kept increasing, a number of archives, data depositories, and digital library services were created All of them are mission-, domain-, or observatory-specific, distinct and independent scientifically, technologically, institutionally, heterogeneous in look-feel, usage, etc. –There was a considerable replication of effort –There was some functional redundancy –There was almost no interoperability –Some standards have been generally adopted (e.g., FITS) All of them were primarily designed for single-object (or single-pointing) queries - and thus inherently unsuitable for the science enabled by the massive and complex data sets The next step was clearly to connect them in a functional manner, and develop interoperability standards, formats, etc.

The Virtual Observatory Concept A complete, dynamical, distributed, open research environment for the new astronomy with massive and complex data sets – Provide and federate content (data, metadata) services, standards, and analysis/compute services – Develop and provide data exploration and discovery tools – Not just the archives! – A part of a broader Cyber-Infrastructure and e-Science movement

Survey Telescope Archive Follow-Up Telescopes Results Target Selection Data Mining From Traditional to Survey to VO Science Highly successful, but inherently limited by the information content of individual sky surveys … What comes next, beyond survey science is the VO science Another Survey/Archive? Data Analysis Results Telescope Traditional: Survey-Based:

Surveys Observatories Missions Survey and Mission Archives Follow-Up Telescopes and Missions Results Data Services Data Mining and Analysis, Target Selection Digital libraries Primary Data Providers VO Secondary Data Providers A Systemic View of the VO-Based Science VO connects the whole system of astronomical research

A Brief History of the VO Concept Early (pre-web!) ideas already in the “Astrophysics Data System” (only the digital library part survives) Concept developed through 1990’s, mainly from large digital sky surveys (DPOSS, SDSS…), discussions at conferences and workshops in the late 1990’s Top recommendation in the “small projects” category in the NAS Decadal Astronomy & Astrophysics survey (the McKee-Taylor report), 2001 The first major VO conference at Caltech in 2000; the NVO White paper National Virtual Observatory Science Definition Team, ESO conferences, Vigorous international efforts, coordinated via International VO Alliance (IVOA)

VO Development and Status NSF-funded framework development project ( ): the U.S. National Virtual Observatory (NVO) Now into a facility regime: Virtual Astro. Obs. (VAO) Joint funding by the NSF and NASA Work largely done in the existing data archives, and thus very data-centric Vigorous international efforts (IVOA) ivoa.net

Scientific Roles and Benefits of a VO Facilitate science with massive data sets (observations and theory/simulations) efficiency amplifier Provide an added value from federated data sets (e.g., multi-wavelength, multi-scale, multi-epoch …) –Discover the knowledge which is present in the data, but can be uncovered only through data fusion Enable and stimulate some qualitatively new science with massive data sets (not just old-but-bigger) Optimize the use of expensive resources (e.g., space missions, large ground-based telescopes, computing …) Provide R&D drivers, application testbeds, and stimulus to the partnering disciplines (CS/IT, statistics …)

VO Represents a New Type of a Scientific Organization for the era of information abundance It is not yet another data center, archive, mission, or a traditional project It does not fit into any of the usual organizational structures –It is inherently distributed, and web-centric –It is fundamentally based on a rapidly developing technology (IT/CS) –It transcends the traditional boundaries between different wavelength regimes, agency domains –It has an unusually broad range of constituents and interfaces –It is inherently multidisciplinary

Broader and Societal Benefits of a VO Professional Empowerment: Scientists and students anywhere with an internet connection would be able to do a first-rate science A broadening of the talent pool in astronomy, democratization of the field Interdisciplinary Exchanges: –The challenges facing the VO are common to most sciences and other fields of the modern human endeavor –Intellectual cross-fertilization, feedback to IT/CS Education and Public Outreach: –Unprecedented opportunities in terms of the content, broad geographical and societal range, at all levels –Astronomy as a magnet for the CS/IT education “Weapons of Mass Instruction”

VO Education and Public Outreach Microsoft’s World Wide Telescope, and Google Sky: use DSS, SDSS, HST data, etc., for easy sky browsing

VO Functionality Today What we did so far: Lots of progress on interoperability, standards, etc. An incipient data grid of astronomy Some useful web services Community training, EPO What we did not do (yet): Significant data exploration and mining tools That is where the science will come from! Thus, little VO-enabled science so far Thus, a slow community buy-in  Development of powerful, usable knowledge discovery tools should be a key priority

An Evolving Sociology We have transitioned from the data poverty regime into an era of exponential data abundance –Most astronomers do not seem too fully realize this –Proprietary periods should be re-thought; there are other modes of data access rights currencies, different scenarios? –Data are cheap, but the expertise is expensive (and creativity is priceless) Telescopes are just the hardware needed to generate the data; and data are just incidental to our real mission, which is knowledge creation –When the data and the exploration tools are on the web, the value of large facilities ownership should be rethought –Computers are (relatively) cheap, but software is expensive — especially if you are not approaching it in a smart way

Information Technology  New Science The information volume grows exponentially Most data will never be seen by humans! The need for data storage, network, database-related technologies, standards, etc. Information complexity is also increasing greatly Most data (and data constructs) cannot be comprehended by humans directly! The need for data mining, KDD, data understanding technologies, hyperdimensional visualization, AI/Machine-assisted discovery … We need to create a new scientific methodology on the basis of applied CS and IT VO is the framework to effect this for astronomy

Some Readings: A quick summary: –“Virtual Observatory: From Concept to Implementation”, Djorgovski, S.G., & Williams, R. 2005, A.S.P. Conf. Ser. 345, 517, available as The original VO White Paper: –“Toward a National Virtual Observatory: Science Goals, Technical Challenges, and Implementation Plan”, in Virtual Observatories of the Future, A.S.P. Conf. Ser. 225, 353, available as The NVO SDT report, from Many other good documents available at (especially the summer school presentations) Technical documents at