Sky Surveys and the Virtual Observatory Alex Szalay The Johns Hopkins University

Living in an Exponential World Astronomers have a few hundred TB now –1 pixel (byte) / sq arc second ~ 4TB –Multi-spectral, temporal, … → 1PB They mine it looking for new (kinds of) objects or more of interesting ones (quasars), density variations in 400-D space correlations in 400-D space Data doubles every year Caused by the emergence of generations of inexpensive sensors + computing

Why Is Astronomy Special? Especially attractive for the wide public Community is not very large It has no commercial value – No privacy concerns, freely share results with others – Great for experimenting with algorithms It is real and well documented – High-dimensional (with confidence intervals) – Spatial, temporal Diverse and distributed – Many different instruments from many different places and many different times The questions are interesting There is a lot of it (soon petabytes)

Goal Create the most detailed map of the Northern sky “The Cosmic Genome Project” Two surveys in one Photometric survey in 5 bands Spectroscopic redshift survey Automated data reduction 150 man-years of development High data volume 40 TB of raw data 5 TB processed catalogs Data is public 2.5 Terapixels of images Sloan Digital Sky Survey The University of Chicago Princeton University The Johns Hopkins University The University of Washington New Mexico State University Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg Sloan Foundation, NSF, DOE, NASA The University of Chicago Princeton University The Johns Hopkins University The University of Washington New Mexico State University Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg Sloan Foundation, NSF, DOE, NASA

Continuous data rate of 8 Mbytes/sec Northern Galactic Cap drift scan of 10,000 square degrees 5 broad-band filters exposure time: 55 sec pixel size: 0.4 arcsec astrometry: 60 mas calibration: 2% at r'=19.8 done only in best seeing (20 nights/year) Southern Galactic Cap multiple scans (> 30 times) of the same stripe The Photometric Survey u‘ g' r‘ i ' z’ u‘ g' r‘ i ' z’

Survey Strategy Overlapping 2.5 degree wide stripes Avoiding the Galactic Plane (dust) Multiple exposures on Southern stripes Overlapping 2.5 degree wide stripes Avoiding the Galactic Plane (dust) Multiple exposures on Southern stripes

SDSS Redshift Survey 1 million galaxies 900,000 r’ limited 100,000 red galaxies volume limited to z= ,000 quasars 100,000 stars Two high throughput spectrographs spectral range Å 640 spectra simultaneously R=2000 resolution, 1.3 Å Features Automated reduction of spectra Very high sampling density and completeness Objects in other catalogs also targeted The Spectroscopic Survey

Data Processing Pipelines

Analyzing the SkyServer Prototype in data publishing –350 million web hits in 6 years –930,000 distinct users vs 10,000 astronomers –Delivered 50,000 hours of lectures to high school students –Delivered 100B rows of data –Everything is a power law GalaxyZoo –27 million visual galaxy classifications by the public –Enormous publicity (CNN, Times, W.Post, BBC) –100,000 people participating

Skyserver Sessions Singh et al (2007)

Trends CMB Surveys (pixels) 1990 COBE Boomerang 10, CBI 50, WMAP 1 Million 2008 Planck10 Million Galaxy Redshift Surveys (obj) 1986 CfA LCRS dF SDSS Angular Galaxy Surveys (obj) 1970 Lick 1M 1990 APM 2M 2005 SDSS200M 2008 VISTA 1000M 2012 LSST 3000M Time Domain QUEST SDSS Extension survey Dark Energy Camera PanStarrs SNAP… LSST… Petabytes/year by the end of the decade…

National Virtual Observatory NSF ITR project, “Building the Framework for the National Virtual Observatory” is a collaboration of 17 funded and 3 unfunded organizations –Astronomy data centers –National observatories –Supercomputer centers –University departments –Computer science/information technology specialists Similar projects now in 15 countries world-wide => International Virtual Observatory Alliance

Continuing Growth How long does the data growth continue? High end always linear Exponential comes from technology + economics  rapidly changing generations –like CCD’s replacing plates, and become ever cheaper How many new generations of instruments are left? Are there new growth areas emerging? Software (collaboration) is becoming an instrument –hierarchical data replication –Value added data/ mashups –data cloning

Technology+Sociology+Economics Neither of them is enough –We have technology changing very rapidly –Sensors, Moore's Law –Trend driven by changing generations of technologies Sociology is changing in unpredictable ways –In general, people will use a new technology if it is Offers something entirely new Or substantially cheaper Or substantially simpler Funding is essentially level

Surveys from Arecibo Facility very suitable for observing known sources or with known redshift in the ’local’ universe Perfect match to SDSS galaxies ( ~0.1) Matching sky coverage of SDSS, GALEX and FIRST Enormous benefit from using new detector technologies (focal plane phased array) Possible large scale survey projects: –Virgo Deep –SDSS+GALEX HI survey –Integrated HI in distant clusters (vs z) –Local universe (<10,000 km/s)

Summary Science is aggregating into ever larger projects Collection of data is separating from science analysis Much of recent evolution is through new sensors VO is inevitable, a new way of doing science Present on every physical scale today, not just astronomy (Earth/Oceans, Biology, MS, HEP) Might be the only way to do 'small science' in 2020