1. Analyzing Large Datasets in Astrophysics Alexander Szalay The Johns Hopkins University Towards an International Virtual Observatory, Garching, 2002 (Living in an exponential world….)

2. Alex Szalay, Garching 20022 Outline Collecting Data Exponential Growth Making Discoveries Publishing Data VO: How will it work? Web Services Atomic vs Composite services Distributed queries with SkyQuery Cross-Matching Algorithm SkyNode Web Services + Portal Statistical Analysis of large data sets

3. Alex Szalay, Garching 20023 The World is Exponential Astrophysical data is growing exponentially Doubling every year (Moore s Law+): both data sizes and number of data sets Computational resources scale the same way Constant $$$ will keep up with the data Main problem is the software component Currently components are not reused Software costs are increasingly larger fraction Aggregate costs are growing exponentially

4. Alex Szalay, Garching 20024 Making Discoveries When and where are discoveries made? Always at the edges and boundaries Going deeper, using more colors …. Metcalfe s law Utility of computer networks grows as the number of possible connections: O(N 2 ) VO: Federation of N archives Possibilities for new discoveries grow as O(N 2 ) Current sky surveys have proven this Very early discoveries from SDSS, 2MASS, DPOSS

5. Alex Szalay, Garching 20025 Publishing Data Roles Authors Publishers Curators Consumers Traditional Scientists Journals Libraries Scientists Emerging Collaborations Project www site Bigger Archives Scientists

6. Alex Szalay, Garching 20026 Changing Roles Exponential growth: Projects last at least 3-5 years Data sent upwards only at the end of the project Data will be never centralized More responsibility on projects Becoming Publishers and Curators Larger fraction of budget spent on software Lot of development duplicated, wasted More standards are needed Easier data interchange, fewer tools More templates are needed Develop less software on your own

7. Alex Szalay, Garching 20027 Emerging New Concepts Standardizing distributed data Web Services, supported on all platforms Custom configure remote data dynamically XML: Extensible Markup Language SOAP: Simple Object Access Protocol WSDL: Web Services Description Language Standardizing distributed computing Grid Services Custom configure remote computing dynamically Build your own remote computer, and discard Virtual Data: new data sets on demand

8. Alex Szalay, Garching 20028 NVO: How Will It Work? Define commonly used `atomic services Build higher level toolboxes/portals on top We do not build `everything for everybody Use the 90-10 rule: Define the standards and interfaces Build the framework Build the 10% of services that are used by 90% Let the users build the rest from the components

9. Alex Szalay, Garching 20029 Atomic Services Metadata information about resources Waveband Sky coverage Translation of names to universal dictionary (UCD) Simple search patterns on the resources Cone Search Image mosaic Unit conversions Simple filtering, counting, histogramming On-the-fly recalibrations

10. Alex Szalay, Garching 200210 Higher Level Services Built on Atomic Services Perform more complex tasks Examples Automated resource discovery Cross-identifications Photometric redshifts Outlier detections Visualization facilities Expectation: Build custom portals in matter of days from existing building blocks (like today in IRAF or IDL)

11. Alex Szalay, Garching 200211 SkyQuery Distributed Query tool using a set of services Feasibility study, built in 6 weeks from scratch Tanu Malik (JHU CS grad student) Tamas Budavari (JHU astro postdoc) Implemented in C# and.NET Won 2 nd prize of Microsoft XML Contest Allows queries like: SELECT o.objId, o.r, o.type, t.objId FROM SDSS:PhotoPrimary o, TWOMASS:PhotoPrimary t WHERE XMATCH(o,t)<3.5 AND AREA(181.3,-0.76,6.5) AND o.type=3 and (o.I - t.m_j)>2

12. Alex Szalay, Garching 200212 Architecture Image cutout SkyNode SDSS SkyNode 2Mass SkyNode First SkyQuery Web Page

13. Alex Szalay, Garching 200213 Cross-id Steps Parse query Get counts Sort by counts Make plan Cross-match Recursively, from small to large Select necessary attributes only Return output Insert cutout image SELECT o.objId, o.r, o.type, t.objId FROM SDSS:PhotoPrimary o, TWOMASS:PhotoPrimary t WHERE XMATCH(o,t)<3.5 AND AREA(181.3,-0.76,6.5) AND (o.i - t.m_j) > 2 AND o.type=3

14. Alex Szalay, Garching 200214 Monte-Carlo Simulation Comparing different algorithms for 3-way xid Transmit all the data Transmit after filtering Recursive cross-match Surveys SDSS 2MASS First Random variables: Sky Area (0..10 sqdeg) Selectivity of each subselect (0..1) Efficiency of join (0.5..2) Selectivity of common select (0..1)

15. Alex Szalay, Garching 200215 SkyNode Metadata functions (SOAP) Info, Tables, Columns, Schema, Functions, Keysearch Query functions (SOAP) Dataset Query(String sqlCmd) Dataset Xmatch(Dataset input, String sqlCmd, float eps) Database MS SQL Server Upload dataset Very fast spatial search engine (HTM-based) crossmatch takes <3 ms/object over 15M in SDSS User defined functions and stored procedures

16. Alex Szalay, Garching 200216 Data Flow SkyNode 1 SkyQuery SkyNode 2 SkyNode 3 query http://www.skyquery.net

17. Alex Szalay, Garching 200217 Optimal Statistics The examples for optimal statistics have poor scaling Correlation functions N 2, likelihood techniques N 3 As data sizes grow at Moore s law, computers can only keep up with at most N logN algorithms What goes? Notion of optimal is in the sense of statistical errors Assumes infinite computational resources Assumes that only source of error is statistical `Cosmic Variance : we can only observe the Universe from one location (finite sample size) Solutions require combination of Statistics and CS New algorithms: not worse than N logN

18. Alex Szalay, Garching 200218 Clever Data Structures Heavy use of tree structures: Up-front cost, but only N logN Large speedup later Tree-codes for correlations (A. Moore et al 2001) Fast, approximate heuristic algorithms No need to be more accurate than cosmic variance Fast CMB analysis by Szapudi etal (2001) N logN instead of N 3 => 1 day instead of 10 million years Take cost of computation into account Controlled level of accuracy Best result in a given time, given our computing resources

19. Alex Szalay, Garching 200219 Angular Clustering with Photo-z w( ) by Peebles and Groth: The first example of publishing and analyzing large data Samples based on rest-frame quantities Strictly volume limited samples Largest angular correlation study to date Very clear detection of Luminosity and color dependence Results consistent with 3D clustering T. Budavari, A. Connolly, I. Csabai, I. Szapudi, A. Szalay, S. Dodelson, J. Frieman, R. Scranton, D. Johnston and the SDSS Collaboration

20. Alex Szalay, Garching 200220 The Samples 343k 254k185k 316k280k 326k185k 127k -20 > M r >-211182k -21 > M r >-23931k 0.1<z<0.3 -20 > M r2.2M -21 > M r >-22662k -22 > M r >-23269k 0.1<z<0.5 -21.4 > M r3.1M 10M 10 stripes: 10M 15M m r <21 : 15M 50M All: 50M 2800 square degrees in 10 stripes, data in custom DB

21. Alex Szalay, Garching 200221 The Stripes 10 stripes over the SDSS area, covering about 2800 square degrees About 20% lost due to bad seeing Masks: seeing, bright stars, etc. Images generated from query by web service

22. Alex Szalay, Garching 200222 The Masks Stripe 11 + masks Masks are derived from the database Search and intersect extended objects with boundaries

23. Alex Szalay, Garching 200223 The Analysis eSpICE : I.Szapudi, S.Colombi and S.Prunet Integrated with the database by T. Budavari Extremely fast processing (N logN) 1 stripe with about 1 million galaxies is processed in 3 mins Usual figure was 10 min for 10,000 galaxies => 70 days Each stripe processed separately for each cut 2D angular correlation function computed w( ): average with rejection of pixels along the scan flat field vector causes mock correlations

24. Alex Szalay, Garching 200224 Angular Correlations I. Luminosity dependence: 3 cuts -20> M > -21 -21> M > -22 -22> M > -23

25. Alex Szalay, Garching 200225 Angular Correlations II. Color Dependence 4 bins by rest-frame SED type

26. Alex Szalay, Garching 200226 Summary Exponential data growth – distributed data Web Services – hierarchical architecture Use the 90-10 rule (maybe 80-20) There are clever ways to federate datasets! Statistical analyses do not follow Moore s law Need to revisit optimal statistics Give interesting new tools into the hands of smart young people … They will quickly turn them into cutting edge science

27. Alex Szalay, Garching 200227 Virtual Observatory Astronomy with an attitude…