Towards a Virtual Observatory Alex Szalay Department of Physics and Astronomy The Johns Hopkins University ADASS 2000.

Slides:



Advertisements
Similar presentations

Advertisements

The Australian Virtual Observatory e-Science Meeting School of Physics, March 2003 David Barnes.
Designing Services for Grid-based Knowledge Discovery A. Congiusta, A. Pugliese, Domenico Talia, P. Trunfio DEIS University of Calabria ITALY
Development of China-VO ZHAO Yongheng NAOC, Beijing Nov
SkewReduce YongChul Kwon Magdalena Balazinska, Bill Howe, Jerome Rolia* University of Washington, *HP Labs Skew-Resistant Parallel Processing of Feature-Extracting.
1 Cyberinfrastructure Framework for 21st Century Science & Engineering (CIF21) NSF-wide Cyberinfrastructure Vision People, Sustainability, Innovation,
C van Ingen, D Agarwal, M Goode, J Gupchup, J Hunt, R Leonardson, M Rodriguez, N Li Berkeley Water Center John Hopkins University Lawrence Berkeley Laboratory.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
Network Management Overview IACT 918 July 2004 Gene Awyzio SITACS University of Wollongong.
A Web service for Distributed Covariance Computation on Astronomy Catalogs Presented by Haimonti Dutta CMSC 691D.
Chapter 1 Introduction 1.1A Brief Overview - Parallel Databases and Grid Databases 1.2Parallel Query Processing: Motivations 1.3Parallel Query Processing:
Data-Intensive Computing in the Science Community Alex Szalay, JHU.
Panel Summary Andrew Hanushevsky Stanford Linear Accelerator Center Stanford University XLDB 23-October-07.
Astro-DISC: Astronomy and cosmology applications of distributed super computing.
UNIVERSITY of MARYLAND GLOBAL LAND COVER FACILITY High Performance Computing in Support of Geospatial Information Discovery and Mining Joseph JaJa Institute.
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc
1 Building National Cyberinfrastructure Alan Blatecky Office of Cyberinfrastructure EPSCoR Meeting May 21,
N Tropy: A Framework for Analyzing Massive Astrophysical Datasets Harnessing the Power of Parallel Grid Resources for Astrophysical Data Analysis Jeffrey.
Ian Fisk and Maria Girone Improvements in the CMS Computing System from Run2 CHEP 2015 Ian Fisk and Maria Girone For CMS Collaboration.
ROOT: A Data Mining Tool from CERN Arun Tripathi and Ravi Kumar 2008 CAS Ratemaking Seminar on Ratemaking 17 March 2008 Cambridge, Massachusetts.
Supported by the National Science Foundation’s Information Technology Research Program under Cooperative Agreement AST with The Johns Hopkins University.
Event Metadata Records as a Testbed for Scalable Data Mining David Malon, Peter van Gemmeren (Argonne National Laboratory) At a data rate of 200 hertz,
Amdahl Numbers as a Metric for Data Intensive Computing Alex Szalay The Johns Hopkins University.
Hopkins Storage Systems Lab, Department of Computer Science A Workload-Driven Unit of Cache Replacement for Mid-Tier Database Caching Xiaodan Wang, Tanu.
DISTRIBUTED DATA FLOW WEB-SERVICES FOR ACCESSING AND PROCESSING OF BIG DATA SETS IN EARTH SCIENCES A.A. Poyda 1, M.N. Zhizhin 1, D.P. Medvedev 2, D.Y.
Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.
National Center for Supercomputing Applications Observational Astronomy NCSA projects radio astronomy: CARMA & SKA optical astronomy: DES & LSST access:
Alex Szalay, Jim Gray Analyzing Large Data Sets in Astronomy.
ARGONNE  CHICAGO Ian Foster Discussion Points l Maintaining the right balance between research and development l Maintaining focus vs. accepting broader.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Future role of DMR in Cyber Infrastructure D. Ceperley NCSA, University of Illinois Urbana-Champaign N.B. All views expressed are my own.
1 The Terabyte Analysis Machine Jim Annis, Gabriele Garzoglio, Jun 2001 Introduction The Cluster Environment The Distance Machine Framework Scales The.
Miguel Branco CERN/University of Southampton Enabling provenance on large-scale e-Science applications.
Science with the Virtual Observatory Brian R. Kent NRAO.
National Virtual Observatory Theory,Computation, and Data Exploration Panel of the AASC Charles Alcock, Tom Prince, Alex Szalay.
Alex Szalay Department of Physics and Astronomy The Johns Hopkins University and the SDSS Project The Sloan Digital Sky Survey.
The X-Tree An Index Structure for High Dimensional Data Stefan Berchtold, Daniel A Keim, Hans Peter Kriegel Institute of Computer Science Munich, Germany.
Frontiers in Massive Data Analysis Chapter 3.  Difficult to include data from multiple sources  Each organization develops a unique way of representing.
10/24/2015OSG at CANS1 Open Science Grid Ruth Pordes Fermilab
LSST: Preparing for the Data Avalanche through Partitioning, Parallelization, and Provenance Kirk Borne (Perot Systems Corporation / NASA GSFC and George.
Indexing and Visualizing Multidimensional Data I. Csabai, M. Trencséni, L. Dobos, G. Herczegh, P. Józsa, N. Purger Eötvös University,Budapest.
Service - Oriented Middleware for Distributed Data Mining on the Grid ,劉妘鑏 Antonio C., Domenico T., and Paolo T. Journal of Parallel and Distributed.
EGEE is a project funded by the European Union under contract IST HEP Use Cases for Grid Computing J. A. Templon Undecided (NIKHEF) Grid Tutorial,
1 Computing Challenges for the Square Kilometre Array Mathai Joseph & Harrick Vin Tata Research Development & Design Centre Pune, India CHEP Mumbai 16.
CCGrid, 2012 Supporting User Defined Subsetting and Aggregation over Parallel NetCDF Datasets Yu Su and Gagan Agrawal Department of Computer Science and.
Ruth Pordes November 2004TeraGrid GIG Site Review1 TeraGrid and Open Science Grid Ruth Pordes, Fermilab representing the Open Science.
6/23/2005 R. GARDNER OSG Baseline Services 1 OSG Baseline Services In my talk I’d like to discuss two questions:  What capabilities are we aiming for.
DDM Kirk. LSST-VAO discussion: Distributed Data Mining (DDM) Kirk Borne George Mason University March 24, 2011.
7. Grid Computing Systems and Resource Management
Infrastructure for Data Warehouses. Basics Of Data Access Data Store Machine Memory Buffer Memory Cache Data Store Buffer Bus Structure.
Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.
CERN IT Department CH-1211 Genève 23 Switzerland t CERN IT Monitoring and Data Analytics Pedro Andrade (IT-GT) Openlab Workshop on Data Analytics.
Large Scale Computations in Astrophysics: Towards a Virtual Observatory Alex Szalay Department of Physics and Astronomy The Johns Hopkins University ACAT2000,
Distributed Archives Interoperability Cynthia Y. Cheung NASA Goddard Space Flight Center IAU 2000 Commission 5 Manchester, UK August 12, 2000.
Tackling I/O Issues 1 David Race 16 March 2010.
Latest Improvements in the PROOF system Bleeding Edge Physics with Bleeding Edge Computing Fons Rademakers, Gerri Ganis, Jan Iwaszkiewicz CERN.
Distributed Physics Analysis Past, Present, and Future Kaushik De University of Texas at Arlington (ATLAS & D0 Collaborations) ICHEP’06, Moscow July 29,
Meeting with University of Malta| CERN, May 18, 2015 | Predrag Buncic ALICE Computing in Run 2+ P. Buncic 1.
Smart Grid Big Data: Automating Analysis of Distribution Systems Steve Pascoe Manager Business Development E&O - NISC.
IT-DSS Alberto Pace2 ? Detecting particles (experiments) Accelerating particle beams Large-scale computing (Analysis) Discovery We are here The mission.
Scientific Computing at Fermilab Lothar Bauerdick, Deputy Head Scientific Computing Division 1 of 7 10k slot tape robots.
Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,
Datamining : Refers to extracting or mining knowledge from large amounts of data Applications : Market Analysis Fraud Detection Customer Retention Production.
Data Warehouse.
National Virtual Observatory
به نام خدا Big Data and a New Look at Communication Networks Babak Khalaj Sharif University of Technology Department of Electrical Engineering.
Ch 4. The Evolution of Analytic Scalability
Google Sky.
Database System Architectures
L. Glimcher, R. Jin, G. Agrawal Presented by: Leo Glimcher
Presentation transcript:

Towards a Virtual Observatory Alex Szalay Department of Physics and Astronomy The Johns Hopkins University ADASS 2000

The Virtual Observatory National/Global distributed in scope across institutions, agencies and countries available to all astronomers and the public Virtual not tied to a single “brick-and-mortar” location supports astronomical “observations” and discoveries via remote access to digital representations of the sky Observatory general purpose access to large areas of the sky at multiple wavelengths supports a wide range of astronomical explorations enables discovery via new computational tools

Why Now ? The past decade has witnessed a thousand-fold increase in computer speed a dramatic decrease in the cost of computing & storage a dramatic increase in access to broadly distributed data large archives at multiple sites and high speed networks significant increases in detector size and performance These form the basis for science of qualitatively different nature

Nature of Astronomical Data Imaging –2D map of the sky at multiple wavelengths Derived catalogs –subsequent processing of images –extracting object parameters (400+ per object) Spectroscopic follow-up –spectra: more detailed object properties –clues to physical state and formation history –lead to distances: 3D maps Numerical simulations All inter-related!

Trends Future dominated by detector improvements Total area of 3m+ telescopes in the world in m 2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels. Moore’s Law growth in CCD capabilities Gigapixel arrays on the horizon Improvements in computing and storage will track growth in data volume Investment in software is critical, and growing

The Age of Mega-Surveys The next generation mega-surveys and archives will change astronomy, due to –top-down design –large sky coverage –sound statistical plans –well controlled systematics The technology to store and access the data is here we are riding Moore’s law Data mining will lead to stunning new discoveries Integrating these archives is for the whole community => Virtual Observatory

Ongoing surveys Large number of new surveys –multi-TB in size, 100 million objects or more –individual archives planned, or under way Multi-wavelength view of the sky –more than 13 wavelength coverage in 5 years Impressive early discoveries –finding exotic objects by unusual colors L,T dwarfs, high-z quasars –finding objects by time variability gravitational microlensing MACHO 2MASS DENIS SDSS GALEX FIRST DPOSS GSC-II COBE MAP NVSS FIRST ROSAT OGLE... MACHO 2MASS DENIS SDSS GALEX FIRST DPOSS GSC-II COBE MAP NVSS FIRST ROSAT OGLE...

The Discovery Process discover significant patterns from the analysis of statistically rich and unbiased image/catalog databases understand complex astrophysical systems via confrontation between data and large numerical simulations Past: observations of small, carefully selected samples of objects in a narrow wavelength band Future: high quality, homogeneous multi-wavelength data on millions of objects, allowing us to The discovery process will rely heavily on advanced visualization and statistical analysis tools

The Necessity of the VO Enormous scientific interest in the survey data The environment to exploit these huge sky surveys does not exist today! –1 Terabyte at 10 Mbyte/s takes 1 day –Hundreds of intensive queries and thousands of casual queries per-day –Data will reside at multiple locations, in many different formats –Existing analysis tools do not scale to Terabyte data sets Acute need in a few years, solution will not just happen

VO- The challenges Size of the archived data 40,000 square degrees is 2 Trillion pixels –One band 4 Terabytes –Multi-wavelength Terabytes –Time dimension10 Petabytes Current techniques inadequate –new archival methods –new analysis tools –new standards Hardware/networking requirements –scalable solutions required Transition to the new astronomy

VO: A New Initiative Priority in the Astronomy and Astrophysics Survey Enable new science not previously possible Maximize impact of large current and future efforts Create the necessary new standards Develop the software tools needed Ensure that the community has network and hardware resources to carry out the science Keep up with evolving technology

New Astronomy- Different! Data “Avalanche” –the flood of Terabytes of data is already happening, whether we like it or not –our present techniques of handling these data do not scale well with data volume Systematic data exploration –will have a central role –statistical analysis of the “typical” objects –automated search for the “rare” events Digital archives of the sky –will be the main access to data –hundreds to thousands of queries per day

Examples: Data Pipelines

Examples: Rare Events Discovery of several new objects by SDSS & 2MASS SDSS T-dwarf (June 1999)

Examples: Reprocessing Gravitational lensing 28,000 foreground galaxies over 2,045,000 background galaxies in test data (McKay etal 1999)

Examples: Galaxy Clustering Shape of fluctuation spectrum –cosmological parameters and initial conditions The new surveys (SDSS) are the first when logN~30 Starts with a query Compute correlation function –All pairwise distances N 2, N log N possible Power spectrum –Optimal: the Karhunen-Loeve transform –Signal-to-noise eigenmodes –N 3 in the number of pixels Likelihood analysis in 30 dimensions: needs to be done many times over

Relation to the HEP Problem Similarities –need to handle large amounts of data –data is located at multiple sites –data should be highly clustered –substantial amounts of custom reprocessing –need for a hierarchical organization of resources –scalable solutions required Differences of Astro from HEP –data migration is in opposite direction –the role of small queries is more important –relations between separate data sets (same sky) –data size currently smaller, we can keep it all on disk

Data Migration Path portal HEP Astro Tier 0 Tier 1 Tier 2 Tier 3

Queries are I/O limited In our applications few fixed access patterns –one cannot build indices for all possible queries –worst case scenario is linear scan of the whole table Increasingly large differences between –Random access controlled by seek time (5-10ms), <1000 random I/O /sec –Sequential I/O dramatic improvements, 100 MB/sec per SCSI channel easy reached 215 MB/sec on a single 2-way Dell server Often much faster to scan than to seek Good layout => more sequential I/O

Fast Query Execution Query plan –given the layout of the database, how can one get the result the fastest possible way –evaluate a cost function, using a heuristic algorithm Indexing –clustered and unclustered database layout is crucial –one-dimensional indices: B-tree, R-tree –higher dimensional indices: Quad-tree, Oc-tree,KD-tree, R+-tree –typically logN access instead of N

Distributed Archives Network speeds are much slower –minimize data transfer –run queries locally I/O will scale almost linearly with nodes –1 GB/sec aggregate I/O engine can be built for <$100K Non-trivial problems in –load balancing –query parallelization –queries across inhomogeneous data sources These problems are not specific to astronomy –commercial solutions are around the corner

User Interface Analysis Engine Master Objectivity RAID Slave Objectivity RAID Slave Objectivity RAID Slave Objectivity RAID Slave SX Engine Objectivity Federation SDSS: Distributed Layout

Astronomy Issues What are the user profiles How will the data be analyzed How to organize searches Geometric indexing Color searches and localization SDSS Architecture Virtual data

User Profiles Power users –sophisticated, with lots of resources –research is centered around the archival data moderate number of very large queries, large output sizes Astronomers –frequent, but casual lookup of objects/regions –archives help their research, but not central large number of small queries, cross-identification requests Wide public –browsing a `Virtual Telescope’ –can have large public appeal –need special packaging very large number of simple requests

Geometric Approach Main problem –fast, indexed searches of Terabytes in N-dim space –searches are not axis-parallel simple B-tree indexing does not work Geometric approach –use the geometric nature of the data –quantize data into containers of `friends’ objects of similar colors close on the sky clustered together on disk –containers represent coarse-grained map of the data multidimensional index-tree (eg KD-tree)

Geometric Indexing “Divide and Conquer” Partitioning 3  N  M3  N  M 3  N  M3  N  M Hierarchical Triangular Mesh Split as k-d tree Stored as r-tree of bounding boxes Using regular indexing techniques AttributesNumber Sky Position 3 Multiband FluxesN = 5+ Other M= 100+ AttributesNumber Sky Position 3 Multiband FluxesN = 5+ Other M= 100+

Computing Virtual Data Analyze large output volumes next to the database –send results only (`Virtual Data’): the system `knows’ how to compute the result (Analysis Engine) Analysis: different CPU to I/O ratio than database –multilayered approach Highly scalable architecture required –distributed configuration – scalable to data grids Multiply redundant network paths between data-nodes and compute-nodes –`Data-wolf’ cluster => Virtual Data Grid

A Data Grid Node Hardware requirements Large distributed database engines –with few Gbyte/s aggregate I/O speed High speed (>10 Gbit/s) backbones –cross-connecting the major archives Scalable computing environment –with hundreds of CPUs for analysis Objectivity RAID Objectivity RAID Objectivity RAID Objectivity RAID Objectivity RAID Objectivity RAID Objectivity RAID Database layer 2 GBytes/sec Compute node Compute layer 200 CPUs Interconnect layer 1 Gbits/sec/node Other nodes 10 Gbits/s

GriPhyN GriPhyN - Grid for Physics Networks Goal: build a prototype Virtual Data Grid Create ~20 Tier 2 centers for the data processing Includes ATLAS, ALICE, CMS, LIGO and SDSS Principals –Paul Avery, Ian Foster, Harvey Newman, Carl Kasselman –collaboration of 16 universities and laboratories Phase 1 funding announced a week ago –NSF ITR Program: $11.9M Total scope about $60M

Clustering of Galaxies Generic features of galaxy clustering: Self organized clustering driven by long range forces These lead to clustering on all scales Clustering hierarchy: distribution of galaxy counts is approximately lognormal Scenarios: ‘top-down’ vs ‘bottom-up’

Clustering of Computers Problem sizes have lognormal distribution –multiplicative process Optimal queuing strategy –run smallest job in queue –median scale set by local resources: largest jobs never finish Always need more computing –‘infall’ to larger clusters nearby –asymptotically long-tailed distribution of compute power Short range forces: supercomputers Long range forces: onset of high speed networking Self-organized clustering of computing resources –the Computational Grid

VO: Conceptual Architecture Data Archives Analysis tools Discovery tools User Gateway

The Flavor/Role of the NVO Highly Distributed and Decentralized Multiple Phases, built on top of another Establish standards, meta-data formats Integrate main catalogs Develop initial querying tools Develop collaboration requirements, establish procedure to import new catalogs Develop distributed analysis environment Develop advanced visualization tools Develop advanced querying tools

NVO Development Functions Software development –query generation/optimization, software agents, user interfaces, discovery tools, visualization tools Standards development –Meta-data, meta-services, streaming formats, object relationships, object attributes,... Infrastructure development –archival storage systems, query engines, compute servers, high speed connections of main centers Train the Next Generation –train scientists equally at home in astronomy and modern computer science, statistics, visualization

The Mission of the VO The Virtual Observatory should  provide seamless integration of the digitally represented multi-wavelength sky  enable efficient simultaneous access to multi- Terabyte to Petabyte databases  develop and maintain tools to find patterns and discoveries contained within the large databases  develop and maintain tools to confront data with sophisticated numerical simulations

Sociological Impact The VO is an entirely new way of doing astronomy Will have a serious sociological impact One can expect a love-hate response from the community Needs to be driven by science goals Technology will constantly evolve Educational impact two-fold –need new skills to use it efficiently –provides enormous new opportunities Astronomy is rather unique in its wide appeal –the VO can reach out to an even wider audience

Conclusions Databases became an essential part of astronomy: most data access will soon be via digital archives Data at separate locations, distributed worldwide, evolving in time: move queries not data (if you can)! Computations in both processing and analysis will be substantial: need to create a `Virtual Data Grid’ Problems similar to HEP, lot of commonalities, but data flow more complex Interoperability of archives is essential: the Virtual Observatory is inevitable