Some Grid Science California Institute of Technology Roy Williams Paul Messina Grids and Virtual Observatory Grids and and LIGO.

Slides:

Advertisements

Similar presentations

Trying to Use Databases for Science Jim Gray Microsoft Research

Advertisements

3 September 2004NVO Coordination Meeting1 Grid-Technologies NVO and the Grid Reagan W. Moore George Kremenek Leesa Brieger Ewa Deelman Roy Williams John.

September 13, 2004NVO Summer School1 VO Protocols Overview Tom McGlynn NASA/GSFC T HE US N ATIONAL V IRTUAL O BSERVATORY.

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.

What does LOFAR have to do with the Virtual Observatory (VO)? LOFAR Science Day 16 December 2003 Melbourne David Barnes The University of Melbourne.

The Australian Virtual Observatory e-Science Meeting School of Physics, March 2003 David Barnes.

Abstraction Layers Why do we need them? –Protection against change Where in the hourglass do we put them? –Computer Scientist perspective Expose low-level.

Grid Astronomy with Image Federation Roy Williams Michael Feldmann California Institute of Technology.

A Very Brief Introduction to iRODS

14 October 2003ADASS 2003 – Strasbourg1 Resource Registries for the Virtual Observatory R.Plante (NCSA), G. Greene (STScI), R. Hanisch (STScI), T. McGlynn.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Data Grid Web Services Chip Watson Jie Chen, Ying Chen, Bryan Hess, Walt Akers.

Grid Computing. What is a Grid? Many definitions exist in the literature Early definitions: Foster and Kesselman, 1998 –“A computational grid is a hardware.

A long tradition. e-science, Data Centres, and the Virtual Observatory why is e-science important ? what is the structure of the VO ? what then must we.

C Copyright © 2009, Oracle. All rights reserved. Appendix C: Service-Oriented Architectures.

LIGO-G9900XX-00-M LIGO and GriPhyN Carl Kesselman Ewa Deelman Sachin Chouksey Roy Williams Peter Shawhan Albert Lazzarini Kent Blackburn CaltechUSC/ISI.

The Japanese Virtual Observatory (JVO) Yuji Shirasaki National Astronomical Observatory of Japan.

Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.

SOS EGEE ‘06 GGF Security Auditing Service: Draft Architecture Brian Tierney Dan Gunter Lawrence Berkeley National Laboratory Marty Humphrey University.

OASIS ebXML Registry Standard Open Forum 2003 on Metadata Registries 10:30 – 11:15 January 20, 2003 Kathryn Breininger The Boeing Company Chair, OASIS.

1 Introduction to Grid Computing. 2 What is a Grid? Many definitions exist in the literature Early definitions: Foster and Kesselman, 1998 “A computational.

1 Technologies for distributed systems Andrew Jones School of Computer Science Cardiff University.

Introduction to Apache OODT Yang Li Mar 9, What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.

Virtual Observatory & LIGO Roy Williams California Institute of Technology.

Science with the Virtual Observatory Brian R. Kent NRAO.

ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.

Interoperability through Library APIs Library Technology Services Open House 7/30/15.

Knowledge Extraction from Scientific Data Roy Williams California Institute of Technology SDMIV 24 October 2002 Edinburgh KE ToolsS Data.

SharePoint 2010 Search Architecture The Connector Framework Enhancing the Search User Interface Creating Custom Ranking Models.

Semantic Grid + Data Federation US National Virtual Observatory Roy Williams California Institute of Technology NVO co-director.

1 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre The Data Portal Glen Drinkwater.

DATABASE MANAGEMENT SYSTEMS IN DATA INTENSIVE ENVIRONMENNTS Leon Guzenda Chief Technology Officer.

ILDG Middleware Status Chip Watson ILDG-6 Workshop May 12, 2005.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

Virtual Data Grid Architecture Ewa Deelman, Ian Foster, Carl Kesselman, Miron Livny.

Federation and Fusion of astronomical information Daniel Egret & Françoise Genova, CDS, Strasbourg Standards and tools for the Virtual Observatories.

Federated Discovery and Access in Astronomy Robert Hanisch (NIST), Ray Plante (NCSA)

Part Four: The LSC DataGrid Part Four: LSC DataGrid A: Data Replication B: What is the LSC DataGrid? C: The LSCDataFind tool.

Kurt Mueller San Diego Supercomputer Center NPACI HotPage Updates.

Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Management of Distributed Data Reagan W. Moore.

Grids - the near future Mark Hayes NIEeS Summer School 2003.

A PPARC funded project Workflow and Job Control in Astrogrid Jeff Lusted Dept Physics and Astronomy University of Leicester.

Slide 1 Archive Computing: Scalable Computing Environments on Very Large Archives Andreas J. Wicenec 13-June-2002.

US LHC OSG Technology Roadmap May 4-5th, 2005 Welcome. Thank you to Deirdre for the arrangements.

LIGO-G E LIGO Scientific Collaboration Data Grid Status Albert Lazzarini Caltech LIGO Laboratory Trillium Steering Committee Meeting 20 May 2004.

German Astrophysical Virtual Observatory Overview and Results So Far W. Voges, G. Lemson, H.-M. Adorf.

NSF Review, 18 Nov 2003 Peter Shawhan (LIGO/Caltech)1 How to Develop a LIGO Search Peter Shawhan (LIGO / Caltech) NSF Review November 18, 2003 LIGO-G E.

XML: A Revolution Unfolds Adam Bosworth General Manager Microsoft Corporation.

December 26, 2015 RHIC/USATLAS Grid Computing Facility Overview Dantong Yu Brookhaven National Lab.

AstroGrid NAM 2001 Andy Lawrence Cambridge NAM 2001 Andy Lawrence Cambridge Belfast Cambridge Edinburgh Jodrell Leicester MSSL.

State of LSC Data Analysis and Software LSC Meeting LIGO Hanford Observatory November 11 th, 2003 Kent Blackburn, Stuart Anderson, Albert Lazzarini LIGO.

LIGO-G W Use of Condor by the LIGO Scientific Collaboration Gregory Mendell, LIGO Hanford Observatory On behalf of the LIGO Scientific Collaboration.

LIGO-G Z1 Using Condor for Large Scale Data Analysis within the LIGO Scientific Collaboration Duncan Brown California Institute of Technology.

Distributed Archives Interoperability Cynthia Y. Cheung NASA Goddard Space Flight Center IAU 2000 Commission 5 Manchester, UK August 12, 2000.

© Roy Williams 2002 The Uphill Battle of Semantic Interoperability Roy Williams California Institute of Technology.

DISCO/UDDI -Ketan Reddy Peddabachi. Agenda Web Service Disco UDDI Disco Vs UDDI.

Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.

1 Open Science Grid: Project Statement & Vision Transform compute and data intensive science through a cross- domain self-managed national distributed.

Preservation Data Services Persistent Archive Research Group Reagan W. Moore October 1, 2003.

Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.

Amy Krause EPCC OGSA-DAI An Overview OGSA-DAI on OMII 2.0 OMII The Open Middleware Infrastructure Institute NeSC,

Maintaining and Searching Metadata Mario Antonioletti, Shannon Hastings, Peter Kunszt, Stephen Langella, Simon Laws, Susan Malaika, Gavin McCance, Alex.

Simulation Production System

GGF OGSA-WG, Data Use Cases Peter Kunszt Middleware Activity, Data Management Cluster EGEE is a project funded by the European.

Cross-matching the sky with database server cluster

Milestone 2 Include the names of the papers

External Sorting The slides for this text are organized into chapters. This lecture covers Chapter 11. Chapter 1: Introduction to Database Systems Chapter.

HAO/SCD: VO, metadata, catalogs, ontologies, querying

CEA Experiences Paul Harrison ESO.

Presentation transcript:

Some Grid Science California Institute of Technology Roy Williams Paul Messina Grids and Virtual Observatory Grids and and LIGO

Virtual Sky: Image Federation Xray (ROSAT) theme Change scale Change theme from Caltech CACR Caltech Astronomy Microsoft Research Optical (DPOSS) Coma cluster Virtual Sky has 140,000,000 tiles 140 Gbyte

VO and Grid Computing Supercomputer center support for Data Pipelines (not just MPI) 12 hr 250 Gb HPSS HP Superdome VS1 Science product HPSS system VS2 Web product Win2000 database Raw data CPU Utilization VirtualSky.org Ingestion Copy raw data and filter (background subtraction) Resampling Write VS1 Build VS2 Write VS2 Today ~ 3 TB 1 TB 140 GB Computation also takes about 12 hrs.

VO and database Join RA Dec mag J mag H mag K glon glat flux I flux Q flux U Infrared catalog (indexed by RA) Radio catalog (indexed by glon) Crossmatch engine SQL select Need same sorting for crossmatch Who does it? Or can we impose a standard? order by glon? order by RA? sort here? Sorting 10^9 objects is expensive

VO Interoperability Directory white/yellow pages –VO Schema repository How to publish, how to define content action> Document schema for –archive content –web service capability –table/image/spectrum How to publish algorithms Plug and play data services –SOAP, UDDI, WSDL, Jini? Semantic web –“Question and answer in the language of the client” –Topic maps

VO and Interoperability Client application (e.g., OASIS) asks for available catalogs then for attributes makes a query then displays the result What is the standard interaction with a catalog service? ClientService Want to be dynamic, be able to add catalogs and the applications still work

VO and International Of course there are many astronomy data collections in the UK, Europe, Japan, etc. We intend to collaborate/coordinate with international efforts

L aser I nterferometric G ravitational wave O bservatory Listening to Collisions of Black Holes and Neutron Stars LIGO

LIGO Pictures

Grid LIGO Architecture Clients eg Web, Script, Agent Clients eg Web, Script, Agent Text request Request Manager Request Manager GriPhyNLDAS Gatekeeper (GRAM) Gatekeeper (GRAM) Science Algorithms Software Collaboratory Parallel Computing GridFTP Local Disk Data HPSS GridFTP Replica Catalog Replica Management Transformation Catalog Virtual Data Catalog Virtual Data Request Data Movement Globus RPC other LDAS Condor jobs Objective: Add security & wide-area data services LDAS is Ligo Data Analysis System

Grid LIGO Logic VD Request  Request Manager –XML, Key-value, script? –Action> Build document schema for this Do we have it already in data caches –if yes, get from caches, use simple transformations –else build LDAS script Move relevant data to LDAS local disk Queue job to LDAS, get jobID Pass updates, then result pointer back to user Perhaps cache result

International GW Grid 50 kbyte/sec GEO

International GW Grid Coincidence in Ligo-Virgo data –Not astrophysics data yet…. –Try seismic, electromagnetic data for now Caltech Virgo Electromagnetic Seismic Coincidence events! Currently using rsync LLO LHO Merging 3 streams to 1 Correlate signals (LDAS) ~12 kbyte/sec each way