Jens G Jensen Atlas Petabyte store Supporting Multiple Interfaces to Mass Storage Providing Tape and Mass Storage to Diverse Scientific Communities.

Slides:

Advertisements

Similar presentations

What does LOFAR have to do with the Virtual Observatory (VO)? LOFAR Science Day 16 December 2003 Melbourne David Barnes The University of Melbourne.

Advertisements

The Quantum Chromodynamics Grid James Perry, Andrew Jackson, Matthew Egbert, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

J Jensen CCLRC RAL Data Management AUZN (mostly about SRM though) GGF 16, Athens J Jensen.

The Atlas Petabyte Datastore A grid enabled, networked data storage system: CrystalGrid Workshop 15 th Sept 2004 David Corney.

Peter Berrisford RAL – Data Management Group SRB Services.

Cloud Storage in Czech Republic Czech national Cloud Storage and Data Repository project.

FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.

High Performance Computing Course Notes Grid Computing.

Data Grids Darshan R. Kapadia Gregor von Laszewski

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Infrastructure overview Arnold Meijster &

23/04/2008VLVnT08, Toulon, FR, April 2008, M. Stavrianakou, NESTOR-NOA 1 First thoughts for KM3Net on-shore data storage and distribution Facilities VLV.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug

QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.

GridPP9 – 5 February 2004 – Data Management DataGrid is a project funded by the European Union GridPP is funded by PPARC WP2+5: Data and Storage Management.

SICSA student induction day, 2009Slide 1 Social Simulation Tutorial Session 6: Introduction to grids and cloud computing International Symposium on Grid.

GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.

1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong

The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Dataset Caitlin Minteer & Kelly Clynes.

Grids and Portals for VLAB Marlon Pierce Community Grids Lab Indiana University.

Miguel Branco CERN/University of Southampton Enabling provenance on large-scale e-Science applications.

QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

Grid Technologies  Slide text. What is Grid?  The World Wide Web provides seamless access to information that is stored in many millions of different.

UKQCD QCDgrid Richard Kenway. UKQCD Nov 2001QCDgrid2 why build a QCD grid? the computational problem is too big for current computers –configuration generation.

Solar-B Global DataGrid Update for AstroGrid SAG 10 RALMSSL Tim Folkes Elizabeth Auden Jens Jensen Paul Lamb Matthew WildMatthew Whillock Len Culhane Elizabeth.

1 CNAP 22nd March 2004 Summary of Atlas Petabyte Data Store User Group Meeting March 4 th 2004.

Data Grid projects in HENP R. Pordes, Fermilab Many HENP projects are working on the infrastructure for global distributed simulated data production, data.

D C a c h e Michael Ernst Patrick Fuhrmann Tigran Mkrtchyan d C a c h e M. Ernst, P. Fuhrmann, T. Mkrtchyan Chep 2003 Chep2003 UCSD, California.

Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.

Virtual Data Grid Architecture Ewa Deelman, Ian Foster, Carl Kesselman, Miron Livny.

Tier-2  Data Analysis  MC simulation  Import data from Tier-1 and export MC data CMS GRID COMPUTING AT THE SPANISH TIER-1 AND TIER-2 SITES P. Garcia-Abia.

Author - Title- Date - n° 1 Partner Logo EU DataGrid, Work Package 5 The Storage Element.

Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)

Enabling Grids for E-sciencE Introduction Data Management Jan Just Keijser Nikhef Grid Tutorial, November 2008.

The Earth System Grid (ESG) Computer Science and Technologies DOE SciDAC ESG Project Review Argonne National Laboratory, Illinois May 8-9, 2003.

CLRC and the European DataGrid Middleware Information and Monitoring Services The current information service is built on the hierarchical database OpenLDAP.

MTA SZTAKI Hungarian Academy of Sciences Introduction to Grid portals Gergely Sipos

Owen SyngeTitle of TalkSlide 1 Storage Management Owen Synge – Developer, Packager, and first line support to System Administrators. Talks Scope –GridPP.

Tevfik Kosar Computer Sciences Department University of Wisconsin-Madison Managing and Scheduling Data.

…building the next IT revolution From Web to Grid…

T3 analysis Facility V. Bucard, F.Furano, A.Maier, R.Santana, R. Santinelli T3 Analysis Facility The LHCb Computing Model divides collaboration affiliated.

ESFRI & e-Infrastructure Collaborations, EGEE’09 Krzysztof Wrona September 21 st, 2009 European XFEL.

Grid User Interface for ATLAS & LHCb A more recent UK mini production used input data stored on RAL’s tape server, the requirements in JDL and the IC Resource.

Michael Doherty RAL UK e-Science AHM 2-4 September 2003 SRB in Action.

High Energy FermiLab Two physics detectors (5 stories tall each) to understand smallest scale of matter Each experiment has ~500 people doing.

USQCD regional grid Report to ILDG /28/09ILDG14, June 5, US Grid Usage  Growing usage of gauge configurations in ILDG file format.  Fermilab.

INFSO-RI Enabling Grids for E-sciencE Introduction Data Management Ron Trompert SARA Grid Tutorial, September 2007.

Storage and Data Movement at FNAL D. Petravick CHEP 2003.

Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.

AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.

Partnerships in Innovation: Serving a Networked Nation Grid Technologies: Foundations for Preservation Environments Portals for managing user interactions.

Bulk Data Transfer Activities We regard data transfers as “first class citizens,” just like computational jobs. We have transferred ~3 TB of DPOSS data.

Distributed Physics Analysis Past, Present, and Future Kaushik De University of Texas at Arlington (ATLAS & D0 Collaborations) ICHEP’06, Moscow July 29,

1 5/4/05 Fermilab Mass Storage Enstore, dCache and SRM Michael Zalokar Fermilab.

J Jensen/J Gordon RAL Storage Storage at RAL Service Challenge Meeting 27 Jan 2005.

1 Particle Physics Data Grid (PPDG) project Les Cottrell – SLAC Presented at the NGI workshop, Berkeley, 7/21/99.

Data Infrastructure in the TeraGrid Chris Jordan Campus Champions Presentation May 6, 2009.

FESR Consorzio COMETA - Progetto PI2S2 Introduction to Grid Computing Pietro Di Primo INFN – Catania , Catania.

9/20/04Storage Resource Manager, Timur Perelmutov, Jon Bakken, Don Petravick, Fermilab 1 Storage Resource Manager Timur Perelmutov Jon Bakken Don Petravick.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.

2 nd EGEE/OSG Workshop Data Management in Production Grids 2 nd of series of EGEE/OSG workshops – 1 st on security at HPDC 2006 (Paris) Goal: open discussion.

Riccardo Zappi INFN-CNAF SRM Breakout session. February 28, 2012 Ingredients 1. Basic ingredients (Fabric & Conn. level) 2. (Grid) Middleware ingredients.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations.

UK GridPP Tier-1/A Centre at CLRC

Grid Portal Services IeSE (the Integrated e-Science Environment)

LHC Data Analysis using a worldwide computing grid

Outline Review of Quiz #1 Distributed File Systems 4/20/2019 COP5611.

Lee Lueking D0RACE January 17, 2002

The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets A.Chervenak, I.Foster, C.Kesselman, C.Salisbury,

Software Architecture Taxonomy

Presentation transcript:

Jens G Jensen Atlas Petabyte store Supporting Multiple Interfaces to Mass Storage Providing Tape and Mass Storage to Diverse Scientific Communities

Jens G Jensen Atlas Petabyte store Requirements Data Management Locate data Retrieve data

Jens G Jensen Atlas Petabyte store Requirements Analyse data on the Grid Dual access – Grid and non-Grid

Jens G Jensen Atlas Petabyte store Requirements Curation Long term storage and archival

Jens G Jensen Atlas Petabyte store Requirements Very large volumes Very large real-time data rates

Jens G Jensen Atlas Petabyte store Requirements Recover data in emergency = backup

Jens G Jensen Atlas Petabyte store What is Atlas? 1.2 Petabyte capacity Tape STK Powderhorn Interfaces, services

Jens G Jensen Atlas Petabyte store What is Atlas? 1.2 Petabyte capacity Tape STK Powderhorn Interfaces, services New robot coming in

Jens G Jensen Atlas Petabyte store We support – Grid protocols SRB –Data management interface –Metadata, data SRM –Built for very large data volumes –Very high transfer rates via GridFTP

Jens G Jensen Atlas Petabyte store What is the Grid? Distributed computing Distributed collaborations and Virtual Organisations

Jens G Jensen Atlas Petabyte store What is the Grid? Access is brokered job Data is replicated

Jens G Jensen Atlas Petabyte store What is the Grid? Well defined protocols (sort of) File access Information providers Job submission Security

Jens G Jensen Atlas Petabyte store Grid Architecture, SRB Atlas SRB Scientist

Jens G Jensen Atlas Petabyte store Grid Architecture, SRB Atlas Local SRB Scientist Group files into container Slow network Fast network Store container Remote SRB

Jens G Jensen Atlas Petabyte store Grid Architecture, SRM Storage Element (SRM) File Transfer Service Replica Manager Replica Catalogue Application Information Services Components fit together to provide Grid services Atlas

Jens G Jensen Atlas Petabyte store We support – non-Grid protocols Tape Disk cache vtp, rfio, dcap,…

Jens G Jensen Atlas Petabyte store Who are the customers GridPP Tier 1 CCLRC facilities Research councils e-Science projects

Jens G Jensen Atlas Petabyte store Who are the customers Small – a few gigabytes To large – hundreds of Terabytes Different customers drive different areas of service Be all things to all people?

Jens G Jensen Atlas Petabyte store Community User group meetings Helpdesk How to tie the community together?

Jens G Jensen Atlas Petabyte store Conclusions Supporting multiple communities via multiple interfaces –Grid interfaces and non-Grid –Multiple requirements Diversity is good – (up to a point?) –Volume and rates driven by GridPP –Metadata driven by e-Science projects and RCUK