OptorSim: A Replica Optimisation Simulator for the EU DataGrid W. H. Bell, D. G. Cameron, R. Carvajal, A. P. Millar, C.Nicholson, K. Stockinger, F. Zini.

Slides:

Advertisements

Similar presentations

EU DataGrid progress Fabrizio Gagliardi EDG Project Leader

Advertisements

An open source approach for grids Bob Jones CERN EU DataGrid Project Deputy Project Leader EU EGEE Designated Technical Director

S.L.LloydATSE e-Science Visit April 2004Slide 1 GridPP – A UK Computing Grid for Particle Physics GridPP 19 UK Universities, CCLRC (RAL & Daresbury) and.

WP2: Data Management Gavin McCance University of Glasgow November 5, 2001.

WP2: Data Management Gavin McCance University of Glasgow.

The Quantum Chromodynamics Grid James Perry, Andrew Jackson, Matthew Egbert, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

Tony Doyle - University of Glasgow GridPP EDG - UK Contributions Architecture Testbed-1 Network Monitoring Certificates & Security Storage Element R-GMA.

Dynamic Grid Optimisation TERENA Conference, Lijmerick 5/6/02 A. P. Millar University of Glasgow.

5-Dec-02D.P.Kelsey, GridPP Security1 GridPP Security UK Security Workshop 5-6 Dec 2002, NeSC David Kelsey CLRC/RAL, UK

Stephen Burke - WP8 Status - 14/2/2002 Partner Logo WP8 Status Stephen Burke, PPARC/RAL.

Particle physics – the computing challenge CERN Large Hadron Collider –2007 –the worlds most powerful particle accelerator –10 petabytes (10 million billion.

WP2 and GridPP UK Simulation W. H. Bell University of Glasgow EDG – WP2.

Tony Doyle GridPP2 Proposal, BT Meeting, Imperial, 23 July 2003.

The National Grid Service and OGSA-DAI Mike Mineter

Andrew McNab - Manchester HEP - 22 April 2002 EU DataGrid Testbed EU DataGrid Software releases Testbed 1 Job Lifecycle Authorisation at your site More.

Data Management Expert Panel - WP2. WP2 Overview.

Presenter Name Facility Name EDG Testbed Status Moving to Testbed Two.

Andrew McNab - Manchester HEP - 2 May 2002 Testbed and Authorisation EU DataGrid Testbed 1 Job Lifecycle Software releases Authorisation at your site Grid/Web.

Andrew McNab - Manchester HEP - 22 April 2002 EU DataGrid Testbed EU DataGrid Software releases Testbed 1 Job Lifecycle Authorisation at your site More.

Author - Title- Date - n° 1 GDMP The European DataGrid Project Team

IoP HEPP 2004 Birmingham, 7/4/04 David Cameron, University of Glasgow 1 Simulation of Replica Optimisation Strategies for Data.

High Performance Computing Course Notes Grid Computing.

MTA SZTAKI Hungarian Academy of Sciences Grid Computing Course Porto, January Introduction to Grid portals Gergely Sipos

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

The CrossGrid project Juha Alatalo Timo Koivusalo.

DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.

Milos Kobliha Alejandro Cimadevilla Luis de Alba Parallel Computing Seminar GROUP 12.

The new The new MONARC Simulation Framework Iosif Legrand  California Institute of Technology.

EU 2nd Year Review – Jan – WP9 WP9 Earth Observation Applications Demonstration Pedro Goncalves :

Report ： Zhen Ming Wu 2008 IEEE 9th Grid Computing Conference.

GridPP9 – 5 February 2004 – Data Management DataGrid is a project funded by the European Union GridPP is funded by PPARC WP2+5: Data and Storage Management.

Grid Data Management A network of computers forming prototype grids currently operate across Britain and the rest of the world, working on the data challenges.

ARGONNE  CHICAGO Ian Foster Discussion Points l Maintaining the right balance between research and development l Maintaining focus vs. accepting broader.

D0 SAM – status and needs Plagarized from: D0 Experiment SAM Project Fermilab Computing Division.

QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.

A Dynamic Data Grid Replication Strategy to Minimize the Data Missed Ming Lei, Susan Vrbsky, Xiaoyan Hong University of Alabama.

GRID IIII D UK Particle Physics GridPP Collaboration meeting - R.P.Middleton (RAL/PPD) 23-25th May Grid Monitoring Services Robin Middleton RAL/PPD24-May-01.

A Grid Computing Use case Datagrid Jean-Marc Pierson.

Your university or experiment logo here Caitriana Nicholson University of Glasgow Dynamic Data Replication in LCG 2008.

The Grid System Design Liu Xiangrui Beijing Institute of Technology.

Instrumentation of the SAM-Grid Gabriele Garzoglio CSC 426 Research Proposal.

1 Grid Related Activities at Caltech Koen Holtman Caltech/CMS PPDG meeting, Argonne July 13-14, 2000.

Ajou University, South Korea GCC 2003 Presentation Dynamic Data Grid Replication Strategy based on Internet Hierarchy Sang Min Park , Jai-Hoon Kim, and.

SouthGrid SouthGrid SouthGrid is a distributed Tier 2 centre, one of four setup in the UK as part of the GridPP project. SouthGrid.

1 WP2: Data Management Gavin McCance RAL Middleware Workshop 24 February 2003.

GridPP Deployment & Operations GridPP has built a Computing Grid of more than 5,000 CPUs, with equipment based at many of the particle physics centres.

Quick Introduction to NorduGrid Oxana Smirnova 4 th Nordic LHC Workshop November 23, 2001, Stockholm.

1 Overview of IEPM-BW - Bandwidth Testing of Bulk Data Transfer Tools Connie Logg & Les Cottrell – SLAC/Stanford University Presented at the Internet 2.

Grid Glasgow Outline LHC Computing at a Glance Glasgow Starting Point LHC Computing Challenge CPU Intensive Applications Timeline ScotGRID.

GridPP Presentation to AstroGrid 13 December 2001 Steve Lloyd Queen Mary University of London.

CLRC and the European DataGrid Middleware Information and Monitoring Services The current information service is built on the hierarchical database OpenLDAP.

Caitriana Nicholson, CHEP 2006, Mumbai Caitriana Nicholson University of Glasgow Grid Data Management: Simulations of LCG 2008.

…building the next IT revolution From Web to Grid…

Grid User Interface for ATLAS & LHCb A more recent UK mini production used input data stored on RAL’s tape server, the requirements in JDL and the IC Resource.

Grid Glasgow Outline LHC Computing at a Glance Glasgow Starting Point LHC Computing Challenge CPU Intensive Applications Timeline ScotGRID.

Networking: Applications and Services Antonia Ghiselli, INFN Stu Loken, LBNL Chairs.

2. WP9 – Earth Observation Applications ESA DataGrid Review Frascati, 10 June Welcome and introduction (15m) 2.WP9 – Earth Observation Applications.

6 march Building the INFN Grid Proposal outline a.ghiselli,l.luminari,m.sgaravatto,c.vistoli INFN Grid meeting, milano.

David Foster LCG Project 12-March-02 Fabric Automation The Challenge of LHC Scale Fabrics LHC Computing Grid Workshop David Foster 12 th March 2002.

WP2: Data Management Gavin McCance University of Glasgow.

Using volunteered resources for data-intensive computing and storage David Anderson Space Sciences Lab UC Berkeley 10 April 2012.

ScotGRID is the Scottish prototype Tier 2 Centre for LHCb and ATLAS computing resources. It uses a novel distributed architecture and cutting-edge technology,

Gavin McCance University of Glasgow GridPP2 Workshop, UCL

GGF OGSA-WG, Data Use Cases Peter Kunszt Middleware Activity, Data Management Cluster EGEE is a project funded by the European.

Distributed Data Access and Resource Management in the D0 SAM System

Grid related projects CERN openlab LCG EDG F.Fluckiger

UK GridPP Tier-1/A Centre at CLRC

Network Requirements Javier Orellana

Wide Area Workload Management Work Package DATAGRID project

Presentation transcript:

OptorSim: A Replica Optimisation Simulator for the EU DataGrid W. H. Bell, D. G. Cameron, R. Carvajal, A. P. Millar, C.Nicholson, K. Stockinger, F. Zini 2 nd Meeting of the EPSRC Pilot Projects NeSC, 30 th January 2003

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects The Project Resource-independent and application-independent services E. g., authentication, authorization, resource location, resource allocation, accounting, optimized data access, data replication, grid monitoring, Grid policies, fault detection Grid Services (Middleware) : Resource-specific implementations of basic services E.g., Transport protocols, name servers, CPU schedulers, site accounting, local policies, directory service, data storage, data access Grid Fabric (Resources) Virtual Organisations High Energy Physics (CERN)Earth Observation (ESA)Biology (CNRS) Data- Intensive Applications Toolkit Application Toolkits Distributed Computing Toolkit Problem Solving Applications Toolkit ……

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects Scheduling Jobs on the Grid Scheduler The Grid Site 1 Site 2 Site 3 User

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects Replica Optimisation Optimise use of computing, storage and network resources. Short term optimisation: - Minimise running time of current job. -Get me the files for my job as quickly as possible. Long term optimisation: - Minimise running time of all jobs. -Make sure files are in the best places for all my future jobs. => Test optimisation algorithms with a grid simulator – OptorSim Various replication strategies possible

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects OptorSim: a Replica Optimisation Simulator Simulate prototype particle physics grid, e.g. EDG, GridPP. Inputs are: - site policies - experiment data files - available resources (CPU, network bandwidth, storage) - file access patterns (sequential, random, unitary random walk, Gaussian random walk) Optimisation algorithms tested are: - no replication - always replicate, delete oldest file - always replicate, delete least valuable - economic model (with and without auctions). EDG Testbed Sites

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects Results for EDG Configuration So far, experiments conducted with EDG sites. Any replication is better than none! Results show economic model is at least 10% faster, for sequential access patterns, without auctions. Economic model tuned for sequential access so this is as expected. Using auction mechanism looks even more promising!

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects Building up Realism: GridPP Previous tests used smaller-than- life-size files and datasets, with no background traffic We can get a more realistic picture of how the Grid will look for GridPP - 6 experiments, 22 sites - predicted available CPUs & storage - realistic file sizes (1GB) and dataset sizes (1TB) - realistic number of jobs (~60 users) - inclusion of background network traffic

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects Background Network in GridPP Available bandwidth (Mbits/sec) per day, averaged over up to 3 months. Measurements of actual available bandwidth between various UK EDG sites. Iperf 1 data gathered from e- science monitoring pages 2 and the GridNM monitoring service run by Yee-Ting Li, UCL 3. ~10 – 90 % of bandwidth available, depending on link Diurnal variation is apparent but usually insignificant

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects Results With Background Traffic Preliminary results show that including background traffic slows the mean time per job by about 10%. Awaiting further results…

Caitriana Nicholson 2nd Meeting of the EPSRC Pilot Projects The Future Run tests with planned GridPP resources and more realistic HEP use cases Get real file access patterns, e.g. SAM Further tuning of algorithms Integrate algorithms into EDG testbed Available for download at data-management/optimisation/optor/ OptorSim DEMO!