Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t DBES Changes in PD2P replication strategy S. Campana (CERN IT/ES) on.

Slides:

Advertisements

Similar presentations

OptorSim: A Replica Optimisation Simulator for the EU DataGrid W. H. Bell, D. G. Cameron, R. Carvajal, A. P. Millar, C.Nicholson, K. Stockinger, F. Zini.

Advertisements

CERN IT Department CH-1211 Geneva 23 Switzerland t ES 1 Alessandro Di Girolamo Experiment Support 4 July 2011 AMOD weekly report 27 th June.

Ganga in the ATLAS Full Dress Rehearsal Birmingham 4th June 2008 Karl Harrison University of Birmingham - ATLAS Full Dress Rehearsal (FDR) involves loading.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Feasibility Study on a Common Analysis Framework for ATLAS & CMS.

Computer Science Lecture 12, page 1 CS677: Distributed OS Last Class Distributed Snapshots –Termination detection Election algorithms –Bully –Ring.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Simplifying Configuration Ricardo Rocha ( on behalf of the LCGDM.

CERN IT Department CH-1211 Geneva 23 Switzerland t Lorena Lobato Pardavila T1s Coordination Meeting Introduction to GoldenGate administration.

Tier-2 Network Requirements Kors Bos LHC OPN Meeting CERN, October 7-8,

CS 4700 / CS 5700 Network Fundamentals Lecture 17.5: Project 5 Hints (Getting a job at Akamai) Revised 3/31/2014.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES News on monitoring for CMS distributed computing operations Andrea.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES WLCG operations: communication channels Andrea Sciabà WLCG operations.

CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Performant and Future Proof: MySQL, Memcache and Raspberry Pi.

CH2 System models.

Operating Systems & Information Services CERN IT Department CH-1211 Geneva 23 Switzerland t OIS Make the most of Office 2010, Expression.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES P. Saiz (IT-ES) AliEn job agents.

CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services Job Monitoring for the LHC experiments Irina Sidorova (CERN, JINR) on.

Your university or experiment logo here Caitriana Nicholson University of Glasgow Dynamic Data Replication in LCG 2008.

Platform & Engineering Services CERN IT Department CH-1211 Geneva 23 Switzerland t PES Network as a Service Use cases for load balancing.

PanDA Summary Kaushik De Univ. of Texas at Arlington ADC Retreat, Naples Feb 4, 2011.

CERN IT Department CH-1211 Genève 23 Switzerland t Castor development status Alberto Pace LCG-LHCC Referees Meeting, May 5 th, 2008 DRAFT.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Successful Common Projects: Structures and Processes WLCG Management.

CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services Overlook of Messaging.

CERN IT Department CH-1211 Genève 23 Switzerland t Monitoring: Tracking your tasks with Task Monitoring PAT eLearning – Module 11 Edward.

Update on replica management

T3 analysis Facility V. Bucard, F.Furano, A.Maier, R.Santana, R. Santinelli T3 Analysis Facility The LHCb Computing Model divides collaboration affiliated.

CERN – IT Department CH-1211 Genève 23 Switzerland t Working with Large Data Sets Tim Smith CERN/IT Open Access and Research Data Session.

AliEn AliEn at OSC The ALICE distributed computing environment by Bjørn S. Nilsen The Ohio State University.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT DPM / LFC and FTS news Ricardo Rocha ( on behalf of the IT/GT/DMS.

PD2P The DA Perspective Kaushik De Univ. of Texas at Arlington S&C Week, CERN Nov 30, 2010.

CERN IT Department t LHCb Software Distribution Roberto Santinelli CERN IT/GS.

PanDA Status Report Kaushik De Univ. of Texas at Arlington ANSE Meeting, Nashville May 13, 2014.

CERN - IT Department CH-1211 Genève 23 Switzerland t High Availability Databases based on Oracle 10g RAC on Linux WLCG Tier2 Tutorials, CERN,

CERN General Infrastructure Services Department CERN GS Department CH-1211 Geneva 23 Switzerland SMS CERN General Infrastructure.

EGI-InSPIRE EGI-InSPIRE RI DDM solutions for disk space resource optimization Fernando H. Barreiro Megino (CERN-IT Experiment Support)

CERN IT Department CH-1211 Geneva 23 Switzerland t A proposal for improving Job Reliability Monitoring GDB 2 nd April 2008.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Upcoming Features and Roadmap Ricardo Rocha ( on behalf of the.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Andrea Sciabà Hammercloud and Nagios Dan Van Der Ster Nicolò Magini.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Author etc Dashboard Latency monitoring (Update) Alexandre Beche.

ATLAS Distributed Computing perspectives for Run-2 Simone Campana CERN-IT/SDC on behalf of ADC.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES CVMFS deployment status Ian Collier – STFC Stefan Roiser – CERN.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Introduction to Data Management Simone Campana CERN IT/ES.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Disk Space publication Simone Campana Fernando Barreiro Wahid.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES L. Betev, A. Grigoras, C. Grigoras, P. Saiz, S. Schreiner AliEn.

Shifters Jamboree Kaushik De ADC Jamboree, CERN December 4, 2014.

CERN IT Department CH-1211 Genève 23 Switzerland t SL(C) 5 Migration at CERN CHEP 2009, Prague Ulrich SCHWICKERATH Ricardo SILVA CERN, IT-FIO-FS.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES P. Saiz The future of AliEn.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Andrea Sciabà Ideal information system - CMS Andrea Sciabà IS.

CERN IT Department CH-1211 Geneva 23 Switzerland t ES 1 how to profit of the ATLAS HLT farm during the LS1 & after Sergio Ballestrero.

Dynamic Data Placement: the ATLAS model Simone Campana (IT-SDC)

ALICE Grid operations +some specific for T2s US-ALICE Grid operations review 7 March 2014 Latchezar Betev 1.

PanDA & Networking Kaushik De Univ. of Texas at Arlington ANSE Workshop, CalTech May 6, 2013.

Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland t CF SINDES Secure INformation DElivery System CERN IT/CF-ASI.

Meeting with University of Malta| CERN, May 18, 2015 | Predrag Buncic ALICE Computing in Run 2+ P. Buncic 1.

LHCb Computing activities Philippe Charpentier CERN – LHCb On behalf of the LHCb Computing Group.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES The Common Solutions Strategy of the Experiment Support group.

CERN IT Department CH-1211 Geneva 23 Switzerland t OIS Web site lifecycles Problem is that web sites live forever –Out of date sites with.

PD2P Planning Kaushik De Univ. of Texas at Arlington S&C Week, CERN Dec 2, 2010.

PD2P, Caching etc. Kaushik De Univ. of Texas at Arlington ADC Retreat, Naples Feb 4, 2011.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Author etc Alarm framework requirements Andrea Sciabà Tony Wildish.

LHCONE Workshop Richard P Mount February 10, 2014 Concerns from Experiments ATLAS Richard P Mount SLAC National Accelerator Laboratory.

Best 20 jobs jobs sites.

Operating Systems & Information Services CERN IT Department CH-1211 Geneva 23 Switzerland t OIS Discussing possibility of deleting archives.

CERN IT Department CH-1211 Genève 23 Switzerland t EGEE09 Barcelona ATLAS Distributed Data Management Fernando H. Barreiro Megino on behalf.

1 LCG-France 22 November 2010 Tier2s connectivity requirements 22 Novembre 2010 S. Jézéquel (LAPP-ATLAS)

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Author etc News from the CMS computing and offline monitoring.

ATLAS activities in the IT cloud in April 2008

The Data Lifetime model

بسم الله الرحمن الرحيم.

Roadmap for Data Management and Caching

Presentation transcript:

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Changes in PD2P replication strategy S. Campana (CERN IT/ES) on behalf of ADC

CERN IT Department CH-1211 Geneva 23 Switzerland t ES Presentation Title - 2 Current PD2P algorithm For T1s: distributes proportionally with T1 MoU share –This is OK, we do not discuss this today For T2s: same algorithm as job brokering –Send a copy to the place which has higher chance to run a job once the dataset gets there –Weight ~ #CPUS * #Running / # Waiting The current algorithm is optimized for re-brokering –And offloading the T1 Not necessarily it is optimal for an balanced data distribution –The main purpose of PD2P is to replicate popular data for reuse

Experiment Support New PD2P algorithm PD2P will use two different algorithm for –Express replica (for quick reuse) –Long Term replica (aimed to balanced data distribution) Express replica –Same algorithm as today. Quick data delivery (use ClosedSites only) Possibility to run promptly a job on the new replica Long Term Replica(s) –Based on size (disk) of the site –Based on performance in Analysis Functional Tests (last month) Presentation title - 3

Experiment Support Pd2P and Pre-Placement PD2P and Pre-Placement of data are (almost) orthogonal –PD2P replicates only what has been used – with a minimal delay –Pre-Placement replicates everything The delay can be very short but also very long (congestion after reprocessing for example) ADC would prefer to have only one mechanism for Data Replication: PD2P Presentation title - 4

Experiment Support Proposal We leave the situation unchanged for T1s and CERN We keep going with no pre-placement for T2s We increase the number of replicas from PD2P create on first use (and subsequent uses): –One Express Replica at T2s –Two Long Term replicas at T2s PD2P is applied to ADOs, (D)ESDs, NTUP –Both Data and MC –Including what is produced by Group Production Presentation title - 5

Experiment Support Monitoring and Docs The PD2P twiki – as/PandaDynamicDataPlacementhttps://twiki.cern.ch/twiki/bin/viewauth/Atl as/PandaDynamicDataPlacement PD2P replication (tables) from Mikhail – uery?mode=pd2phttp://panda.cern.ch/server/pandamon/q uery?mode=pd2p PD2P logs from Tadashi – uery?mode=mon&hours=240&name=pa nda.mon.prod&type=pd2phttp://panda.cern.ch/server/pandamon/q uery?mode=mon&hours=240&name=pa nda.mon.prod&type=pd2p Plots from Jarka (will be moved soon, check the twiki above) – 2p/pd2p_index.html Presentation title - 6