Design rationale and status of the org.glite.overlay component

Slides:



Advertisements
Similar presentations
EU 2nd Year Review – Jan – Title – n° 1 WP1 Speaker name (Speaker function and WP ) Presentation address e.g.
Advertisements

Workload management Owen Maroney, Imperial College London (with a little help from David Colling)
Condor-G: A Computation Management Agent for Multi-Institutional Grids James Frey, Todd Tannenbaum, Miron Livny, Ian Foster, Steven Tuecke Reporter: Fu-Jiun.
INFSO-RI Enabling Grids for E-sciencE EGEE Middleware The Resource Broker EGEE project members.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
Workload Management Workpackage Massimo Sgaravatto INFN Padova.
Workload Management Massimo Sgaravatto INFN Padova.
DIRAC API DIRAC Project. Overview  DIRAC API  Why APIs are important?  Why advanced users prefer APIs?  How it is done?  What is local mode what.
Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug
Pilots 2.0: DIRAC pilots for all the skies Federico Stagni, A.McNab, C.Luzzi, A.Tsaregorodtsev On behalf of the DIRAC consortium and the LHCb collaboration.
DIANE Overview Germán Carrera, Alfredo Solano (CNB/CSIC) EMBRACE COURSE Monday 19th of February to Friday 23th. CNB-CSIC Madrid.
Workload Management WP Status and next steps Massimo Sgaravatto INFN Padova.
F.Fanzago – INFN Padova ; S.Lacaprara – LNL; D.Spiga – Universita’ Perugia M.Corvo - CERN; N.DeFilippis - Universita' Bari; A.Fanfani – Universita’ Bologna;
:: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: GridKA School 2009 MPI on Grids 1 MPI On Grids September 3 rd, GridKA School 2009.
Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.
SmartGRID Ongoing research work in Univ. Fribourg and Univ. Applied Sciences of Western Switzerland (HES-SO) SwiNG Grid Day, Bern, Nov. 26th, 2009 Ye HUANG.
Enabling Grids for E-sciencE Workload Management System on gLite middleware Matthieu Reichstadt CNRS/IN2P3 ACGRID School, Hanoi (Vietnam)
- Distributed Analysis (07may02 - USA Grid SW BNL) Distributed Processing Craig E. Tull HCG/NERSC/LBNL (US) ATLAS Grid Software.
Enabling Grids for E-sciencE EGEE-III INFSO-RI Using DIANE for astrophysics applications Ladislav Hluchy, Viet Tran Institute of Informatics Slovak.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks WMSMonitor: a tool to monitor gLite WMS/LB.
 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.
A PanDA Backend for the Ganga Analysis Interface J. Elmsheuser 1, D. Liko 2, T. Maeno 3, P. Nilsson 4, D.C. Vanderster 5, T. Wenaus 3, R. Walker 1 1: Ludwig-Maximilians-Universität.
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
EGEE is a project funded by the European Union under contract IST WS-Based Advance Reservation and Co-allocation Architecture Proposal T.Ferrari,
Transformation System report Luisa Arrabito 1, Federico Stagni 2 1) LUPM CNRS/IN2P3, France 2) CERN 5 th DIRAC User Workshop 27 th – 29 th May 2015, Ferrara.
Tool Integration with Data and Computation Grid “Grid Wizard 2”
DIRAC Pilot Jobs A. Casajus, R. Graciani, A. Tsaregorodtsev for the LHCb DIRAC team Pilot Framework and the DIRAC WMS DIRAC Workload Management System.
Daniele Spiga PerugiaCMS Italia 14 Feb ’07 Napoli1 CRAB status and next evolution Daniele Spiga University & INFN Perugia On behalf of CRAB Team.
DIRAC Project A.Tsaregorodtsev (CPPM) on behalf of the LHCb DIRAC team A Community Grid Solution The DIRAC (Distributed Infrastructure with Remote Agent.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Practical using WMProxy advanced job submission.
D.Spiga, L.Servoli, L.Faina INFN & University of Perugia CRAB WorkFlow : CRAB: CMS Remote Analysis Builder A CMS specific tool written in python and developed.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Mario Reale – GARR NetJobs: Network Monitoring Using Grid Jobs.
Tutorial on Science Gateways, Roma, Catania Science Gateway Framework Motivations, architecture, features Riccardo Rotondo.
Claudio Grandi INFN Bologna Virtual Pools for Interactive Analysis and Software Development through an Integrated Cloud Environment Claudio Grandi (INFN.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
Enabling Grids for E-sciencE Work Load Management & Simple Job Submission Practical Shu-Ting Liao APROC, ASGC EGEE Tutorial.
Enabling Grids for E-sciencE Agreement-based Workload and Resource Management Tiziana Ferrari, Elisabetta Ronchieri Mar 30-31, 2006.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract IST Report from.
DIRAC: Workload Management System Garonne Vincent, Tsaregorodtsev Andrei, Centre de Physique des Particules de Marseille Stockes-rees Ian, University of.
Practical using C++ WMProxy API advanced job submission
WLCG IPv6 deployment strategy
Workload Management Workpackage
Grid Computing: Running your Jobs around the World
L’analisi in LHCb Angelo Carbone INFN Bologna
Duncan MacMichael & Galen Deal CSS 534 – Autumn 2016
How to connect your DG to EDGeS? Zoltán Farkas, MTA SZTAKI
Workload Management System on gLite middleware
Workload Management System ( WMS )
and Alexandre Duarte OurGrid/EELA Interoperability Meeting
Joint JRA1/JRA3/NA4 session
INFN-GRID Workshop Bari, October, 26, 2004
Spark Presentation.
Introduction to Grid Technology
Grid2Win: Porting of gLite middleware to Windows XP platform
LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.
Workload Management System
WP1 activity, achievements and plans
Short update on the latest gLite status
Interoperability & Standards
WLCG Collaboration Workshop;
Time Gathering Systems Secure Data Collection for IBM System i Server
Wide Area Workload Management Work Package DATAGRID project
Overview of Workflows: Why Use Them?
a middleware implementation
I Datagrid Workshop- Marseille C.Vistoli
rvGAHP – Push-Based Job Submission Using Reverse SSH Connections
Servizi di Grid e impatto sulla rete
Grid Computing Software Interface
The LHCb Computing Data Challenge DC06
Presentation transcript:

Design rationale and status of the org.glite.overlay component Elisabetta Ronchieri INFN CNAF JRA1 All Hands CERN, 22-24 Mar 2006

Overview Motivation Proposal Current Implementation Future Directions Related Work JRA1 All Hands, CERN, 22-24 Mar 2006

Motivation Applications want To run a set of smaller tasks in order to be scheduled in a more efficient way To obtain real time response As solution We propose a pull policy approach by using the application overlay network framework We want to provide a toolkit of components that are easily composed and integrated in a wide system characterizing an application JRA1 All Hands, CERN, 22-24 Mar 2006

Proposal: overlay network framework Distributed Computing Model Via overlay network of agents which can communicate each other to satisfy a certain application Built on top of the Grid middleware The overlay network can be configured as a master-slave system It is also possible a master-slaves system Slave is an agent Master is a sort of communication gateway Simple exercise of this proposal is available in the subsystem org.glite.overlay Not fully committed Tested by using the user interface cert-ui-01.cnaf.infn.it, the Workload Management System spank.cnaf.infn.it and each CE/WN reachable by using CEMon and BDII JRA1 All Hands, CERN, 22-24 Mar 2006

Current Implementation [ AgentRetry = 1; MasterHostName = “cert-ui-01.cnaf.infn.it”; Arguments = “spank.cnaf.infn.it 1200”; Requirements = “”; InputSandbox = “glite-overlay-agent”; Executable = “glite-overlay-agent”; ……. ] User submits agent JDL WMS sends job to CE and agent starts running on WN Agent establishes connection with master Master searches user jobs in the TQ selecting job that can be submitted on the WN job is found Master sends job to the agent that runs it on the WN The output files is sent to the user Submit JDL 1. WMS TQ UI 4. 5. 7. Master 2. 3. spank 6. Agent WN JRA1 All Hands, CERN, 22-24 Mar 2006

Current Implementation WN: Agent START Establish a connection with the master Wait for jobs from the master Receive the job Job Found Execute the job Yes No AgentRetry != 0 Yes Send the output back No END JRA1 All Hands, CERN, 22-24 Mar 2006

Current Implementation Advantage: Several user requests can be handled until No jobs are in the task queue No jobs match with the agent requirements and the WN characteristics More efficient execution of a small set of tasks Disadvantage: We need outbound connectivity for the WNs The application can use WN without any time limit with the exception of the maximum wall time associated to the queue where the agent was submitted JRA1 All Hands, CERN, 22-24 Mar 2006

Future Directions The code in the org.glite.overlay subsystem is not the final solution for this problem, it is just an exercise The problem definition needs to be improved This session should help us to better define the problem JRA1 All Hands, CERN, 22-24 Mar 2006

Future Directions Security Aspect Communication System Fault-Tolerance we believe that it is not careful to start the agent without a user identity we prefer to adopt a more cautious approach starting the agent for a specific user Communication System Fault-Tolerance Integration with the current Grid services JRA1 All Hands, CERN, 22-24 Mar 2006

Related Work DIRAC – Distributed Infrastructure with Remote Agent Control See “A practical use case for pilot job based production: DIRAC”, N. Brook, P. Charpentier DIANE – Distributed Analysis Environment It is focused on interfacing semi-interactive parallel applications with distributed GRID technology It provides a master-worker workflow management layer above low-level GRID services Egrid project for Economics and Finance research It is focused on achieving real time response from grid enabled applications It books resources in advance and establishes a direct connection between WN and UI Egrid: It books resources in advance in order to have resources ready when needed. It is required outbound connectivity for the WNs JRA1 All Hands, CERN, 22-24 Mar 2006

Thank you! JRA1 All Hands, CERN, 22-24 Mar 2006