Design rationale and status of the org.glite.overlay component Elisabetta Ronchieri INFN CNAF JRA1 All Hands CERN, 22-24 Mar 2006

Overview Motivation Proposal Current Implementation Future Directions Related Work JRA1 All Hands, CERN, 22-24 Mar 2006

Motivation Applications want To run a set of smaller tasks in order to be scheduled in a more efficient way To obtain real time response As solution We propose a pull policy approach by using the application overlay network framework We want to provide a toolkit of components that are easily composed and integrated in a wide system characterizing an application JRA1 All Hands, CERN, 22-24 Mar 2006

Proposal: overlay network framework Distributed Computing Model Via overlay network of agents which can communicate each other to satisfy a certain application Built on top of the Grid middleware The overlay network can be configured as a master-slave system It is also possible a master-slaves system Slave is an agent Master is a sort of communication gateway Simple exercise of this proposal is available in the subsystem org.glite.overlay Not fully committed Tested by using the user interface cert-ui-01.cnaf.infn.it, the Workload Management System spank.cnaf.infn.it and each CE/WN reachable by using CEMon and BDII JRA1 All Hands, CERN, 22-24 Mar 2006

Current Implementation [ AgentRetry = 1; MasterHostName = “cert-ui-01.cnaf.infn.it”; Arguments = “spank.cnaf.infn.it 1200”; Requirements = “”; InputSandbox = “glite-overlay-agent”; Executable = “glite-overlay-agent”; ……. ] User submits agent JDL WMS sends job to CE and agent starts running on WN Agent establishes connection with master Master searches user jobs in the TQ selecting job that can be submitted on the WN job is found Master sends job to the agent that runs it on the WN The output files is sent to the user Submit JDL 1. WMS TQ UI 4. 5. 7. Master 2. 3. spank 6. Agent WN JRA1 All Hands, CERN, 22-24 Mar 2006

Current Implementation WN: Agent START Establish a connection with the master Wait for jobs from the master Receive the job Job Found Execute the job Yes No AgentRetry != 0 Yes Send the output back No END JRA1 All Hands, CERN, 22-24 Mar 2006

Current Implementation Advantage: Several user requests can be handled until No jobs are in the task queue No jobs match with the agent requirements and the WN characteristics More efficient execution of a small set of tasks Disadvantage: We need outbound connectivity for the WNs The application can use WN without any time limit with the exception of the maximum wall time associated to the queue where the agent was submitted JRA1 All Hands, CERN, 22-24 Mar 2006

Future Directions The code in the org.glite.overlay subsystem is not the final solution for this problem, it is just an exercise The problem definition needs to be improved This session should help us to better define the problem JRA1 All Hands, CERN, 22-24 Mar 2006

Future Directions Security Aspect Communication System Fault-Tolerance we believe that it is not careful to start the agent without a user identity we prefer to adopt a more cautious approach starting the agent for a specific user Communication System Fault-Tolerance Integration with the current Grid services JRA1 All Hands, CERN, 22-24 Mar 2006

Related Work DIRAC – Distributed Infrastructure with Remote Agent Control See “A practical use case for pilot job based production: DIRAC”, N. Brook, P. Charpentier DIANE – Distributed Analysis Environment It is focused on interfacing semi-interactive parallel applications with distributed GRID technology It provides a master-worker workflow management layer above low-level GRID services Egrid project for Economics and Finance research It is focused on achieving real time response from grid enabled applications It books resources in advance and establishes a direct connection between WN and UI Egrid: It books resources in advance in order to have resources ready when needed. It is required outbound connectivity for the WNs JRA1 All Hands, CERN, 22-24 Mar 2006

Thank you! JRA1 All Hands, CERN, 22-24 Mar 2006