1. Design rationale and status of the org.glite.overlay component
Elisabetta Ronchieri
INFN CNAF
JRA1 All Hands
CERN, Mar 2006

2. Overview Motivation Proposal Current Implementation Future Directions
Related Work
JRA1 All Hands, CERN, Mar 2006

3. Motivation
Applications want
To run a set of smaller tasks in order to be scheduled in a more efficient way
To obtain real time response
As solution
We propose a pull policy approach by using the application overlay network framework
We want to provide a toolkit of components that are easily composed and integrated in a wide system characterizing an application
JRA1 All Hands, CERN, Mar 2006

4. Proposal: overlay network framework
Distributed Computing Model
Via overlay network of agents which can communicate each other to satisfy a certain application
Built on top of the Grid middleware
The overlay network can be configured as a master-slave system
It is also possible a master-slaves system
Slave is an agent
Master is a sort of communication gateway
Simple exercise of this proposal is available in the subsystem org.glite.overlay
Not fully committed
Tested by using the user interface cert-ui-01.cnaf.infn.it, the Workload Management System spank.cnaf.infn.it and each CE/WN reachable by using CEMon and BDII
JRA1 All Hands, CERN, Mar 2006

5. Current Implementation[
AgentRetry = 1;
MasterHostName = “cert-ui-01.cnaf.infn.it”;
Arguments = “spank.cnaf.infn.it 1200”;
Requirements = “”;
InputSandbox = “glite-overlay-agent”;
Executable = “glite-overlay-agent”;
……. ]
User submits agent JDL
WMS sends job to CE and agent starts running on WN
Agent establishes connection with master
Master searches user jobs in the TQ selecting job that can be submitted on the WN
job is found
Master sends job to the agent that runs it on the WN
The output files is sent to the user
Submit JDL 1.
WMS TQ UI 4. 5. 7.
Master 2. 3.
spank 6.
Agent WN
JRA1 All Hands, CERN, Mar 2006

6. Current Implementation
WN: Agent
START
Establish
a connection
with the master
Wait for jobs
from the master
Receive the job
Job Found
Execute the job
Yes No
AgentRetry != 0
Yes
Send the
output back No
END
JRA1 All Hands, CERN, Mar 2006

7. Current Implementation
Advantage:
Several user requests can be handled until
No jobs are in the task queue
No jobs match with the agent requirements and the WN characteristics
More efficient execution of a small set of tasks
Disadvantage:
We need outbound connectivity for the WNs
The application can use WN without any time limit with the exception of the maximum wall time associated to the queue where the agent was submitted
JRA1 All Hands, CERN, Mar 2006

8. Future Directions
The code in the org.glite.overlay subsystem is not the final solution for this problem, it is just an exercise
The problem definition needs to be improved
This session should help us to better define the problem
JRA1 All Hands, CERN, Mar 2006

9. Future Directions Security Aspect Communication System Fault-Tolerance
we believe that it is not careful to start the agent without a user identity
we prefer to adopt a more cautious approach starting the agent for a specific user
Communication
System Fault-Tolerance
Integration with the current Grid services
JRA1 All Hands, CERN, Mar 2006

10. Related Work
DIRAC – Distributed Infrastructure with Remote Agent Control
See “A practical use case for pilot job based production: DIRAC”, N. Brook, P. Charpentier
DIANE – Distributed Analysis Environment
It is focused on interfacing semi-interactive parallel applications with distributed GRID technology
It provides a master-worker workflow management layer above low-level GRID services
Egrid project for Economics and Finance research
It is focused on achieving real time response from grid enabled applications
It books resources in advance and establishes a direct connection between WN and UI
Egrid:
It books resources in advance in order to have resources ready when needed.
It is required outbound connectivity for the WNs
JRA1 All Hands, CERN, Mar 2006

11. Thank you!
JRA1 All Hands, CERN, Mar 2006