1. EGEE is a project funded by the European Union under contract IST-2003-508833 Distributed Superscheduling Matteo Mordacchini INFN – Padova Università Ca’ Foscari di Venezia JRA1 IT-CZ Meeting, 14-15/12/2004 www.eu-egee.org

2. JRA1 IT-CZ Meeting, 14-15/12/2004 - 2 Distributed Superscheduling Problem When submitting a job, the UI uses a list of known RBs If the first RB fails to satisfy the request, the UI will re- submit the request to the second RB on the list, and so on. Problems Maybe not all the RBs with resources useful for the user are included in the list. If the RBs capable of satisfying the request are not on top of the list, there is a great waste of time and resources before reaching the right site. Some brokers could be overloaded (while other brokers could be under-loaded)

3. JRA1 IT-CZ Meeting, 14-15/12/2004 - 3 Distributed Superscheduling Problem Aim Find a (fast) way to allow RBs to collaborate in order to satisfy a user’s request. Don’t affect the current scheduling methods in the case the first RB already has the resources capable of processing the job. Forwarding of job requests to another WMS must be analysed in detail, but seems feasible

4. JRA1 IT-CZ Meeting, 14-15/12/2004 - 4 Distributed Superscheduling Some Considerations This “collaboration” among brokers should be transparent to the end-user Having *all* resources (CEs) available on the Grid seen by a single broker (or by all brokers) can have scalability problems Solutions in literature Construct a distributed index of the resources and organize them into a overlay network  Different brokers considering different subsets of the existing Grid resources

5. JRA1 IT-CZ Meeting, 14-15/12/2004 - 5 Distributed Superscheduling Indexing Some solutions suggest to organize the resources into a multidimensional space, where every attribute represents a dimension of the space. The space will be divided into zones. Every index entry points to one of this zone and then it is used to access all the resources that lay in that region of space.

6. JRA1 IT-CZ Meeting, 14-15/12/2004 - 6 Distributed Superscheduling Overlay Networks Given the above space subdivision, each region is assigned to a node in the network. Every node is responsible of  giving the information about the resources that belong to its region  routing request that affect other regions, passing it to the neighbour that is the closest to the interested regions. The network is self-organizing: the presence of a resource is communicated to only one node; then it will collaborate with the other nodes in order to find the right region (and associated node) where the resource has to be indexed

7. JRA1 IT-CZ Meeting, 14-15/12/2004 - 7 Distributed Superscheduling - Proposal Use some of the nodes in the Grid to construct an overlay network similar to that described above. Instead of indexing the single resources (CEs), we suggest to index RBs, in order to find the best set of RBs that are (with high probability) able to solve a given user request. RB s

8. JRA1 IT-CZ Meeting, 14-15/12/2004 - 8 Distributed Superscheduling - Proposal In order to index RBs, we need to create a sort of “RB profile”  Considering in some way the content of the ISM The creation of the “RB profile” is done as follows:  The resources of the RB are grouped into sets based on the similarities of the characteristics of the CEs.  A representative of every set is created.  The RB profile is formed by the representatives of the sets. The RB will be indexed in as much regions of the space as the number of different representative he has in its profile

9. JRA1 IT-CZ Meeting, 14-15/12/2004 - 9 Distributed Superscheduling - Proposal Search operations If a RB cannot satisfy a user request, it asks to a node in the overlay network to find a set of RB that can process the given request. Once it has this set, it could ask to the RBs in the set to check if they really can process the request (they really have the proper CEs available); then, he will transfer the request to one of the RBs that have answered positively.

10. JRA1 IT-CZ Meeting, 14-15/12/2004 - 10 Distributed Superscheduling - Proposal Advantages For registration and query operations, every RB needs only to know the address of one node in the overlay network. RB profiles could be updated less frequently than the descriptions of CEs. The process is completely transparent to the user. Disadvantages In order to improve performances, it is better to replicate the nodes in the overlay network. Staging of Input- and Output-Sandbox Communications to the LB.

11. JRA1 IT-CZ Meeting, 14-15/12/2004 - 11 Distributed Superscheduling References S. Ratnasamy, P. Francis, M. Handley, R. Karp, “A Scalable Content Addressable Network”, ACM SIGCOMM’01, Aug. 27-31, 2001 I. Stoica, R. Morris, D. Karger, M. Frans Kaashoek, H. Balakrishnan, “Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications”, ACM SIGCOMM’01, Aug. 27-31, 2001 A. Andrzejak, Z. Xu, “Scalable Efficient Range Queries for Grid Information Services”, 2nd IEEE International Conference on Peer-to-Peer Computing, Sept. 5-7, 2002, Linköping, Sweden P. Ganesan, B. Yang, H. Garcia-Molina, “One Torus to Rule Them All: Multi-dimensional Queries in P2P Systems”, 7th International Workshop on the Web and Databases (WebDB2004), June 17-18, 2004, Paris, France

12. JRA1 IT-CZ Meeting, 14-15/12/2004 - 12 Distributed Superscheduling C. Schmidt, M. Parashar, “Flexible Information Discovery in Decentralized Distributed Systems”, 12th IEEE International Symposium on High Performance Distributed Computing (HPDC'03), June 22 - 24, 2003, Seattle, Washington M. Cai, M. Frank, J. Chen, P. Szekely, “MAAN: A Multi-Attribute Addressable Network for Grid Information Services”, 4th International Workshop on Grid Computing, Phoenix, Arizona, 2003 K. Aberer, P. Cudré-Mauroux, A. Datta, Z. Despotovic, M. Hauswirth, M. Punceva, R. Schmidt, “P-Grid: A Self-organizing Structured P2P System”, ACM SIGMOD Record, Vol. 32, No. 3, September 2003 A. M. Ouksel, G. Moro, “G-Grid: A Class of Scalable and Self- organizing Data Structures for Multi-dimensional Querying and Content Routing in P2P Networks”, Technical Report no. DEIS-LIA- 002-04, February 2004, Univ. of Bologna, Univ. of Illinois at Chicago

13. JRA1 IT-CZ Meeting, 14-15/12/2004 - 13 Distributed Superscheduling E. Tanin, A. Harwood, H. Samet, “Indexing Distributed Complex Data for Complex Queries”, Proceedings of the National Conference on Digital Government Research, pp. 81-90, Seattle, Washington, 2004