SmartGRID Decentralized, dynamic grid scheduling framework on swarm agent-based intelligence Seminar in HUST, Wuhan, China. Oct. 22, 2008 Ye HUANG, Amos BROCCO Grid Group, Dept of Information and Communication Technologies, EIA-FR, Switzerland Pervasive Artificial Intelligence Group, Dept of Informatics, University of Fribourg, Switzerland

2 Outline  Aim  SmartGRID architecture  SmartGRID in depth  MaGate scheduler  Data warehouse interface  Agent-based swarm intelligence  Summary  Future work

3 Aim  Vision of Grid:  Large scale distributed resources  Decentralized  Different policies  Unstable, low reliability  SmartGRID: grid resource management framework  Utilizing different scheduling algorithms  Interoperation emphasized scheduler  Dynamic resource discovery by swarm intelligent algorithm

4 SmartGRID layered architecture  Loosely coupled layered architecture.  Two layers and one internal interface. Smart Resource Management Layer Data Warehouse Interface Smart Signaling Layer

5 SmartGRID Node (SG-Node)  SG-Node: the logical unit of SmartGRID framework  MaGate scheduler, DW interface, Ant nest MaGate Scheduler Nest Info. collector DW Interface

6 MaGate scheduler  MaGate stands for “Magnetic Gateway for scheduler”  Core of Smart Resource Management Layer (SRML)  Targets:  Open platform to different scheduling algorithms  Based on dynamic infrastructure information  Gateway between schedulers (MaGate & others, etc. PBS, MSS)  Allow heterogenous and dynamic Grid scheduling  Manage community of resources  Decentralized view of the Grid  Interface to external services  To deal with corresponding issues, e.g. network behavior analyzing  MaGate focuses on:  Decentralization & Interoperability

7 MaGate architecture  1. Self-management  2. Access to invoker  3. Community management  4. LRM utilization  5. External components Remote MaGate Local Resource Management Grid applications

8 MaGate behavior MaGate Community  Job executor  Interface to invoker  Router  Interface to external service  Full functional package

9 MaGate current roadmap  Community Component  Interoperation protocol & behavior, negotiation model  DRM Component  SAGA API  Interface Component  Application-Interface (App-I) to POP-C++  External Component Multi- scheduling algorithm adoption mechanism Utilizing agent-based dynamic resource discovery

10 A simple use case

11 MaGate technical timetable for current roadmap  Simulation  Basic on GridSim, integrated with Alea, GSSIM (Done)  DataWarehouse Interface prototype  Communication channel between SRML and SSL (Ongoing)  Candidate standard specification  Scheduler Interoperability best practice, WS-Agreement, JSDL, CSG (Ongoing)  First standalone prototype

12

13

14

15

16 Smart Signaling Layer  Services:  network monitoring  resource discovery  Adaptive, Reliable, Robust  Swarm Intelligence / Ant Colony Algorithms

17 Datawarehouse interface – Loosely coupled communication between layers Actions triggered by updates in the data warehouse – Persistent and cached Grid information storage local information monitored by the scheduler coordinated scheduling information negotiated by scheduler network information gathered by ants service request queries

18 Ant-based swarm intelligence  Swarm intelligence: artificial intelligence inspired by the behavior of swarms (of insects)  Ants:  Lightweight mobile agents traveling across the network  Ants can only access local resources on nodes  Inherently fully distributed algorithms:  Collaboration between individuals through indirect communication (stigmergy, pheromone trails)

19 BlåtAnt Algorithm: introduction – Fully distributed algorithm to construct and maintain a peer-to-peer overlay topology with bounded diameter Keeps the network connected while optimizing communication between nodes: Pure peer-to-peer: no superpeers Balanced link distributions: no large hubs Fault resilient Minimal number of links per node

20 BlåtAnt Algorithm: logic – Ensure that the network diameter d is D ≤ d < 2D - 1 – Create and remove logical links: – Connection Rule: two nodes are connected if their distance (in hops) is greater than 2D – 1 – Disconnection Rule: two adjacent nodes are disconnected if there exist an alternative path between them of length less than D

21 BlåtAnt Algorithm: implementation – Different species of ants with different tasks: Collect and spread information Connect / Disconnect peers Ensure fault resilience

22 BlåtAnt Algorithm: example graph before augmentation graph after augmentation diameter = 4 (< 2D – 1, D=3) diameter = 19

23 BlåtAnt Algorithm: evaluation (1) diameteredge count

24 BlåtAnt Algorithm: evaluation (2) ant populationDynamic scenario (125+ nodes)

25 SSL Roadmap SSL Middleware: – Solenopsis Framework integration BlåtAnt Algorithm: – Current work: Resource discovery algorithms, and proactive monitoring – Future: Large scale simulation and integration with DWI/SRML

26 Summary Smart Resource Management Layer – modular scheduler architecture – decentralized, distributed – open to existing local schedulers and external services Data Warehouse Interface Smart Signaling Layer – ant algorithms to provide network services – underlying peer-to-peer topology constructed using the BlåtAnt algorithm

27 Future work Smart Resource Management Scheduler (SRMS) – First prototype – SG-Node validation Data Warehouse – First development and integration Smart Signaling Layer (SSL) – resource discovery algorithms, round-trip time optimization, pro-active monitoring – Solenopsis 2.0 Middleware

28 Thanks! Questions?