1. SEE Grid 2 F E I T Introduction to Grid systems Faculty of Electrical Engineering and Information Technology 12 February 2008 Skopje, Macedonia

2. SEE Grid 2 2008 Skopje, Macedonia F E I T What is Grid It is a volatile heterogeneous distributed system. It is all about sharing resources. It integrates all the heterogeneous resources that are spread across multiple administrative domains (educational institutions, offices, industries) across the world. It gives transparent/collaborative access to these resources and has distributed Job management system. It gives a feeling of Desktop supercomputing - means you are sitting in front of your desktop but you are connected to supercomputer. Grid grows and shrinks dynamically. There is nothing as such you have a static set of resources which we call as Grid. Cluster is not a Grid. 2

3. SEE Grid 2 2008 Skopje, Macedonia F E I T Virtual Organizations An industrial consortium formed to develop a feasibility study for a next generation supersonic aircraft undertakes a highly accurate multidisciplinary simulation of the entire aircraft. A crisis management teams responds to a chemical spill by using local weather and soil models to estimate the spread of the spill, planning and coordinating evacuation, notifying hospitals and so forth. Thousands of physicists come together to design, create, operate and analyze products by pooling together computing, storage, networking resources to create a Data Grid. 3

4. SEE Grid 2 2008 Skopje, Macedonia F E I T 4 Grid Resources PCs, Desktops, Workstations, Clusters, Software's Cluster of Clusters, Clusters of workstations, Piles of PCs Storage elements Mainframes, Supercomputers Visualization Instruments Collaborative Scientists and end users Applications (we may note here that resources involved in Grid are heterogeneous. For example : different type of processors Pentium, AMD, PowerPC)‏ 4

5. SEE Grid 2 2008 Skopje, Macedonia F E I T Architecture Application Collective Resource Connectivity Fabric Application Transport Internet Link GRID Internet 5

6. SEE Grid 2 2008 Skopje, Macedonia F E I T Fabric Layer Fabric layer: Provides the resources to which shared access is mediated by Grid protocols. Example: computational resources, storage systems, catalogs, network resources, and sensors. Fabric components implement local, resource specific operations. Richer fabric functionality enables more sophisticated sharing operations. Sample resources: computational resources, storage resources, network resources, code repositories, catalogs. 6

7. SEE Grid 2 2008 Skopje, Macedonia F E I T Connectivity Layer Communicating easily and securely. Connectivity layer defines the core communication and authentication protocols required for grid- specific network functions. This enables the exchange of data between fabric layer resources. Support for this layer is drawn from TCP/IP’s IP, TCL and DNS layers. Authentication solutions: single sign on, etc. 7

8. SEE Grid 2 2008 Skopje, Macedonia F E I T Resources Layer Resource layer defines protocols, APIs, and SDKs for secure negotiations, initiation, monitoring control, accounting and payment of sharing operations on individual resources. Two protocols information protocol and management protocol define this layer. Information protocols are used to obtain the information about the structure and state of the resource, ex: configuration, current load and usage policy. Management protocols are used to negotiate access to the shared resource, specifying for example qos, advanced reservation, etc. 8

9. SEE Grid 2 2008 Skopje, Macedonia F E I T Collective Layer Coordinating multiple resources. Contains protocols and services that capture interactions among a collection of resources. It supports a variety of sharing behaviors without placing new requirements on the resources being shared. Sample services: directory services, coallocation, brokering and scheduling services, data replication service, workload management services, collaboratory services. 9

10. SEE Grid 2 2008 Skopje, Macedonia F E I T Applications Layer These are user applications that operate within VO environment. Applications are constructed by calling upon services defined at any layer. Each of the layers are well defined using protocols, provide access to useful services. Well defined APIs also exist to work with these services. A toolkit Globus implements all these layers and supports grid application development. 10

11. SEE Grid 2 2008 Skopje, Macedonia F E I T Grid middleware The Grid relies on advanced software – the middleware - which interfaces between resources and the applications The GRID middleware Finds convenient places for the application to be executed Optimises use of resources Organises efficient access to data Deals with authentication to the different sites that are used Run the job & monitors progress Transfers the result back to the scientist 11

12. SEE Grid 2 2008 Skopje, Macedonia F E I T 12 Why do we need Grid? To integrate and aggregate affordable(PC,Desktop,Printers) and unaffordable (clusters,supercomputers,mainframes,giant telescope etc) resources To provide high throughput To deal with increasing amounts of data To build and harvest collaborative boundaries across various communities in research. To give the user feeling of using a most powerful computer. Make fruitful use of the underlying resources and make it as a commodity Concorde (15 Km) ‏ Balloon (30 Km) ‏ CD stack with 1 year LHC data! (~ 20 Km) ‏ Mt. Blanc (4.8 Km) ‏

13. SEE Grid 2 2008 Skopje, Macedonia F E I T CERN - Large Hadron Collider The biggest scientific instrument in the world starts running 2007 13

14. SEE Grid 2 2008 Skopje, Macedonia F E I T The solution: The Grid … securely share distributed resources (computation, storage, etc) so that users can collaborate within Virtual Organisations (VO)‏ 14

15. SEE Grid 2 2008 Skopje, Macedonia F E I T 15 Difference between Grid and Cluster Highly Available. Good quality of service Highly Volatile, resource join and leave. No guarantee of quality of Service Availability It is High Performance ComputingIt is High Throughput Computing (HTC)‏Computing Only access to the nodes of the Cluster and its attached storage. It gives transparent access to whatever resources connected. Transparent Access Mostly Open Source Software'sMixture of Open Source and ProprietarySoftware's used Centralized scheduling, Fault Tolerance and security. Everything is distributed, concept of Local resource managers and Resource brokers Fault Tolerance Scheduling Security Ranges from 10 to 100kRanges from 10s to millionsNumber of Nodes CentralizedDistributedJob Management System Yes No Number of autonomous systems Single system Image Homogeneous (only one kind of processor)‏ Heterogeneous (Pentium, AMD, PowerPC, Different Operating systems, Multiple owners)‏ SystemClusterGridCharacteristics

16. SEE Grid 2 2008 Skopje, Macedonia F E I T 16 How can we build a Grid? How do we take part in the Grid? Or How do we contribute to the Grid ? Or How do we use the Grid? Grid is a social being. Grid is analogous to Electricity Grid. Now if you ask How do we build Electricity Grid? Answers may be  Connect to an existing Grid and use it (as a user) and contribute (as a resource).  Build a grid using the existing resources using Grid toolkits, Core Middleware's, User Level Middleware's.  Getting connected is being part of the Grid.

17. SEE Grid 2 2008 Skopje, Macedonia F E I T 17 EGEE – Enabling Grids for E-Science EGEE’s Grid middleware: gLite –Introducing the core services

18. SEE Grid 2 2008 Skopje, Macedonia F E I T 18 The EGEE project EGEE –1 April 2004 – 31 March 2006 –71 partners in 27 countries, federated in regional Grids EGEE-II –1 April 2006 – 31 March 2008 –91 partners in 32 countries –13 Federations Objectives –Large-scale, production-quality infrastructure for e-Science –Attracting new resources and users from industry as well as science –Improving and maintaining “gLite” Grid middleware US partners in EGEE-II: Univ. Chicago Univ. South. California Univ. Wisconsin RENCI

19. SEE Grid 2 2008 Skopje, Macedonia F E I T 19 Applications on EGEE Applications from an increasing number of domains –Astrophysics –Computational Chemistry –Earth Sciences –Financial Simulation –Fusion –Geophysics –High Energy Physics –Life Sciences –Multimedia –Material Sciences –…

20. SEE Grid 2 2008 Skopje, Macedonia F E I T 20 EU projects related to EGEE EUGRID

21. SEE Grid 2 2008 Skopje, Macedonia F E I T 21 Sustainability: Beyond EGEE-II Need to prepare for permanent Grid infrastructure –Ensure a reliable and adaptive support for all sciences –Independent of short project funding cycles –Infrastructure managed in collaboration with national grid initiatives

22. SEE Grid 2 2008 Skopje, Macedonia F E I T The middleware 22

23. SEE Grid 2 2008 Skopje, Macedonia F E I T 23 “gLite 3.0” the current middleware Being deployed on EGEE production Grid now Runs on various Linux releases –“Scientific Linux” most common –Ports to other Operating Systems in progress History –During last 2 years, some new services were created in releases of new middleware, up to gLite 1.5, has been in pre-production use –A subset of these is deployed with some of the previous middleware (LCG 2.7)‏  All components already in LCG 2.7.0 plus upgrades this already includes new versions of VOMS, R-GMA and FTS  The Workload Management System (with LB, CE, UI) of gLite 1.5.0

24. SEE Grid 2 2008 Skopje, Macedonia F E I T 24 gLite is the next generation middleware for grid computing. Born from the collaborative efforts from academic and industrial research centers as part of the EGEE Project. The gLite Grid services follow a Service Oriented Architecture –facilitate interoperability among Grid services –allow easier compliance with upcoming standards Architecture is not bound to specific implementations –services are expected to work together –services can be deployed and used independently The gLite service decomposition has been largely influenced by the work performed in the LCG project gLite

25. SEE Grid 2 2008 Skopje, Macedonia F E I T 25 gLite Grid Middleware Services API Access Workload Mgmt Services Computing Element Workload Management Metadata Catalog Data Management Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Application Monitoring Connectivity Accounting Auditing Job Provenance Package Manager CLI

26. SEE Grid 2 2008 Skopje, Macedonia F E I T 26 User Interface (UI) User Interface (UI):The place where users logon to the Grid Computing Element (CE) Computing Element (CE): A batch queue on a site’s computers where the user’s job is executed Storage Element (SE) Storage Element (SE): provides (large-scale) storage for files Resource Broker (RB) Resource Broker (RB): Matches the user requirements with the available resources on the Grid Main components Information System Information System: Characteristics and status of CE and SE (Uses “GLUE schema”)‏

27. SEE Grid 2 2008 Skopje, Macedonia F E I T 27 VOMS Virtual Organization Membership Service –Multiple VOs –Multiple roles in VO  Compatible X509 extensions  Signed by VOMS server –Web admin interface –Supports MyProxy –Resources providers grant access to VOs or roles –Sites map VO members/roles to local auth mechanism (unix users accounts)‏  Allows for local policy

28. SEE Grid 2 2008 Skopje, Macedonia F E I T 28 Global Components Grid – or VO- wide –Security  Virtual Organization Server (VOMS)‏  MyProxy server (Proxy)‏ –Information System –Job handling  Workload Management System (WMS)‏  Logging & Bookkeping (LB)‏ –Data management  File catalog (FiReMan)‏  File Transfer Service (FTS)‏  File Placement Service (FPS)‏

29. SEE Grid 2 2008 Skopje, Macedonia F E I T 29 Local Components Site: –Computing Element (CE)‏  Gateway to local computing resources (cluster de worker nodes)‏ –Worker Nodes (WN)‏ –Storage Element (SE)‏  Gateway to local storage (disk, tape)‏  A gridftp server, and SRM Interface, IO server –User Interfaces (UI)‏  User’s access point to the grid  Client programs using some/all grid services.

30. SEE Grid 2 2008 Skopje, Macedonia F E I T 30 MK-02-ETF Core Services CE SE –new SE has 1TB storage space MON UI

31. SEE Grid 2 2008 Skopje, Macedonia F E I T 31 MK-02-ETF WN fleet 12 WN nodes –PIV HT 3.0 Ghz –1GB RAM declared as 24 nodes

32. SEE Grid 2 2008 Skopje, Macedonia F E I T MK-02-ETF Near Future 32

33. SEE Grid 2 2008 Skopje, Macedonia F E I T MK-02-ETF Far Future?

34. SEE Grid 2 2008 Skopje, Macedonia F E I T Questions ?