2. Grid Computing - AAU 14/11-051 Grid Computing Josva Kleist Danish Center for Grid Computing www.dcgc.dk

3. The ATLAS experiment

4. Grid Computing - AAU 14/11-053 Agenda E-science Grid Computing An example Grid – NorduGrid ARC Demo Internals of NorduGrid Future

5. E-science ”Science (increasingly) done through distributed global collaborations enabled by the Internet, using very large data collections, tera-scale computing resources and high performance visualisation.”

6. E-science the old fashioned way +

7. Grid Computing - AAU 14/11-056 The grand vision A huge virtual distributed computer.

8. Grid Computing - AAU 14/11-057 Definition 1 “A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities.” The Grid – a blueprint for a new computing infrastructure, 1998

9. Grid Computing - AAU 14/11-058 Definition 2 “The real and specific problem that underlies the Grid concept is coordinated ressource sharing and problem solving in dynamic, multiinstitutional virtual organizations. The sharing that we are concerned with is not primarily file exchange but rather direct access to computers, software, data, and other resources, as is required by a range of collaborative problem solving and resource-brokering strategies emerging in industry, science, and engineering. This sharing is, necessarily, highly controlled, with resource providers and consumers defining clearly and carefully just what is shared, who is allowed to share, and the conditions under which sharing occurs. A set of individuals and/or institutions defined by such sharing rules form what we call a virtual organization.” The anatomy of the Grid, 2000

10. Grid Computing - AAU 14/11-059 Keysentences coordinates resources that are not subject to centralized control … … using standard, open, general-purpose protocols and interfaces … … to deliver nontrivial qualities of service.

11. Challenges Make hardware owned by different organizations available to non-members of that organization. In such a way that normal operation of the equipment can continue. In such a way that the organization still can control who gets access. In such a way that we can control who gets access to specific pieces of data. In such a way that operations can be performed anonymously. And still charge for the use of hard- and software.

12. Grid Computing - AAU 14/11-0511 Challenges Resource allocation and scheduling Authentication and authorization Protection Control Accounting

13. Grid Computing - AAU 14/11-0512 Globus An open source software toolkit used for building grids. Includes software services and libraries for resource monitoring, discovery, and management, plus security and file management. Web: www.globus.org

14. Grid Computing - AAU 14/11-0513 The globus model

15. Grid Computing - AAU 14/11-0514 NorduGrid NorduGrid is a collaboration between a number of universities mostly located in the Nordic contries. NorduGrid Advanced Resource Connector is: A Globus-based Grid middleware solution of choice in Scandinavia and Finland NorduGrid is a production Grid Approximately 5000 CPUs Approximately 75 TB of storage Web: www.nordugrid.org

16. Grid Computing - AAU 14/11-0515 ARC Components

17. Workflow Grid Manager Gatekeeper GridFTP RSL Front-end Cluster Source NorduGrid.org

18. Grid Computing - AAU 14/11-0517 Front-end

19. Grid Computing - AAU 14/11-0518 The user-interface ngsubto submit a task ngstatto obtain the status of jobs and clusters ngcatto display the stdout or stderr of a running job nggetto retrieve the result from a finished job ngkillto cancel a job request ngcleanto delete a job from a remote cluster ngrenewto renew user’s proxy ngsyncto synchronize the local job info with the MDS ngcopyto transfer files to, from and between clusters ngremoveto remove files

20. Grid Computing - AAU 14/11-0519 Broker The user must be authorized to use the cluster and the queue The cluster’s and queue’s characteristics must match the requirements specified in the xRSL string (max CPU time, required free disk space, installed software etc) If the job requires a file that is registered in a Replica Catalog, the brokering gives priority to clusters where a copy of the file is already present From all queues that fulfills the criteria one is chosen randomly, with a weight proportional to the number of free CPUs available for the user in each queue If there are no available CPUs in any of the queues, the job is submitted to the queue with the lowest number of queued job per processor

21. Grid Computing - AAU 14/11-0520 Demo

22. Grid Computing - AAU 14/11-0521 To-do Better resource brokering. Accounting. Scheduling. Security. Monitoring.