Grid Computing - AAU 14/ Grid Computing Josva Kleist Danish Center for Grid Computing

The ATLAS experiment

Grid Computing - AAU 14/ Agenda E-science Grid Computing An example Grid – NorduGrid ARC Demo Internals of NorduGrid Future

E-science ”Science (increasingly) done through distributed global collaborations enabled by the Internet, using very large data collections, tera-scale computing resources and high performance visualisation.”

E-science the old fashioned way +

Grid Computing - AAU 14/ The grand vision A huge virtual distributed computer.

Grid Computing - AAU 14/ Definition 1 “A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities.” The Grid – a blueprint for a new computing infrastructure, 1998

Grid Computing - AAU 14/ Definition 2 “The real and specific problem that underlies the Grid concept is coordinated ressource sharing and problem solving in dynamic, multiinstitutional virtual organizations. The sharing that we are concerned with is not primarily file exchange but rather direct access to computers, software, data, and other resources, as is required by a range of collaborative problem solving and resource-brokering strategies emerging in industry, science, and engineering. This sharing is, necessarily, highly controlled, with resource providers and consumers defining clearly and carefully just what is shared, who is allowed to share, and the conditions under which sharing occurs. A set of individuals and/or institutions defined by such sharing rules form what we call a virtual organization.” The anatomy of the Grid, 2000

Grid Computing - AAU 14/ Keysentences coordinates resources that are not subject to centralized control … … using standard, open, general-purpose protocols and interfaces … … to deliver nontrivial qualities of service.

Challenges Make hardware owned by different organizations available to non-members of that organization. In such a way that normal operation of the equipment can continue. In such a way that the organization still can control who gets access. In such a way that we can control who gets access to specific pieces of data. In such a way that operations can be performed anonymously. And still charge for the use of hard- and software.

Grid Computing - AAU 14/ Challenges Resource allocation and scheduling Authentication and authorization Protection Control Accounting

Grid Computing - AAU 14/ Globus An open source software toolkit used for building grids. Includes software services and libraries for resource monitoring, discovery, and management, plus security and file management. Web:

Grid Computing - AAU 14/ The globus model

Grid Computing - AAU 14/ NorduGrid NorduGrid is a collaboration between a number of universities mostly located in the Nordic contries. NorduGrid Advanced Resource Connector is: A Globus-based Grid middleware solution of choice in Scandinavia and Finland NorduGrid is a production Grid Approximately 5000 CPUs Approximately 75 TB of storage Web:

Grid Computing - AAU 14/ ARC Components

Workflow Grid Manager Gatekeeper GridFTP RSL Front-end Cluster Source NorduGrid.org

Grid Computing - AAU 14/ Front-end

Grid Computing - AAU 14/ The user-interface ngsubto submit a task ngstatto obtain the status of jobs and clusters ngcatto display the stdout or stderr of a running job nggetto retrieve the result from a finished job ngkillto cancel a job request ngcleanto delete a job from a remote cluster ngrenewto renew user’s proxy ngsyncto synchronize the local job info with the MDS ngcopyto transfer files to, from and between clusters ngremoveto remove files

Grid Computing - AAU 14/ Broker The user must be authorized to use the cluster and the queue The cluster’s and queue’s characteristics must match the requirements specified in the xRSL string (max CPU time, required free disk space, installed software etc) If the job requires a file that is registered in a Replica Catalog, the brokering gives priority to clusters where a copy of the file is already present From all queues that fulfills the criteria one is chosen randomly, with a weight proportional to the number of free CPUs available for the user in each queue If there are no available CPUs in any of the queues, the job is submitted to the queue with the lowest number of queued job per processor

Grid Computing - AAU 14/ Demo

Grid Computing - AAU 14/ To-do Better resource brokering. Accounting. Scheduling. Security. Monitoring.