1 Short Course on Grid Computing Jornadas Chilenas de Computación 2010 INFONOR-CHILE 2010 November 15th - 19th, 2010 Antofagasta, Chile Dr. Barry Wilkinson University of North Carolina Charlotte Oct 12, 2010 © Barry Wilkinson Session 1 Using a Grid portal, command line, and job schedulers to run Grid jobs

Workshop Grid platform

Users Each dual Xeon processors (3.4Ghz) Four quad-core Xeon processors (2.93Ghz) 64GB main memory 1.2 TB disk coit- grid04 coit-grid01.uncc.edu – coit-grid05.uncc.edu switch coit- grid05 coit- grid02 coit- grid03 coit- grid01 All user’s home directories (NFS) UNC-Charlotte system used for workshop

Grid computing portal Portal provides single sign-on to all grid resources. 4

PURSe registration portlet (select Registration tab from login page)‏ Used for students to register and get their accounts. 5

Registration activities 6 For this workshop, we have setup workshop accounts and user certificates as an expedient.

1-2.7 Grid information tab Once logged into Grid portal, user will see a number of tabs across top, which enable user to perform many basic tasks. 7

1-2.8 Proxies To use many services, you are required to have a proxy certificate (a proxy). Proxy is an electronic document that enables resources to be accessed on user’s behalf. Very convenient to use credential management service called myProxy to hold proxies Usually, Gridsphere automatically obtains a proxy from the myProxy server for you when you log in. 8

Proxy management tab 9 For security reasons, proxies have limited lifetimes, so may need to renew them

File management tab 10

11 Interactive job submission tab

12 Command line access to a Grid platform

13 You will need an ssh client to make a secure shell connection. For Windows, more than one option: Putty ssh client WinSCP sftp client (also includes restricted console interface and simple text editor) (Linux already has an ssh client.) Command line access to a Grid platform

All files are provided for the workshop but you will need an editor such as nano below to edit the files. Control key commands: - Control-O to save, Control-X to exit 14

Each user needs a certificate issued (signed) by a “certificate authority” (CA). A certificate authority is set up on coit- grid02.uncc.edu. Uses Globus simpleCA package. In our course at UNCC, students learn how to get and store their signed certificates and private key in their account (.globus directory) For this workshop, we will use the myProxy server that more conveniently holds users credentials for the user. User “certificates” 15

Now need to get a proxy certificate to delegate authority. Obtain a proxy from the myProxy server by issuing the command: myproxy-logon -s coit-grid02.uncc.edu You will be prompted for a passphase, which will be the same as your portal password. Can check that you got a proxy with the command: grid-proxy-info which will show the proxy subject and time left. Getting a proxy 16

Submit a simple job to the local machine Using globusrun-ws command: globusrun-ws -submit -F localhost:8440 -s -c /bin/hostname Job above is bin/hostname –c option to specify a binary executable –s option to specific streaming output back to the console: 17

Submit a simple job to remote machine globusrun-ws command: globusrun-ws -submit -F torvalds.uncw.edu -s -c /bin/hostname 18

19 Using a graphical workflow editor to construct and run distributed Grid jobs

GridNexus GridNexus is a graphical workflow editor that allows the user to create/edit/run workflows, developed by University of North Carolina at Wilmington. Based on Ptolemy II which provides GUI and workflow features. Has been extended to provide Grid computing functionality including to create distributed job and web/grid service workflows Freely available for download 20

GridNexus User Interface Main frame on right is palette for building workflows Upper left frame provides library of modules Lower left frame is a thumbnail of entire workflow 21 Select and drag workflow components and interconnect them Run workflow

Sources, sinks, and transformers Sources produce data without needing input Sinks consume data but may have side effects (such as displaying results) All workflows must start with sources and end with sinks. Transformers take inputs and produce outputs 22

Simple Example 1 A link (or relation) is created between the two modules The output of Const is consumed by the JxplDisplay 23

Simple Example 1 Click on the run button ( ) JxplDisplay* evaluates the input and produces a display window to show the results. *JXMP is a Lisp-based XML language designed for Grid Nexus 24

Simple Example 2 Transformers are modules that take input, transform it, and produce new output This example computes the express: (23 + 6) ∙ -2 25

The Prog Prog (short for Program) module allows one to execute separate workflows Prog primitive evaluates each of its inputs in turn It returns the results of the last one evaluated 26

GridExec (GRAM Client) The GridExec module will run an application on a service using GRAM The Factory Contact is the URL of the server If a Simple command is given, GridExec will construct an RSL file to run it 27 /bin/echo Hello world from GridNexus ${GLOBUS_USER_HOME}/stdout ${GLOBUS_USER_HOME}/stderr

28 Hands-on practice sessions on Grid platform

29 Hands-on practice sessions on Grid platform A distributed job will first be executed using the portal and then the same job will be executed on the command line. Then it will be executed using a graphical workflow editor Although at first sight, one would think a portal is best, we shall see how present-day portals can be restrictive.

30 Task – Mulch on a Flowerbed Add 4 inches of mulch. How much mulch? Uses both UNC-C servers and UNC-W cluster. Compute the volume/cost of mulch to cover a flowerbed 4 inches thick.

31 Two programs myIntegral.java Program to compute the area of the flowerbed To be executed on coit-grid01.uncc.edu myMulch.java Program to compute the volume and cost of mulch given the area of the flowerbed computed above. To be executed on torvalds.uncw.edu As an expedient, we will provide the programs.

32 Compute area of flower bed torvaldscoit-grid01 area_ output Compute volume and cost of mulch globus-url-copy mulch_ output copy mulch_ output area_ output Note- this is a pipeline computation – increased performance would only result if multiple jobs could be submitted in succession.

33 Using Portal Interactive job submission window File management window

34 Now an XML job description file will be used, which describes the job (executable, input and outfiles, etc.) More powerful way than simply specifying an executable with –c option in globusrun-ws. Job Description Files Using Command Line

/usr/local/java/bin/java ${GLOBUS_USER_HOME} myIntegral ${GLOBUS_USER_HOME}/area_output ${GLOBUS_USER_HOME}/area_error 1 35 Java program to execute myIntegral job description file

Submit job using: globusrun-ws -submit -F localhost:8440 -f myIntegral.xml 36 Submit a job with a job description document

Transfer the output to torvalds Use globus-url-copy command: globus-url-copy file:///nfs-home/ /area_output gsiftp://torvalds.uncw.edu/home/grid/ / 37

38 myMulch job description /usr/local/java/bin/java ${GLOBUS_USER_HOME} myMulch area_output ${GLOBUS_USER_HOME}/mulch_output ${GLOBUS_USER_HOME}/mulch_error 1 Java program to execute

Next step is run second program, myMulch.java, from coit-grid01 but to run on remote machine torvalds.cis.uncw.edu Command: globusrun-ws -submit -F torvalds.uncw.edu -f myMulch.xml 39 Submit a job to run on a different computer

40 File staging Job description file can be modified to include moving area file to torvalds and moving result back automatically rather than use globus-url- copy, and also add file clean-up.

41 Using GridNexus Workflow of complete computation across two servers (Grid nodes), one at UNC-C and one at UNC-W

42 Questions