Presentation is loading. Please wait.

Presentation is loading. Please wait.

SEE-GRID-SCI Hands-On Session: Using Grid Vladimir Slavnic Institute of Physics, Belgrade Serbia The SEE-GRID-SCI initiative.

Similar presentations


Presentation on theme: "SEE-GRID-SCI Hands-On Session: Using Grid Vladimir Slavnic Institute of Physics, Belgrade Serbia The SEE-GRID-SCI initiative."— Presentation transcript:

1 www.see-grid-sci.eu SEE-GRID-SCI Hands-On Session: Using Grid Vladimir Slavnic Institute of Physics, Belgrade Serbia slavnic@scl.rs The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338 Regional SEE-GRID-SCI Meteo Training Faculty of Electrical Engineering Banja Luka July 2-3, 2009

2 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 2 2 Setting up the user account

3 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 3 3 Setting up the user account User Interface (UI) ‏ Access point to Grid User must have a local account on machine Basic grid operations:  list resources suitable for job  submit jobs  cancel jobs  retrieve job output  show job status  file operations  …

4 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 4 4 Setting up the user account Certificates Grid passport X.509 Subject Name (SN) ‏ /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Vladimir Slavnic Certification Authority (CA) ‏ Obtaining a certificate $ grid-cert-request userkey.pem usercert_request.pem usercert.pem Virtual Organisation (VO) ‏

5 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 5 5 Setting up the user account Setting up.globus Right permissions: – 444 usercert.pem – 400 userkey.pem Checking a certificate $ grid-cert-info $ grid-cert-info -subject (-enddate) (-issuer) ‏ $ openssl verify -CApath \ /etc/grid-security/certificates/ ~/.globus/usercert.pem $ grid-proxy-init -verify

6 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 6 6 Setting up the user account Proxies Standard proxies $ grid-proxy-init $ grid-proxy-info $ grid-proxy-destroy VOMS proxies – Group – Role $ voms-proxy-init -voms $ voms-proxy-init –voms : [Role= ] $ voms-proxy-info (-all) ‏

7 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 7 7 Setting up the user account Proxy renewal MyProxy server $ echo $MYPROXY_SERVER $myproxy-init -s -d -n $myproxy-info -s -d $myproxy-destroy -s -d

8 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 8 8 Information service

9 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 9 9 INFORMATION SERVICE lcg-infosites & lcg-info $lcg-infosites --vo -v -f --is $ lcg-infosites --vo seegrid ce $ lcg-infosites --vo seegrid se $ lcg-infosites --vo seegrid closeSE $ lcg-info –list-attrs $ lcg-info --list-se --query 'SE=se.csk.kg.ac.yu' - -attrs CloseCE

10 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 10 10 INFORMATION SERVICE ldap ldap search examples # RESOURCE ldapsearch -x -H ldap://ce64.phy.bg.ac.yu:2170/ -b “Mdsvo- name=resource,o=Grid” # sBDII ldapsearch -x -H ldap://ce64.phy.bg.ac.yu:2170 -b mds-voname=AEGIS01-PHY-SCL,o=grid # top-level BDII ldapsearch -x -H ldap://bdii.phy.bg.ac.yu:2170/ -b 'Mds-voname=local,o=Grid'

11 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 11 11 WatG Browser Web based Grid Information System browser Visualization of BDII information Highly responsive tool because it supports partial refreshes and desynchronization of a web page. Developed with Google Web Toolkit (GWT) ‏ open source Java software development framework. '

12 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 12 12 WatG Browser http://watgbrowser.scl.rs:8080/

13 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 13 13 Data management

14 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 14 14 DATA MANAGEMENT Storage element  Classic SE  LCG Disk pool manager  CASTOR Storage Resource Manager (SRM) ‏ LFC Concept lfc and lcg commands

15 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 15 15 DATA MANAGEMENT Grid file LFN – Logical File Name GUID - Grid Unique Identifier SURL – Storage URL

16 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 16 16 DATA MANAGEMENT LFC scheme

17 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 17 17 DATA MANAGEMENT LFC commands echo $LFC_HOST lfc-ls - List file/directory entries in a directory lfc-mkdir - Create directory lfc-ln - Make a symbolic link to a file/directory lfc-ls - List file/directory entries in a directory lfc-chmod- Change access mode of a LFC file/directory lfc-chown - Change owner and group of a LFC file/directory lfc-getacl - Get file/directory access control lists lfc-setacl - Set file/directory access control lists …

18 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 18 18 DATA MANAGEMENT LFC commands examples (1) ‏ Listing the entries of a LFC directory $ lfc-ls /grid/seegrid/vlada new_file result Creating directories in the LFC $ lfc-mkdir /grid/seegrid/vlada/new_dir $ lfc-ls -l /grid/seegrid/vlada drwxrwxr-x 0 165 101 0 Feb 19 00:30 new_dir -rw-rw-r-- 1 165 101 27 Feb 18 01:08 new_file -rw-rw-r-- 1 165 101 83 Feb 15 00:30 result

19 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 19 19 DATA MANAGEMENT LFC commands examples (2) ‏ Creation of symbolic links $ lfc-ln -s /grid/seegrid/vlada/result \ /grid/seegrid/vlada/result_link $ lfc-ls -l /grid/seegrid/vlada/result_link lrwxrwxrwx 1 165 101 0 Feb 19 00:42 /grid/seegrid/vlada/rezult_link -> /grid/seegrid/vlada/rezult Adding metadata information to LFC entries $ lfc-setcomment /grid/seegrid/vlada/result "best result" $ lfc-ls --comment /grid/seegrid/vlada/result /grid/seegrid/vlada/result best result

20 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 20 20 DATA MANAGEMENT lcg_utils-Replica Management echo $LCG_GFAL_INFOSYS lcg-cp Copies a Grid file to a local destination (download) ‏ lcg-cr Copies a file to a SE and registers the file in the catalogue (upload) ‏ lcg-del Deletes one file (either one replica or all replicas) ‏ lcg-rep Copies a file from one SE to another SE and registers it in the catalogue (replicate) ‏ lcg-gt Gets the TURL for a given SURL and transfer protocol

21 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 21 21 DATA MANAGEMENT lcg_utils-File Catalogue Interaction lcg-aa Adds an alias in the catalogue for a given GUID lcg-ra Removes an alias in the catalogue for a given GUID lcg-rf Registers in the catalogue a file residing on an SE lcg-uf Unregisters in the the catalogue a file residing on an SE lcg-la Lists the aliases for a given LFN, GUID or SURL lcg-lr Lists the replicas for a given LFN, GUID or SURL lcg-ls Lists file information for given SURLs or LFNs

22 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 22 22 DATA MANAGEMENT lcg_utils examples(1) ‏ Upload a file $ lcg-cr file:/home/vlada/banja_luka/mpi.jdl guid:830e7a82-f9fb-4c04-aafd-e24a5f25acdb specify lfn: $ lcg-cr -l lfn:/grid/seegrid/vlada/mpi \ file:/home/vlada/banja_luka/mpi.jdl guid:25fdcd16-bc76-4458-9c1a-8c597fb33194 Replicate a file (one replica per SE) $ lcg-rep -d grid15.rcub.bg.ac.yu \ lfn:/grid/seegrid/vlada/mpi

23 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 23 23 DATA MANAGEMENT lcg_utils examples(2) ‏ List replicas $ lcg-lr lfn:/grid/seegrid/vlada/mpi srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/h ome/ae gis/generated/2009-02-18/file8f99889c-c00a- 4b7b- 89d3-d9abf66ba103 srm://se.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/a egis/generated/2009-02-18/file576fbae0-ba6a- 4b21-a205-3d115b2cf7fb List guid $ lcg-lg lfn:/grid/seegrid/vlada/mpi guid:25fdcd16-bc76-4458-9c1a-8c597fb33194

24 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 24 24 DATA MANAGEMENT lcg_utils examples(3) ‏ Listing files and directories $ lcg-ls -l lfn:/grid/seegrid/vlada/mpi -rw-rw-r-- 1 165 101 349 lfn:/grid/seegrid/vlada/mpi Copying files out of the Grid (download) ‏ $ lcg-cp lfn:/grid/seegrid/vlada/mpi \ file:/home/vlada/banja_luka/down/mpi Deleting replicas lcg-del srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/ home/aegis/generated/2009-02- 19/file36d25d0d-ee61-477eb667- 40b3791786a4 $ lcg-del -a lfn:/grid/aegis/vlada/mpi

25 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 25 25 Data Management Web Portal Developed by Belgrade University Computer Centre Usage  Data Management  Logical Files  Development Environments and Portals Description  Enables easy interaction with grid file catalogue using simple Web-based user interface.  Based on SEE-GRID File Management Java API.  Can be integrated with other Web applications.

26 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 26 26 Data Management Web Portal Available features  Directory browsing  Directory management operations: creating new subdirectories, renaming/moving and deleting directories, replicating directory content, reading directory attributes and permissions.  Copying and registering of new files to the catalog as well as file deletion.  Reading file attributes and permissions.  Replica & alias management operations.  Upload and download of files with possible integration of application-defined upload handlers

27 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 27 27 Data Management Web Portal http://grid02.rcub.bg.ac.yu:8080/NewDataManagement/l ogin-REFS.jsp http://grid02.rcub.bg.ac.yu:8080/NewDataManagement/l ogin-REFS.jsp http://grid02.rcub.bg.ac.yu:8080/NewDataManageme nt/login-WRF-ARW.jsp http://grid02.rcub.bg.ac.yu:8080/NewDataManageme nt/login-WRF-ARW.jsp

28 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 28 28 Job operations

29 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 29 29 JDL JDL (Job Description Language) file attribute = expression Executable = "test.sh"; Arguments = "fileA fileB"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"test.sh", "fileA", "fileB"}; OutputSandbox = {"std.out", "std.err"}; Environment = {“AEGIS_PATH=$HOME/AEGIS"}; Requirements = RegExp("ce64.phy.bg.ac.rs*",other.GlueCEUniqu eID);

30 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 30 30 SUBMITING JOBS Generating proxy $ voms-proxy-init -voms Myproxy $ myproxy-init -d –n -s Delegate proxy to WMS $ glite-wms-job-delegate-proxy -d Job matching $ glite-wms-job-list-match -a test.jdl Job submission $ glite-wms-job-submit -d -i test.jdl

31 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 31 31 MONITORING JOBS View status of job $ glite-wms-job-status $ glite-wms-job-status -i View logging details $ glite-wms-job-logging-info -v $ glite-wms-job-logging-info -v -i Verbosity level can be from 0 to 3.

32 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 32 32 CANCELING JOBS AND RETRIEVING OUTPUT Cancel a job $ glite-wms-job-cancel $ glite-wms-job-cancel -i Retrieve files from Output Sandbox $ glite-wms-job-output $ glite-wms-job-output -i $ glite-wms-job-output --dir

33 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 33 33 ADVANCED JOB TYPES Job Collection - submission of a set of jobs whose description files are placed in one directory Parametric Jobs - submission of a set of jobs having the same descriptions apart from the values of the parametric attributes. Use when your jobs differs only in argument values or input/output files. DAG jobs (Direct Acyclic Graphs) - submission of a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs. Interactive Jobs MPI Jobs

34 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 34 34 JOB COLLECTION Type = "Collection"; InputSandbox = (“fileA”); Requirements = RegExp("ce64.phy.bg.ac.yu*",other.GlueCEUniqueID); nodes = { [ JobType = "normal"; Executable = "/bin/hostname"; Arguments = "-i"; StdOutput = "out"; InputSandbox = (root.InputSandbox, “fileB”); StdError = "err"; OutputSandbox ={"out","err"}; ], [ JobType = "Normal"; Executable = "/bin/date"; StdOutput = "date"; StdError = "err"; OutputSandbox= {"date"}; ] };

35 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 35 35 JOB COLLECTION submission Collection defined in one file $ glite-wms-job-submit -a -o id collection.jdl Folder contains jdl files $ glite-wms-job-submit -a --collection jdl Collection status view Collection output $ glite-wms-job-output -i id --dir kolekcija $ ls kolekcija ids_nodes.map Node_0 Node_1

36 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 36 36 PARAMETRIC JOBS Type = "Job"; JobType = "Parametric"; Executable = "/bin/echo"; Arguments = "_PARAM_"; Parameters = {red, green, blue}; StdOutput = "colour_PARAM_.txt"; StdError = "error_PARAM_.txt"; OutputSandbox = {"colour_PARAM_.txt","error_PARAM_.txt"};

37 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 37 37 PARAMETRIC JOBS Type = "Job"; JobType = “Parametric"; Executable = "exec"; StdInput = "input_PARAM_.txt"; StdOutput = "output_PARAM_.txt"; Parameters = 10; ParameterStart = 1; ParameterStep = 1; InputSandbox = {" ", " "}; OutputSandbox = {"output_PARAM_.txt"};

38 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 38 38 DAG JOBS [ Type = "dag"; InputSandbox = {"job.sh"}; Nodes = [ nodeA = [ file ="A.jdl" ; ]; nodeB = [ file =“b.jdl" ; ]; nodeC = [ file =“C.jdl" ; ]; nodeD = [ Description = [ JobType = "Normal"; Executable = "job.sh"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {root.InputSandbox[0]}; OutputSandbox = {"std.out","std.err"}; ]; Dependencies = { {nodeA,nodeB},{nodeA,nodeC},{{nodeB,nodeC},nodeD} }; ]

39 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 39 39 MPI JOBS JDL file: Type = "Job"; JobType = "MPICH"; NodeNumber = 5; Executable = "mpi.sh"; Arguments = "mpi"; StdOutput = "mpi.out"; StdError = "mpi.err"; InputSandbox = {"mpi.sh","mpi.c"}; OutputSandbox =\ {"mpi.err","mpi.out","mpiexec.out"};

40 Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 40 40 LINKS http://aegis-ca.rcub.bg.ac.yu/ https://edms.cern.ch/file/722398//gLite-3- UserGuide.pdf http://wiki.egee- see.org/index.php/Quick_User_Guide_for_Subm itting_Jobs https://edms.cern.ch/document/590869/1/ http://wiki.egee-see.org/index.php/SEE- GRID_MPI_User_Guide http://wiki.egee-see.org http://wiki.egee- see.org/index.php/Data_Management_Web_Por tal_-_User_Guide


Download ppt "SEE-GRID-SCI Hands-On Session: Using Grid Vladimir Slavnic Institute of Physics, Belgrade Serbia The SEE-GRID-SCI initiative."

Similar presentations


Ads by Google