Download presentation
Presentation is loading. Please wait.
Published byDarcy Malone Modified over 9 years ago
1
www.see-grid-sci.eu SEE-GRID-SCI Hands-On Session: Using Grid Vladimir Slavnic Institute of Physics, Belgrade Serbia slavnic@scl.rs The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338 Regional SEE-GRID-SCI Meteo Training Faculty of Electrical Engineering Banja Luka July 2-3, 2009
2
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 2 2 Setting up the user account
3
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 3 3 Setting up the user account User Interface (UI) Access point to Grid User must have a local account on machine Basic grid operations: list resources suitable for job submit jobs cancel jobs retrieve job output show job status file operations …
4
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 4 4 Setting up the user account Certificates Grid passport X.509 Subject Name (SN) /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Vladimir Slavnic Certification Authority (CA) Obtaining a certificate $ grid-cert-request userkey.pem usercert_request.pem usercert.pem Virtual Organisation (VO)
5
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 5 5 Setting up the user account Setting up.globus Right permissions: – 444 usercert.pem – 400 userkey.pem Checking a certificate $ grid-cert-info $ grid-cert-info -subject (-enddate) (-issuer) $ openssl verify -CApath \ /etc/grid-security/certificates/ ~/.globus/usercert.pem $ grid-proxy-init -verify
6
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 6 6 Setting up the user account Proxies Standard proxies $ grid-proxy-init $ grid-proxy-info $ grid-proxy-destroy VOMS proxies – Group – Role $ voms-proxy-init -voms $ voms-proxy-init –voms : [Role= ] $ voms-proxy-info (-all)
7
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 7 7 Setting up the user account Proxy renewal MyProxy server $ echo $MYPROXY_SERVER $myproxy-init -s -d -n $myproxy-info -s -d $myproxy-destroy -s -d
8
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 8 8 Information service
9
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 9 9 INFORMATION SERVICE lcg-infosites & lcg-info $lcg-infosites --vo -v -f --is $ lcg-infosites --vo seegrid ce $ lcg-infosites --vo seegrid se $ lcg-infosites --vo seegrid closeSE $ lcg-info –list-attrs $ lcg-info --list-se --query 'SE=se.csk.kg.ac.yu' - -attrs CloseCE
10
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 10 10 INFORMATION SERVICE ldap ldap search examples # RESOURCE ldapsearch -x -H ldap://ce64.phy.bg.ac.yu:2170/ -b “Mdsvo- name=resource,o=Grid” # sBDII ldapsearch -x -H ldap://ce64.phy.bg.ac.yu:2170 -b mds-voname=AEGIS01-PHY-SCL,o=grid # top-level BDII ldapsearch -x -H ldap://bdii.phy.bg.ac.yu:2170/ -b 'Mds-voname=local,o=Grid'
11
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 11 11 WatG Browser Web based Grid Information System browser Visualization of BDII information Highly responsive tool because it supports partial refreshes and desynchronization of a web page. Developed with Google Web Toolkit (GWT) open source Java software development framework. '
12
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 12 12 WatG Browser http://watgbrowser.scl.rs:8080/
13
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 13 13 Data management
14
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 14 14 DATA MANAGEMENT Storage element Classic SE LCG Disk pool manager CASTOR Storage Resource Manager (SRM) LFC Concept lfc and lcg commands
15
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 15 15 DATA MANAGEMENT Grid file LFN – Logical File Name GUID - Grid Unique Identifier SURL – Storage URL
16
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 16 16 DATA MANAGEMENT LFC scheme
17
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 17 17 DATA MANAGEMENT LFC commands echo $LFC_HOST lfc-ls - List file/directory entries in a directory lfc-mkdir - Create directory lfc-ln - Make a symbolic link to a file/directory lfc-ls - List file/directory entries in a directory lfc-chmod- Change access mode of a LFC file/directory lfc-chown - Change owner and group of a LFC file/directory lfc-getacl - Get file/directory access control lists lfc-setacl - Set file/directory access control lists …
18
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 18 18 DATA MANAGEMENT LFC commands examples (1) Listing the entries of a LFC directory $ lfc-ls /grid/seegrid/vlada new_file result Creating directories in the LFC $ lfc-mkdir /grid/seegrid/vlada/new_dir $ lfc-ls -l /grid/seegrid/vlada drwxrwxr-x 0 165 101 0 Feb 19 00:30 new_dir -rw-rw-r-- 1 165 101 27 Feb 18 01:08 new_file -rw-rw-r-- 1 165 101 83 Feb 15 00:30 result
19
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 19 19 DATA MANAGEMENT LFC commands examples (2) Creation of symbolic links $ lfc-ln -s /grid/seegrid/vlada/result \ /grid/seegrid/vlada/result_link $ lfc-ls -l /grid/seegrid/vlada/result_link lrwxrwxrwx 1 165 101 0 Feb 19 00:42 /grid/seegrid/vlada/rezult_link -> /grid/seegrid/vlada/rezult Adding metadata information to LFC entries $ lfc-setcomment /grid/seegrid/vlada/result "best result" $ lfc-ls --comment /grid/seegrid/vlada/result /grid/seegrid/vlada/result best result
20
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 20 20 DATA MANAGEMENT lcg_utils-Replica Management echo $LCG_GFAL_INFOSYS lcg-cp Copies a Grid file to a local destination (download) lcg-cr Copies a file to a SE and registers the file in the catalogue (upload) lcg-del Deletes one file (either one replica or all replicas) lcg-rep Copies a file from one SE to another SE and registers it in the catalogue (replicate) lcg-gt Gets the TURL for a given SURL and transfer protocol
21
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 21 21 DATA MANAGEMENT lcg_utils-File Catalogue Interaction lcg-aa Adds an alias in the catalogue for a given GUID lcg-ra Removes an alias in the catalogue for a given GUID lcg-rf Registers in the catalogue a file residing on an SE lcg-uf Unregisters in the the catalogue a file residing on an SE lcg-la Lists the aliases for a given LFN, GUID or SURL lcg-lr Lists the replicas for a given LFN, GUID or SURL lcg-ls Lists file information for given SURLs or LFNs
22
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 22 22 DATA MANAGEMENT lcg_utils examples(1) Upload a file $ lcg-cr file:/home/vlada/banja_luka/mpi.jdl guid:830e7a82-f9fb-4c04-aafd-e24a5f25acdb specify lfn: $ lcg-cr -l lfn:/grid/seegrid/vlada/mpi \ file:/home/vlada/banja_luka/mpi.jdl guid:25fdcd16-bc76-4458-9c1a-8c597fb33194 Replicate a file (one replica per SE) $ lcg-rep -d grid15.rcub.bg.ac.yu \ lfn:/grid/seegrid/vlada/mpi
23
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 23 23 DATA MANAGEMENT lcg_utils examples(2) List replicas $ lcg-lr lfn:/grid/seegrid/vlada/mpi srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/h ome/ae gis/generated/2009-02-18/file8f99889c-c00a- 4b7b- 89d3-d9abf66ba103 srm://se.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/a egis/generated/2009-02-18/file576fbae0-ba6a- 4b21-a205-3d115b2cf7fb List guid $ lcg-lg lfn:/grid/seegrid/vlada/mpi guid:25fdcd16-bc76-4458-9c1a-8c597fb33194
24
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 24 24 DATA MANAGEMENT lcg_utils examples(3) Listing files and directories $ lcg-ls -l lfn:/grid/seegrid/vlada/mpi -rw-rw-r-- 1 165 101 349 lfn:/grid/seegrid/vlada/mpi Copying files out of the Grid (download) $ lcg-cp lfn:/grid/seegrid/vlada/mpi \ file:/home/vlada/banja_luka/down/mpi Deleting replicas lcg-del srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/ home/aegis/generated/2009-02- 19/file36d25d0d-ee61-477eb667- 40b3791786a4 $ lcg-del -a lfn:/grid/aegis/vlada/mpi
25
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 25 25 Data Management Web Portal Developed by Belgrade University Computer Centre Usage Data Management Logical Files Development Environments and Portals Description Enables easy interaction with grid file catalogue using simple Web-based user interface. Based on SEE-GRID File Management Java API. Can be integrated with other Web applications.
26
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 26 26 Data Management Web Portal Available features Directory browsing Directory management operations: creating new subdirectories, renaming/moving and deleting directories, replicating directory content, reading directory attributes and permissions. Copying and registering of new files to the catalog as well as file deletion. Reading file attributes and permissions. Replica & alias management operations. Upload and download of files with possible integration of application-defined upload handlers
27
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 27 27 Data Management Web Portal http://grid02.rcub.bg.ac.yu:8080/NewDataManagement/l ogin-REFS.jsp http://grid02.rcub.bg.ac.yu:8080/NewDataManagement/l ogin-REFS.jsp http://grid02.rcub.bg.ac.yu:8080/NewDataManageme nt/login-WRF-ARW.jsp http://grid02.rcub.bg.ac.yu:8080/NewDataManageme nt/login-WRF-ARW.jsp
28
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 28 28 Job operations
29
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 29 29 JDL JDL (Job Description Language) file attribute = expression Executable = "test.sh"; Arguments = "fileA fileB"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {"test.sh", "fileA", "fileB"}; OutputSandbox = {"std.out", "std.err"}; Environment = {“AEGIS_PATH=$HOME/AEGIS"}; Requirements = RegExp("ce64.phy.bg.ac.rs*",other.GlueCEUniqu eID);
30
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 30 30 SUBMITING JOBS Generating proxy $ voms-proxy-init -voms Myproxy $ myproxy-init -d –n -s Delegate proxy to WMS $ glite-wms-job-delegate-proxy -d Job matching $ glite-wms-job-list-match -a test.jdl Job submission $ glite-wms-job-submit -d -i test.jdl
31
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 31 31 MONITORING JOBS View status of job $ glite-wms-job-status $ glite-wms-job-status -i View logging details $ glite-wms-job-logging-info -v $ glite-wms-job-logging-info -v -i Verbosity level can be from 0 to 3.
32
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 32 32 CANCELING JOBS AND RETRIEVING OUTPUT Cancel a job $ glite-wms-job-cancel $ glite-wms-job-cancel -i Retrieve files from Output Sandbox $ glite-wms-job-output $ glite-wms-job-output -i $ glite-wms-job-output --dir
33
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 33 33 ADVANCED JOB TYPES Job Collection - submission of a set of jobs whose description files are placed in one directory Parametric Jobs - submission of a set of jobs having the same descriptions apart from the values of the parametric attributes. Use when your jobs differs only in argument values or input/output files. DAG jobs (Direct Acyclic Graphs) - submission of a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs. Interactive Jobs MPI Jobs
34
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 34 34 JOB COLLECTION Type = "Collection"; InputSandbox = (“fileA”); Requirements = RegExp("ce64.phy.bg.ac.yu*",other.GlueCEUniqueID); nodes = { [ JobType = "normal"; Executable = "/bin/hostname"; Arguments = "-i"; StdOutput = "out"; InputSandbox = (root.InputSandbox, “fileB”); StdError = "err"; OutputSandbox ={"out","err"}; ], [ JobType = "Normal"; Executable = "/bin/date"; StdOutput = "date"; StdError = "err"; OutputSandbox= {"date"}; ] };
35
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 35 35 JOB COLLECTION submission Collection defined in one file $ glite-wms-job-submit -a -o id collection.jdl Folder contains jdl files $ glite-wms-job-submit -a --collection jdl Collection status view Collection output $ glite-wms-job-output -i id --dir kolekcija $ ls kolekcija ids_nodes.map Node_0 Node_1
36
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 36 36 PARAMETRIC JOBS Type = "Job"; JobType = "Parametric"; Executable = "/bin/echo"; Arguments = "_PARAM_"; Parameters = {red, green, blue}; StdOutput = "colour_PARAM_.txt"; StdError = "error_PARAM_.txt"; OutputSandbox = {"colour_PARAM_.txt","error_PARAM_.txt"};
37
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 37 37 PARAMETRIC JOBS Type = "Job"; JobType = “Parametric"; Executable = "exec"; StdInput = "input_PARAM_.txt"; StdOutput = "output_PARAM_.txt"; Parameters = 10; ParameterStart = 1; ParameterStep = 1; InputSandbox = {" ", " "}; OutputSandbox = {"output_PARAM_.txt"};
38
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 38 38 DAG JOBS [ Type = "dag"; InputSandbox = {"job.sh"}; Nodes = [ nodeA = [ file ="A.jdl" ; ]; nodeB = [ file =“b.jdl" ; ]; nodeC = [ file =“C.jdl" ; ]; nodeD = [ Description = [ JobType = "Normal"; Executable = "job.sh"; StdOutput = "std.out"; StdError = "std.err"; InputSandbox = {root.InputSandbox[0]}; OutputSandbox = {"std.out","std.err"}; ]; Dependencies = { {nodeA,nodeB},{nodeA,nodeC},{{nodeB,nodeC},nodeD} }; ]
39
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 39 39 MPI JOBS JDL file: Type = "Job"; JobType = "MPICH"; NodeNumber = 5; Executable = "mpi.sh"; Arguments = "mpi"; StdOutput = "mpi.out"; StdError = "mpi.err"; InputSandbox = {"mpi.sh","mpi.c"}; OutputSandbox =\ {"mpi.err","mpi.out","mpiexec.out"};
40
Regional SEE-GRID-SCI Grid MeteoTraining, Faculty of Electrical Engineering Banja Luka, July 2-3, 2009 40 40 LINKS http://aegis-ca.rcub.bg.ac.yu/ https://edms.cern.ch/file/722398//gLite-3- UserGuide.pdf http://wiki.egee- see.org/index.php/Quick_User_Guide_for_Subm itting_Jobs https://edms.cern.ch/document/590869/1/ http://wiki.egee-see.org/index.php/SEE- GRID_MPI_User_Guide http://wiki.egee-see.org http://wiki.egee- see.org/index.php/Data_Management_Web_Por tal_-_User_Guide
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.