A proposal for standardizing the working environment for a LCG/EGEE job David Bouvet (dbouvet@in2p3.fr) Fabio Hernandez (fabio@in2p3.fr) IN2P3 Computing Centre - Lyon HEPIX, Karlsruhe, 13/05/2005 LCG Operations Workshop, Bologna, 25/05/2005

Motivation Problem raised some months ago by Jeff Templon: D0 jobs encountered problems in Lyon due to different use of environment variables to address scratch/temp disk space Standard is defined for: Environment Variables « IEEE Std 1003.1, 2004 POSIX Part 1: Base definitions, Amendment 8 » http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html among which: HOME, PATH, PWD, SHELL, TMPDIR, USER Batch Environment Services « IEEE Std 1003.1, 2004 POSIX Part 2: Shell and Utilities, Amendment 1 » http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap03.html PBS_ENVIRONMENT, PBS_JOBID, PBS_JOBNAME, PBS_QUEUE PBS_O_HOME, PBS_O_HOST, PBS_O_LOGNAME, PBS_O_PATH, PBS_O_QUEUE, PBS_O_SHELL, PBS_O_WORKDIR There is no standard definition of environment variables for grid batch jobs  Proposal for LCG/EGEE sites of a common definition of minimal set of environment variables for grid batch jobs D. Bouvet / F. Hernandez 2

Conditions of test: ATLAS VO, short queue Current status Environment variables for grid batch job have been checked on several LCG/EGEE sites (among which all the LCG Tier-1s) Conditions of test: ATLAS VO, short queue Batch system CEs distribution # CEs checked BQS 3 2 CONDOR 4 TORQUE 72 11 PBS 36 13 LSF 5 D. Bouvet / F. Hernandez 3

Current status: POSIX variables  : defined  : not defined on some sites  not all these variables are defined on all CEs Variable BQS CONDOR TORQUE PBS LSF POSIX basic: HOME PATH PWD SHELL TMPDIR USER     POSIX batch D. Bouvet / F. Hernandez 4

Current status (cont.)  : defined  : not defined on some sites  even for Globus, not all the sites define the same set of environment variables. Variable BQS CONDOR TORQUE PBS LSF GLOBUS variables: GLOBUS_LOCATION GLOBUS_PATH GLOBUS_TCP_PORT_RANGE X509_USER_PROXY    MYPROXY_SERVER (useful for proxy renewal)  D. Bouvet / F. Hernandez 5

Current status: LCG environment variables (middleware related) (list from the LCG Users Guide) Variable Definition BQS CONDOR TORQUE PBS LSF EDG_LOCATION Base of the installed EDG software    LCG_LOCATION Base of the installed LCG software EDG_WL_JOBID Job ID (for a running job) in a WN EDG_WL_LOCATION Base of the EDG’s WMS software EDG_WL_PATH Path for EDG’s WMS commands  EDG_WL_RB_BROKERINFO Location of the .BrokerInfo file in a WN LCG_GFAL_INFOSYS Location of the BDII for lcg-utils and GFAL LCG_CATALOG_TYPE Type of file catalog used (edg or lfc) for lcg-utils and GFAL LFC_HOST Location of the LFC catalog (only for catalog type lfc) D. Bouvet / F. Hernandez 6

Current status: LCG environment variables (job related) (list from the LCG Users Guide) Variable Definition BQS CONDOR TORQUE PBS LSF EDG_TMP Temp directory    LCG_TMP VO_<VO-name>_DEFAULT_SE Default SE defined for a CE in a WN VO_<VO-name>_SW_DIR Base directory of the VO’s software in a WN D. Bouvet / F. Hernandez 7

Current status: gLite environment variables gLite environment variables on WN (in config. files and scripts) from gLite installation guide: GLITE_LOCATION /opt/glite GLITE_LOCATION_VAR /var/glite GLITE_LOCATION_LOG /var/log/glite GLITE_LOCATION_TMP /tmp/glite GLITE_LOCATION_TMP  another tmp directory to clean! D. Bouvet / F. Hernandez 8

Proposal for standardization Variable type Definition Name POSIX Home directory of job user on WN HOME Temp directory TMPDIR (currently LCG_TMP, EDG_TMP, GLITE_LOCATION_TMP) PWD SHELL PATH Grid batch jobs Job working directory on WN GRID_WORKDIR Site name on which the job run (same as siteName in Information Provider) GRID_SITENAME WN hostname on which the job runs GRID_HOSTNAME CE and queue names on which the job run (same as GlueCEUniqueID in Information Provider) GRID_CEID Job ID in local batch system GRID_LOCAL_JOBID Job ID on grid GRID_GLOBAL_JOBID (currently EDG_WL_JOBID) User’s DN of certificate GRID_USERID D. Bouvet / F. Hernandez 9

Proposal for standardization (cont.) Use of POSIX variable when existing TMPDIR: POSIX variable which could also be used by middleware for creating temporary files (courrently LCG_TMP, EDG_TMP, GLITE_LOCATION_TMP) HOME: MPI jobs need a home directory and some grids (like OSG) have a permanent mapping for each grid user D. Bouvet / F. Hernandez 10

Proposal for standardization (cont.) Minimal set of environment variable (not related to middleware). The naming convention must be independent of grid middleware name for easing portability of grid jobs GRID_WORKDIR: job-specific working directory (file permissions 700) e.g.: /scratch/atlas0011293.ccwl0092 GRID_SITENAME: to know on which site the job run (same as siteName in the Information System) e.g.: IN2P3-CC GRID_HOSTNAME: full host name - could be useful to know the WN hostname for problem tracking (and parallel jobs?) [not strictly necessary but may be convenient for users to have it] e.g.: ccwl0006.in2p3.fr GRID_CEID: CE name on which the job run (same as GlueCEUniqueID in Information System) e.g.: heplnx201.pp.rl.ac.uk:2119/jobmanager-torque-short GRID_LOCAL_JOBID: useful for problem tracking (and parallel jobs?) e.g.: lcg0509104420-07243 GRID_GLOBAL_JOBID: same as EDG_WL_JOBID for LCG e.g.: https://lxn1188.cern.ch:9000/HPMN2WVHurMlji-Fnqba0A GRID_USERID: DN of user’s certificate (already set on some sites) e.g.: /O=GRID-FR/C=FR/O=CNRS/OU=CC-LYON/CN=David Bouvet/Email=dbouvet@in2p3.fr D. Bouvet / F. Hernandez 11

Proposal for standardization (cont.) Site-specific TMPDIR HOME GRID_WORKDIR GRID_LOCAL_JOBID GRID_HOSTNAME GRID_SITENAME GRID_USERID Grid-specific GRID_CEID GRID_GLOBAL_JOBID Minimal modification to the RB possible but not required Modifications to the job managers are required => need to identify the people responsible for maintaining them Sites need some way to configure the job managers to specify the site-dependent parameters D. Bouvet / F. Hernandez 12

Proposal for standardization (cont.) When agreed on a set of variables and a naming convention, this standard should be implemented on all LCG/EGEE CEs. A (short) document is being written describing the meaning of the variables the compulsory/optional character of each one of them The document will be distributed and your feedback will be welcome D. Bouvet / F. Hernandez 13

Please give us your feedback Questions/Comments Please give us your feedback D. Bouvet / F. Hernandez 14