1. A proposal for standardizing the working environment for a LCG/EGEE job
David Bouvet
Fabio Hernandez
IN2P3 Computing Centre - Lyon
HEPIX, Karlsruhe, 13/05/2005
LCG Operations Workshop, Bologna, 25/05/2005

2. Motivation Problem raised some months ago by Jeff Templon:
D0 jobs encountered problems in Lyon due to different use of environment variables to address scratch/temp disk space
Standard is defined for:
Environment Variables « IEEE Std , 2004 POSIX Part 1: Base definitions, Amendment 8 »
among which: HOME, PATH, PWD, SHELL, TMPDIR, USER
Batch Environment Services « IEEE Std , 2004 POSIX Part 2: Shell and Utilities, Amendment 1 »
PBS_ENVIRONMENT, PBS_JOBID, PBS_JOBNAME, PBS_QUEUE
PBS_O_HOME, PBS_O_HOST, PBS_O_LOGNAME, PBS_O_PATH, PBS_O_QUEUE, PBS_O_SHELL, PBS_O_WORKDIR
There is no standard definition of environment variables for grid batch jobs
 Proposal for LCG/EGEE sites of a common definition of minimal set of environment variables for grid batch jobs
D. Bouvet / F. Hernandez

3. Conditions of test: ATLAS VO, short queue
Current status
Environment variables for grid batch job have been checked on several LCG/EGEE sites (among which all the LCG Tier-1s)
Conditions of test: ATLAS VO, short queue
Batch system
CEs distribution
# CEs checked
BQS 3 2
CONDOR 4
TORQUE 72 11
PBS 36 13
LSF 5
D. Bouvet / F. Hernandez

4. Current status: POSIX variables
 : defined  : not defined on some sites
 not all these variables are defined on all CEs
Variable
BQS
CONDOR
TORQUE
PBS
LSF
POSIX basic: HOME PATH PWD SHELL TMPDIR USER  
 
POSIX batch
D. Bouvet / F. Hernandez

5. Current status (cont.)
 : defined
 : not defined on some sites
 even for Globus, not all the sites define the same set of environment variables.
Variable
BQS
CONDOR
TORQUE
PBS
LSF
GLOBUS variables: GLOBUS_LOCATION GLOBUS_PATH GLOBUS_TCP_PORT_RANGE X509_USER_PROXY 
 
MYPROXY_SERVER (useful for proxy renewal) 
D. Bouvet / F. Hernandez

6. Current status: LCG environment variables (middleware related)
(list from the LCG Users Guide)
Variable
Definition
BQS
CONDOR
TORQUE
PBS
LSF
EDG_LOCATION
Base of the installed EDG software 
 
LCG_LOCATION
Base of the installed LCG software
EDG_WL_JOBID
Job ID (for a running job) in a WN
EDG_WL_LOCATION
Base of the EDG’s WMS software
EDG_WL_PATH
Path for EDG’s WMS commands 
EDG_WL_RB_BROKERINFO
Location of the .BrokerInfo file in a WN
LCG_GFAL_INFOSYS
Location of the BDII for lcg-utils and GFAL
LCG_CATALOG_TYPE
Type of file catalog used (edg or lfc) for lcg-utils and GFAL
LFC_HOST
Location of the LFC catalog (only for catalog type lfc)
D. Bouvet / F. Hernandez

7. Current status: LCG environment variables (job related)
(list from the LCG Users Guide)
Variable
Definition
BQS
CONDOR
TORQUE
PBS
LSF
EDG_TMP
Temp directory 
 
LCG_TMP
VO_<VO-name>_DEFAULT_SE
Default SE defined for a CE in a WN
VO_<VO-name>_SW_DIR
Base directory of the VO’s software in a WN
D. Bouvet / F. Hernandez

8. Current status: gLite environment variables
gLite environment variables on WN (in config. files and scripts) from gLite installation guide:
GLITE_LOCATION /opt/glite
GLITE_LOCATION_VAR /var/glite
GLITE_LOCATION_LOG /var/log/glite
GLITE_LOCATION_TMP /tmp/glite
GLITE_LOCATION_TMP  another tmp directory to clean!
D. Bouvet / F. Hernandez

9. Proposal for standardization
Variable type
Definition
Name
POSIX
Home directory of job user on WN
HOME
Temp directory
TMPDIR (currently LCG_TMP, EDG_TMP, GLITE_LOCATION_TMP)
PWD
SHELL
PATH
Grid batch jobs
Job working directory on WN
GRID_WORKDIR
Site name on which the job run (same as siteName in Information Provider)
GRID_SITENAME
WN hostname on which the job runs
GRID_HOSTNAME
CE and queue names on which the job run (same as GlueCEUniqueID in Information Provider)
GRID_CEID
Job ID in local batch system
GRID_LOCAL_JOBID
Job ID on grid
GRID_GLOBAL_JOBID (currently EDG_WL_JOBID)
User’s DN of certificate
GRID_USERID
D. Bouvet / F. Hernandez

10. Proposal for standardization (cont.)
Use of POSIX variable when existing
TMPDIR: POSIX variable which could also be used by middleware for creating temporary files (courrently LCG_TMP, EDG_TMP, GLITE_LOCATION_TMP)
HOME: MPI jobs need a home directory and some grids (like OSG) have a permanent mapping for each grid user
D. Bouvet / F. Hernandez

11. Proposal for standardization (cont.)
Minimal set of environment variable (not related to middleware). The naming convention must be independent of grid middleware name for easing portability of grid jobs
GRID_WORKDIR: job-specific working directory (file permissions 700)
e.g.: /scratch/atlas ccwl0092
GRID_SITENAME: to know on which site the job run (same as siteName in the Information System)
e.g.: IN2P3-CC
GRID_HOSTNAME: full host name - could be useful to know the WN hostname for problem tracking (and parallel jobs?) [not strictly necessary but may be convenient for users to have it]
e.g.: ccwl0006.in2p3.fr
GRID_CEID: CE name on which the job run (same as GlueCEUniqueID in Information System)
e.g.: heplnx201.pp.rl.ac.uk:2119/jobmanager-torque-short
GRID_LOCAL_JOBID: useful for problem tracking (and parallel jobs?)
e.g.: lcg
GRID_GLOBAL_JOBID: same as EDG_WL_JOBID for LCG
e.g.:
GRID_USERID: DN of user’s certificate (already set on some sites)
e.g.: /O=GRID-FR/C=FR/O=CNRS/OU=CC-LYON/CN=David
D. Bouvet / F. Hernandez

12. Proposal for standardization (cont.)
Site-specific
TMPDIR
HOME
GRID_WORKDIR
GRID_LOCAL_JOBID
GRID_HOSTNAME
GRID_SITENAME
GRID_USERID
Grid-specific
GRID_CEID
GRID_GLOBAL_JOBID
Minimal modification to the RB possible but not required
Modifications to the job managers are required => need to identify the people responsible for maintaining them
Sites need some way to configure the job managers to specify the site-dependent parameters
D. Bouvet / F. Hernandez

13. Proposal for standardization (cont.)
When agreed on a set of variables and a naming convention, this standard should be implemented on all LCG/EGEE CEs.
A (short) document is being written describing
the meaning of the variables
the compulsory/optional character of each one of them
The document will be distributed and your feedback will be welcome
D. Bouvet / F. Hernandez

14. Please give us your feedback
Questions/Comments
Please give us your feedback
D. Bouvet / F. Hernandez