AliEn Tutorial MODEL th May, 2009

2 May 2009 Installation of the AliEn software AliEn and the GRID Authentication File Catalogue Submitting jobs Outline

3 May 2009 Installation Please, do: wget chmod +x alien-torrent-installer./alien-torrent-installer Default installation First two commands fast Let’s leave the 3 rd running...

4 May 2009 AliEn, AliEn 2 ALICE ENvironment on the GRID GRID middleware Developed as single entry point to the GRID for ALICE (CERN) All the components necessary to build a GRID and interact with other GRIDs File System with metadata Authorization, authentication, job optimization and execution, storage management Audit, quotas, monitoring Interfaces to various GRID implementations Used since: 2002 for centrally managed productions 2006 for user analysis Used by several Virtual Organizations: ALICE, PANDA, CBM, GPCALMA

5 May 2009 Pablo Saiz 5 CERN

6 May 2009 Pablo Saiz 6 LHC 27km particle accelerator 100 meters underground About to start collecting data

7 May 2009 Pablo Saiz 7 A Large Ion Collider Experiment

8 May 2009 Why do we need a catalogue? Loads of data (several PB per year!!) Distributed community Same structure from everywhere Contains location of the file It does not contain the file Possibility of several copies of files Select the closest one Metadata information We will see this later on...

9 May 2009 AliEn 2 File Catalogue Mapping from LFN to SE and PFN UNIX-like file system Metadata GUID Used by all other services Tier1 |--./ | |--cern.ch/ | | |--user/ | | | |--a/ | | | | |--admin/ | | | | | | | | | |--aliprod/ | | | | | |--f/ | | | | |--fca/ | | | | | |--p/ | | | | |--psaiz/ | | | | | |--as/ | | | | | | | | |--dos/ | | | | | | | | |--local/ |--simulation/ | | / | | |--V3.05/ | | | |--Config.C | | | |--grun.C | |--36/ | | |--stderr | | |--stdin | | |--stdout | | |--37/ | | |--stderr | | |--stdin | | |--stdout | | |--38/ | | |--stderr | | |--stdin | | |--stdout | | | | | |--b/ | | | | |--barbera/ ALICE USERS ALICE DATA ALICE REMOTE

10 May 2009 LFN and GUID catalogues / /alice /alice/user/p/psaiz /alice/simulation/2006 … Index 1-JAN JAN FEB AUG-2008 … Index GUID  PFN LFN Catalogue GUID Catalogue AliEn File & Metadata Catalogue LFN  GUID LFN: Logical File Name (what the user sees) GUID: Global Unique Identifier (another way of identifying files) PFN: Physical File Name

11 May 2009 Who uses the catalogue? Everybody IF A FILE IS NOT IN THE CATALOGUE, IT DOES NOT EXIST ON THE GRID Users User data, user executables... Job execution Executable, input, output Packages Source code, binary code, post installation

12 May 2009 Back to the exercises... Did the installation finish? Let’s get ready to access the GRID X509 Certificate authentication Public vs Private key Certificate Authority Proxy certificates

13 May 2009 Preparing certificates In $HOME/.globus: userkey.pem usercert.pem Readable only by the user (chmod 400) Make alien use those files: mkdir $HOME/.alien ln –s $HOME/.globus $HOME/.alien/globus

14 May 2009 Environment files $ALIEN_ROOT/.Environment $HOME/.alien/Environment Bash files sourced before starting alien cat $HOME/.alien/Environment export ALIEN_ORGANISATION=PANDA export ALIEN_USER= EOF Put alien in the PATH mkdir –p $HOME/bin ln –s $HOME/alien/bin/alien $HOME/bin/

15 May 2009 alien proxy-init, alien proxy-info Getting a GRID proxy *****

16 May 2009 Globus related: Permissions on $HOME/.globus/userkey.pem are not private to the user – chmod 400 userkey.pem Your certificate authority is exotic and not known to the server. Your certificate has expired. Clock skew: Your local computer time is in the future with respect to the server's time. Your local computer time is more in the past than the certificate life time. Authentication problems

17 May 2009 Check the certificates in your machines: Store them under e.g: $HOME/.globus/ Check that the key file is read only Setup the environment $HOME/.alien/Environment Put alien in the path Get a valid alien token alien proxy-init Try it out !!

18 May 2009 Standard bash shell with grid comands Main bash features are available. Not all shell helper programs are available. Some local commands (like ls, cat etc) are overwritten with the corresponding GRID commands. File/path tab completion in the virtual GRID directory. Accessing the shell

19 May 2009 Tab completion working!!! Basic commands I

20 May 2009 Basic commands II

21 May 2009 whereis command Gives (list of) SE and pfns for a file

22 May 2009 Viewing the files

23 May 2009 Shell – Copying files (add/get)

24 May 2009 Access the alien shell. alien Check your user name: whoami List the contents of your home directory ls Print the working directory pwd Create the following directory structure in your AliEn space: $HOME/bin (if it is not there) $HOME/macros and $HOME/jdl Get the information of the file: whereis /bin/date Shell – Try it out I

25 May 2009 See the file /bin/date cat /bin/date Copy the file to your local machine cp /bin/date /tmp/my_file Exit the shell and check the file Shell – Try it out II

26 May 2009 Job & Transfer models Jobs TaskQueue Job Broker Job Manager CE JA Central services Site services SplittingExpired PrioritiesMerging Zombies Job optimizers File catalogue LFN GUID Meta data FTD Packman MonALISA JA CE SE Packman MonALISA FTD SE Site A Site B Transfer Manager TransferQueue Transfer Broker Transfer optimizers fts bbftp xrdcp

27 May 2009 ALICE sites Europe Asia North America Africa See Costin’s presentation

28 May 2009 Running jobs More than 25 CPU/years per day

29 May 2009 AliEn AliEn user interface VTDEDG LCG /GLITECONDOR ARC/ NORDUGRID Nice! I STILL do not have to worry about ever changing GRID environment… Interconnecting GRIDs

30 May 2009 Job status

31 May 2009 Your job is submitted and a.is assigned to it. Submitting jobs

32 May 2009 Checking the job status

33 May 2009 Getting the output

34 May 2009 Other useful commands jobListMatch List of CE that can execute the job Spy Output of the job while it is running Masterjob Give information about

35 May 2009 Connect to alien alien login Submit the job submit <<EOF Executable=“date”; EOF Follow the execution top –id, ps trace, ps jdl Get the output cp /proc/ / /job-output/stdout /tmp/out Try it out!!

36 May 2009 Executable: Compulsory field where we give the lfn of the executable that should be stored in /bin or $V0/bin or $HOME/bin. Arguments: They will be passed to the executable. Packages: Type packages in the shell to see what kind of packages are installed. InputFile: The files that will be transported to the node where the job will run. InputData: It will require that the job will be executed in a site close to the files specified here. InputDataList: The filename in which the Job Agent will write the InputData list. InputDataListFormat: The format of the InputData list. JDL fields I

37 May 2009 OutputFile: The files that will be registered in the catalog once the job finishes. OutputArchive: What files will be archived in a zip file. Validationcommand: Specifies the script to be used as a validation script. Receive a mail when the job finishes. TTL: The maximum run time of your job. Split: Split the jobs in several sub jobs. JDL fields II

38 May 2009 Packages packman gives a list of the packages defined/installed installs packages

39 May 2009 Add: Try other commands: Executable=“pandaroot”; Multiple submission Split=“production:1-5”; Create your own executables Try it out !!

40 May 2009 Conclusion AliEn Interface to the GRID Developed by ALICE Also used by PANDA, CBM, GPCALMA In production since 2002 File Catalogue File and Metadata Job submission model Data movement and replication

41 May 2009 Registration – Certificates: AliEn: Gshell: User's guide: dev/alice/apiservice/AA-UserGuide-0.0m.pdf References I

42 May 2009 aliensh Grid Command Online Reference V1.0 dev/alice/apiservice/guide/guide-1.0.htm Previous tutorials: Event Tag System: m/EventTags.html#Event%20tag%20system (INTERNAL NOTE) References II

43 May