SARA Reken- en NetwerkdienstenToPoS | 3 juni 2007 More efficient job submission Evert Lammerts SARA Computing and Networking Services High Performance Computing and Visualization e-Science Support

SARA Reken- en Netwerkdiensten Job submission works! But it’s up to YOU to monitor the cycle of a job strange return codes does the job status (Success, Failed) really mean what you think it does resubmit failed jobs Etc, etc, etc… Success rate: 70% ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Pilot Jobs I Normally: With Pilot jobs: ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Pilot Jobs II Pilot jobs: Are basically loops Find out what to do by fetching tasks Run till there are no more tasks, or… Run till they are out of Wall Time Tasks: Contain parameters Are served by ToPoS from a webserver When there are no more tasks, you’re sure that you’re done! ToPoS| 3 juni 2007

SARA Reken- en Netwerkdiensten ToPoS is a service is based on HTTP can be queried by any HTTP client! is strong because it’s simple! Has task realms and pools ToPos| 3 juni 2007

SARA Reken- en Netwerkdiensten Great! But now what… A trivial example: matrix multiplication ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Approach 1 file per row of matrix A 1 file per column of matrix B The Task: Contains an A[row] and B[col] Is a shell script that can be sourced The Pilot job: Fetches a task (None? Exit!) Sources the task Calculates the result Puts the result in a file ([row]-[col].txt) Deletes the task ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Upload files to UI Wget WinSCP? ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Lets go! I Login to UI ssh (or putty!) Create proxy startGridSession [VO] Enter certificate password Unpack the demo: tar -xvvf toposDemo.tar.gz Unzip -xvvf toposDemo.zip ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Transfer input data export LFC_HOST='lfc.grid.sara.nl‘ lfc-mkdir /grid/[VO]/[USERNAME] inputdata/copyFiles.sh: #!/bin/bash if [ $# -ne 2 ] then echo "Supply your VO name and your username!" exit 0 fi for x in *.txt do lcg-cr --vo ${1} -d srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/${1}/${2}/${x} -l lfn:/grid/${1}/${2}/${x} "file://${PWD}/${x}" done ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Pilot Job Pilotjob.sh ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten JDL pilotjob.jdl: Type = "Job"; JobType = "Normal"; Executable = "/bin/sh"; Arguments = "pilotjob.sh [REALM_URL] [VO] [USER]"; StdOutput = "stdout.log"; StdError = "stderr.log"; InputSandbox = {"pilotjob.sh"}; OutputSandbox = {"stdout.log","stderr.log"}; ShallowRetryCount = 1; ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Submit 10 pilot jobs submit.sh: #!/bin/bash touch jobids.txt for i in {1..10} do glite-wms-job-submit –d $USER pilotjob.jdl done ToPoS | 3 juni 2007

SARA Reken- en Netwerkdiensten Check status glite-wms-job-status --input jobids.txt ToPoS | 3 juni 2007