12th September 2007UK e-Science All Hands Meeting1 John Kewley Grid Technology Group e-Science Centre STFC Daresbury Laboratory GROWL Scripts: Lightweight Access to Grid Resources

12th September 2007UK e-Science All Hands Meeting2 GROWL Collaborative project between CCLRC (now STFC) Daresbury Laboratory and the Universities of Cambridge and Lancaster, funded by the JISC VRE programme. Project Objectives: to produce a lightweight client-side Grid connection toolkit. Project completed January 2007

12th September 2007UK e-Science All Hands Meeting3 GROWL GROWL addresses the three barriers that newcomers find when using the Grid for the first time: 1.Setting up the client-side middleware 2.Handling of certificates 3.Job submission in the presence of firewalls This talk looks at GROWL Scripts, GROWL Web Services takes a different approach

12th September 2007UK e-Science All Hands Meeting4 Typically need to be root to install (according to documentation) Software must be downloaded from various locations There are many choices for type of installation (too many options?) Client Middleware: Problems

12th September 2007UK e-Science All Hands Meeting5 Installation GROWL scripts provide an alternative way of installing Grid middleware on your client Linux machine to that given on the NGS website: Advantages: Don't need to be a privileged user Will download client middleware packages for your system (assuming it is supported) Minimal setup/configuration About 10–15 mins (if all goes well !)

12th September 2007UK e-Science All Hands Meeting6 The Virtual Data Toolkit (VDT) is an easy to install and configure ensemble of grid middleware GROWL Scripts installs the pre-WS globus client from VDT, as well as gsi- enabled openssl and the best known (IGTF accredited) CA certificates.

12th September 2007UK e-Science All Hands Meeting7 1.Download GROWL Scripts $ cd $ wget 2.Install into home directory $ tar -zxvf Growl.tar.gz 3.Build VDT client (a software distribution that includes globus) $ cd Growl; make VDT Before using any GROWL Scripts, bash users should source ~/Growl/setup.sh while csh users should source ~/Growl/setup.csh Installing Grid Client using GROWL

12th September 2007UK e-Science All Hands Meeting8 Certificate helper scripts –mk-cert –growl-info, growl-login, growl-logout VDT client installation of globus and MyProxy –grid-proxy-init, grid-proxy-info –globus-job-submit, globus-job-run –gsissh, gsiscp, openssl –myproxy-init, myproxy-info, myproxy-logon GROWL wrapper scripts –growl-submit, growl-status, growl-get-output, –growl-sh, growl-cp, growl-mkdir, growl-rm, growl-mv, –growl-pwd, growl-which, growl-get-jobmanager, growl-queue GROWL Scripts: Contents

12th September 2007UK e-Science All Hands Meeting9 Hard to remember openssl commands are wrapped for you –Fewer passwords need to be entered –Correct file and directory permissions are applied Certificate Manipulation

12th September 2007UK e-Science All Hands Meeting10 mk-cert $ openssl pkcs12 –in mykey.p12 \ -clcerts –nokeys -out usercert.pem $ openssl pkcs12 –in mykey.p12 \ –nocerts -out userkey.pem [confirm] $ chmod 444 usercert.pem $ chmod 400 userkey.pem $ mv userkey.pem ~/.globus $ mv usercert.pem ~/.globus $ chmod 700 ~/.globus $ mk-cert mykey.p12 [ ]

12th September 2007UK e-Science All Hands Meeting11 growl-login If you need to upload your certificate to MyProxy and generate a local proxy, growl-login is provided: $ grid-proxy-init... $ myproxy-init Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley Enter GRID pass phrase for this identity:... Enter MyProxy pass phrase: Verifying - Enter MyProxy pass phrase: $ growl-login Password to protect MyProxy credential: Enter GRID pass phrase for this id:

12th September 2007UK e-Science All Hands Meeting12 growl-info A wrapper for grid-cert-info, grid-proxy-info and myproxy-info $ growl-info Certificate Information (including validity) subject= /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley notBefore=Jun 15 16:10: GMT notAfter=Jun 15 16:10: GMT Local proxy certificate(s) subject : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley issuer : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley identity : /C=UK/O=eScience/OU=CLRC/L=DL/CN=john kewley type : Proxy draft (pre-RFC) compliant impersonation proxy strength : 512 bits path : /tmp/x509up_u13445 timeleft : 11:57:19

12th September 2007UK e-Science All Hands Meeting13 GROWL job submission Help with transparency - user shouldn't really need to know –Machine's jobmanager –Home directory location –Location in your path of executable Firewall problems minimised

12th September 2007UK e-Science All Hands Meeting14 Running a grid job (1) $ growl-submit dl1.nw-grid.ac.uk hostname $ growl-status PENDING $ growl-status DONE $ growl-get-output comp023.nw-grid.ac.uk $ growl-submit -c dl1.nw-grid.ac.uk hostname $ growl-status PENDING $ growl-status DONE $ growl-get-output comp021.nw-grid.ac.uk

12th September 2007UK e-Science All Hands Meeting15 Globus + Firewalls Client Grid Resource globus-job-submit Results gsiscp jobmanager sshd gsissh /GSI-SSHTerm globus-job-get_result

12th September 2007UK e-Science All Hands Meeting16 GROWL + Firewalls Client Grid Resource growl-submit jobmanager sshd globus-job-get-output growl-get-output (using gsissh )

12th September 2007UK e-Science All Hands Meeting17 growl-submit : uses growl-get-jobmanager to obtain default parallel queue, rather than defaulting to jobmanager-fork uses growl-which to get full path of executable, ensuring it is in your path growl-get-output : uses gsissh to do remote retrieval, avoiding client firewall problem Advantages

12th September 2007UK e-Science All Hands Meeting18 Equivalents of many of the standard unix command tools are provided for remote filestore manipulation. growl-ls : contents of directory growl-mkdir : (sub)directory creation growl-rm : file removal growl-mv : renaming/moving files growl-which : finds executable in your path growl-pwd : prints your home directory on the grid resource growl-sh : gsissh wrapper (using default ports) growl-cp : remote file copying, including "3 rd party" An additional parameter (the grid resource) is required Remote filestore manipulation

12th September 2007UK e-Science All Hands Meeting19 growl-cp can be used to stage and retrieve files. The syntax follows that of scp. It can also be used for "3 rd party" file transfers For 3 rd party transfers to work, there has to be a route through all firewalls between the 2 remote resources in one direction or the other $ growl-cp my_input_file.txt dl1.nw-grid.ac.uk:. $ growl-cp dl1.nw-grid.ac.uk:my_output.txt. $ growl-cp lv1.nw-grid.ac.uk:my_file.txt dl1.nw-grid.ac.uk:. Remote file copying using growl-cp

12th September 2007UK e-Science All Hands Meeting20 growl-cp (1) Client Grid Resources B A

12th September 2007UK e-Science All Hands Meeting21 growl-cp (2) Client Grid Resources B A

12th September 2007UK e-Science All Hands Meeting22 growl-cp (3) Client Grid Resources B A

12th September 2007UK e-Science All Hands Meeting23 1.Easy way to build VDT 2.As above + certificate scripts 3.As above + use of job submission features Usage patterns

12th September 2007UK e-Science All Hands Meeting24 1.Scripting help for a simplistic DIY meta-scheduler 2.Use of Java CoG-kit + GSI-SSHTERM on Windows Current/future work

12th September 2007UK e-Science All Hands Meeting25 GROWL + Firewalls (3-tier) Client GROWL ServerGrid Resource WS I/F GROWL Scripts WS sshd jobmanager growl-submit growl-get-output growl-cp

12th September 2007UK e-Science All Hands Meeting26 Summary 1.Useful as an easy way to build VDT 2.Simpler job submission: –less need be known about Grid resources –less firewall pain for retrieving data 3.Firewall-aware 3 rd party file copying

12th September 2007UK e-Science All Hands Meeting27 GROWL Web Services Analogous interface to those we have seen on previous slides. –Remote file manipulation –Job submission –Interfacing to SRB Wrappers being developed for access from R, Fortran, C and C++ Users at both Lancaster and Bristol using GROWL Web Services