Reading e-Science Centre VEIGA: Very Easy Interface to Grid Applications (formerly known as "Styx Grid Services") Jon Blower Technical Director Reading e-Science Centre jdb@mail.nerc-essc.ac.uk http://www.resc.rdg.ac.uk

Overview of VEIGA Aim is to make running applications on a Grid as easy as running local applications Apps can be run from remote locations using the same command as for running the app locally Simple workflows can be created using simple shell scripts Data can be streamed directly from app to app Can run applications on Condor and Globus resources Secure

Example: GULP General Utility Lattice Program (Julian Gale, IC) Used extensively in mineral sciences (e.g. NERC e-Minerals project) Takes a single input file (input.txt) describing a crystal lattice Outputs a single file (gulp_0001.txt) with bond lengths, Helmholtz free energy etc Having deployed GULP on a VEIGA server, it is run by clients like so: veigarun <server> <port> gulp Or can create simple wrapper script so that we can just type: gulp

How it works GULP is installed on the VEIGA server VEIGA server holds an XML description of the GULP program: <gridservice name="gulp" command="c:\gulp\gulp_win.exe"> <inputs> <input type="file" name="input.txt"/> </inputs> <outputs> <output type="file" name="gulp_0001.txt"/> </outputs> </gridservice> veigarun is a completely generic VEIGA client program and performs the following tasks: Downloads the XML description of GULP Uses it to parse the command line to find out what input and output files to expect (in case of GULP, these happen to have fixed names) Uploads input file(s) to VEIGA server Starts GULP, which runs on the VEIGA server Redirects the standard streams (stdin, stdout, stderr) downloads the output file(s) as they are produced (doesn't wait for finish)

Some Benefits Very easy to deploy and run applications in the VEIGA environment See http://www.resc.rdg.ac.uk/veiga Useful when application is tied to a particular platform but you want to run it from elsewhere and pretend it's local Can easily substitute the "Grid-enabled" GULP for your local GULP There is a GUI for GULP that wraps the GULP executable. Can simply point this to the remote GULP instead of a local installation: no change to GUI code Can use the remote application in place of local applications in shell scripts Easy to create "workflows" Can use all shell constructs such as pipes etc and combine with local apps Can arrange for data to take most direct route between servers Can do computational steering Simple implementation but it works Custom clients and server plug-ins can be created using a Java API Documentation on this is not great yet… WSRF bindings have been created (Andrew Harrison, Cardiff): VEIGA app modelled as WS-Resource

VEIGA and Condor Worker nodes VEIGA Server + Condor VEIGA client If VEIGA server is a Condor submit host, can delegate jobs to a Condor pool Helps to load balance Not everything runs on VEIGA server Multiple jobs can be run with a single command: Upload directories instead of single files: myprog –i inputs –o outputs Easy high-throughput computing Demo VEIGA Server + Condor VEIGA client

VEIGA and Globus myproxy NGS server VEIGA server VEIGA client VEIGA can be used as a portal to Globus resources E.g. NGS Users log on to VEIGA with MyProxy user/password VEIGA retrieves proxy certificate Helps with Globus firewall problem VEIGA server takes pain on behalf of client VEIGA server used as staging post for files Could transfer files direct from client to destination but would require client to have GridFTP or similar VEIGA server VEIGA client

Some technical points of (possible) interest VEIGA uses the Styx protocol for distributed systems JStyx – Java implementation of Styx developed at ReSC http://jstyx.sf.net Styx is ~20 years old Single server process handles all traffic File upload/download, progress monitoring, control messages Styx is a multiplexing protocol Can tunnel traffic through SSH Client-server connections are persistent Servers only need one incoming port open through firewall Clients need no incoming ports open VEIGA only requires a JRE – includes all other dependencies Whole system is a 3MB download

Security VEIGA can be set to permit access to registered users only Contains set of usernames/passwords. Passwords stored as secure hash, not plain text No client certificates! (my bank does not use client certs) All traffic can be encrypted using SSL Server has a certificate Or, everything can be tunnelled down SSH Because VEIGA is a single server process with single port Keeps some sysadmins happy For access to GSI-protected resources, users can log into a VEIGA server with MyProxy password Then VEIGA retrieves proxy certificate for them VEIGA behaves like a Grid portal

Possible test case: GCEP Grid for Coupled Ensemble Prediction Run ensembles of climate simulations and get stats of results Use resources at Reading, RAL, British Antarctic Survey (BAS), NGS Need to share results among project partners and allow partners to submit jobs Need a resource broker (even if very simple) Can we do the whole thing using SSH (and VEIGA)? No Globus, no GSI, no proxy certs (except for NGS access) SSH very well understood and trusted by all relevant sysadmins and already installed Shared file system with sshfs (or SRB if not) Give me a good reason why not!

Conclusions VEIGA favours simplicity over efficiency A simple tool for simple jobs Users already know how to use it! Remote apps can be run just like local ones Doesn't feel like job submission Simple workflows are created with shell scripts Can use VEIGA as an easy-to-use "portal" to Condor and Globus Client interface is identical in all cases Could use same principles to launch jobs using AHE / GridSAM VEIGA client does not know about the back end VEIGA as AHE client plug-in? See http://www.resc.rdg.ac.uk/veiga (as of 4/5/06) Contains tutorials, links to papers etc Current release (0.2.0) does not contain everything I've talked about today – next release (0.3.0)!