Presentation is loading. Please wait.

Presentation is loading. Please wait.

Reading e-Science Centre

Similar presentations


Presentation on theme: "Reading e-Science Centre"— Presentation transcript:

1 Reading e-Science Centre
VEIGA: Very Easy Interface to Grid Applications (formerly known as "Styx Grid Services") Jon Blower Technical Director Reading e-Science Centre

2 Overview of VEIGA Aim is to make running applications on a Grid as easy as running local applications Apps can be run from remote locations using the same command as for running the app locally Simple workflows can be created using simple shell scripts Data can be streamed directly from app to app Can run applications on Condor and Globus resources Secure

3 Example: GULP General Utility Lattice Program (Julian Gale, IC)
Used extensively in mineral sciences (e.g. NERC e-Minerals project) Takes a single input file (input.txt) describing a crystal lattice Outputs a single file (gulp_0001.txt) with bond lengths, Helmholtz free energy etc Having deployed GULP on a VEIGA server, it is run by clients like so: veigarun <server> <port> gulp Or can create simple wrapper script so that we can just type: gulp

4 How it works GULP is installed on the VEIGA server
VEIGA server holds an XML description of the GULP program: <gridservice name="gulp" command="c:\gulp\gulp_win.exe"> <inputs> <input type="file" name="input.txt"/> </inputs> <outputs> <output type="file" name="gulp_0001.txt"/> </outputs> </gridservice> veigarun is a completely generic VEIGA client program and performs the following tasks: Downloads the XML description of GULP Uses it to parse the command line to find out what input and output files to expect (in case of GULP, these happen to have fixed names) Uploads input file(s) to VEIGA server Starts GULP, which runs on the VEIGA server Redirects the standard streams (stdin, stdout, stderr) downloads the output file(s) as they are produced (doesn't wait for finish)

5 Some Benefits Very easy to deploy and run applications in the VEIGA environment See Useful when application is tied to a particular platform but you want to run it from elsewhere and pretend it's local Can easily substitute the "Grid-enabled" GULP for your local GULP There is a GUI for GULP that wraps the GULP executable. Can simply point this to the remote GULP instead of a local installation: no change to GUI code Can use the remote application in place of local applications in shell scripts Easy to create "workflows" Can use all shell constructs such as pipes etc and combine with local apps Can arrange for data to take most direct route between servers Can do computational steering Simple implementation but it works Custom clients and server plug-ins can be created using a Java API Documentation on this is not great yet… WSRF bindings have been created (Andrew Harrison, Cardiff): VEIGA app modelled as WS-Resource

6 VEIGA and Condor Worker nodes VEIGA Server + Condor VEIGA client
If VEIGA server is a Condor submit host, can delegate jobs to a Condor pool Helps to load balance Not everything runs on VEIGA server Multiple jobs can be run with a single command: Upload directories instead of single files: myprog –i inputs –o outputs Easy high-throughput computing Demo VEIGA Server + Condor VEIGA client

7 VEIGA and Globus myproxy NGS server VEIGA server VEIGA client
VEIGA can be used as a portal to Globus resources E.g. NGS Users log on to VEIGA with MyProxy user/password VEIGA retrieves proxy certificate Helps with Globus firewall problem VEIGA server takes pain on behalf of client VEIGA server used as staging post for files Could transfer files direct from client to destination but would require client to have GridFTP or similar VEIGA server VEIGA client

8 Some technical points of (possible) interest
VEIGA uses the Styx protocol for distributed systems JStyx – Java implementation of Styx developed at ReSC Styx is ~20 years old Single server process handles all traffic File upload/download, progress monitoring, control messages Styx is a multiplexing protocol Can tunnel traffic through SSH Client-server connections are persistent Servers only need one incoming port open through firewall Clients need no incoming ports open VEIGA only requires a JRE – includes all other dependencies Whole system is a 3MB download

9 Security VEIGA can be set to permit access to registered users only
Contains set of usernames/passwords. Passwords stored as secure hash, not plain text No client certificates! (my bank does not use client certs) All traffic can be encrypted using SSL Server has a certificate Or, everything can be tunnelled down SSH Because VEIGA is a single server process with single port Keeps some sysadmins happy For access to GSI-protected resources, users can log into a VEIGA server with MyProxy password Then VEIGA retrieves proxy certificate for them VEIGA behaves like a Grid portal

10 Possible test case: GCEP
Grid for Coupled Ensemble Prediction Run ensembles of climate simulations and get stats of results Use resources at Reading, RAL, British Antarctic Survey (BAS), NGS Need to share results among project partners and allow partners to submit jobs Need a resource broker (even if very simple) Can we do the whole thing using SSH (and VEIGA)? No Globus, no GSI, no proxy certs (except for NGS access) SSH very well understood and trusted by all relevant sysadmins and already installed Shared file system with sshfs (or SRB if not) Give me a good reason why not!

11 Conclusions VEIGA favours simplicity over efficiency
A simple tool for simple jobs Users already know how to use it! Remote apps can be run just like local ones Doesn't feel like job submission Simple workflows are created with shell scripts Can use VEIGA as an easy-to-use "portal" to Condor and Globus Client interface is identical in all cases Could use same principles to launch jobs using AHE / GridSAM VEIGA client does not know about the back end VEIGA as AHE client plug-in? See (as of 4/5/06) Contains tutorials, links to papers etc Current release (0.2.0) does not contain everything I've talked about today – next release (0.3.0)!


Download ppt "Reading e-Science Centre"

Similar presentations


Ads by Google