1. 23.06.2006GAT1 Alexander Beck-Ratzka OFFIS, 23.06.06

2. 23.06.2006GAT2 GAT: Agenda Introduction –Why a new Grid-API? –What GAT offers, what not? –Examples –Architecture The single API groups –File (-stream) management, logical file management, –AdvertService management, –Job management Adaptors Conclusion

3. 23.06.2006GAT3 Why a new Grid-API? GAT as a uniform API for the access to heterogeneous grid technologies / grid middleware. GAT is only a framework; the real grid operationen must be done by adaptors. GAT offers the possibility for accessing adpaptors. New grid technologies only needs to be coupled to GAT via adaptors -> No changes in the source sode necessary, even not If using new grid technologies. GAT enables a simple access to grid technologies. GAT uses the grid middleware, which is available at the moment, where is should be used, and this by only typing a single statement for GAT_JobSubmit or GATFileCopy...

4. 23.06.2006GAT4 File copy: CoG/RFT package org.globus.ogsa.gui; import java.io.BufferedReader; import java.io.File; import java.io.FileReader; import java.net.URL; import java.util.Date; import java.util.Vector; import javax.xml.rpc.Stub; import org.apache.axis.message.MessageElement; import org.apache.axis.utils.XMLUtils; import org.globus.* import org.gridforum.ogsi.* import org.gridforum.ogsi.holders.TerminationTimeTypeHolder; import org.w3c.dom.Document; import org.w3c.dom.Element; public class RFTClient { public static void copy (String source_url, String target_url) { try { File requestFile = new File (source_url); BufferedReader reader = null; try { reader = new BufferedReader (new FileReader (requestFile)); } catch (java.io.FileNotFoundException fnfe) { } Vector requestData = new Vector (); requestData.add (target_url); TransferType[] transfers1 = new TransferType[transferCount]; RFTOptionsType multirftOptions = new RFTOptionsType (); multirftOptions.setBinary (Boolean.valueOf ( (String)requestData.elementAt (0)).booleanValue ()); multirftOptions.setBlockSize (Integer.valueOf ( (String)requestData.elementAt (1)).intValue ()); multirftOptions.setTcpBufferSize (Integer.valueOf ( (String)requestData.elementAt (2)).intValue ()); multirftOptions.setNotpt (Boolean.valueOf ( (String)requestData.elementAt (3)).booleanValue ()); multirftOptions.setParallelStreams (Integer.valueOf ( (String)requestData.elementAt (4)).intValue ()); multirftOptions.setDcau(Boolean.valueOf( (String)requestData.elementAt (5)).booleanValue ()); int i = 7; for (int j = 0; j < transfers1.length; j++) { transfers1[j] = new TransferType (); transfers1[j].setTransferId (j); transfers1[j].setSourceUrl ((String)requestData.elementAt (i++)); transfers1[j].setDestinationUrl ((String)requestData.elementAt (i++)); transfers1[j].setRftOptions (multirftOptions); } TransferRequestType transferRequest = new TransferRequestType (); transferRequest.setTransferArray (transfers1); int concurrency = Integer.valueOf ((String)requestData.elementAt(6)).intValue(); if (concurrency > transfers1.length) { System.out.println ("Concurrency should be less than the number" "of transfers in the request"); System.exit (0); } transferRequest.setConcurrency (concurrency); TransferRequestElement requestElement = new TransferRequestElement (); requestElement.setTransferRequest (transferRequest); ExtensibilityType extension = new ExtensibilityType (); extension = AnyHelper.getExtensibility (requestElement); OGSIServiceGridLocator factoryService = new OGSIServiceGridLocator (); Factory factory = factoryService.getFactoryPort (new URL (source_url)); GridServiceFactory gridFactory = new GridServiceFactory (factory); LocatorType locator = gridFactory.createService (extension); System.out.println ("Created an instance of Multi-RFT"); MultiFileRFTDefinitionServiceGridLocator loc = new MultiFileRFTDefinitionServiceGridLocator(); RFTPortType rftPort = loc.getMultiFileRFTDefinitionPort (locator); ((Stub)rftPort)._setProperty (Constants.AUTHORIZATION, NoAuthorization.getInstance()); ((Stub)rftPort)._setProperty (GSIConstants.GSI_MODE, GSIConstants.GSI_MODE_FULL_DELEG); ((Stub)rftPort)._setProperty (Constants.GSI_SEC_CONV, Constants.SIGNATURE); ((Stub)rftPort)._setProperty (Constants.GRIM_POLICY_HANDLER, new IgnoreProxyPolicyHandler ()); int requestid = rftPort.start (); System.out.println ("Request id: " + requestid); } catch (Exception e) { System.err.println (MessageUtils.toString (e)); }

5. 23.06.2006GAT5 What GAT offers? (1) GAT is not a new grid middleware. GAT enables an easy to different grid infrastructures. GRAM Condor Unicore GridFTP... GAT is an OpenSource project.

6. 23.06.2006GAT6 What GAT offers? (2) Applications call the GAT-API for grid operations.  Applications hav to be linked against the GAT. Applications are independend of the underlying grid infrastructure.  GAT Engine loads available adaptors at runtime  During a calls of the GAT-API the GAT engine decides, which adaptor performs the grid operation.  In error case during a „grid operation“, another adaptor will be accessed. default adaptors offer local operations  grid applications can be compiled, linked and tested without available grid services.  The same application can run in a „complete grid environment“: without a new build.

7. 23.06.2006GAT7 What GAT offers? (3) The GAT-API doesn‘t change. Changes e.g. in Globus job submit are included in the GAT Globus adaptor. GAT offers reliability. If one grid service is not available, another grid service is used. GAT is much eaysier to install as Globus. GAT offers grid with a minimum effort for the end user.

8. 23.06.2006GAT8 What GAT not offers? GAT doesn‘t replace any function of a grid middleware. Without an adaptor for a grid middleware GAT is useless for this middleware. GAT has no resource broker.

9. 23.06.2006GAT9 Examples (Java-GAT) File operations Job submit

10. 23.06.2006GAT10 File copy: Java-GAT import org.gridlab.gat.*; import org.gridlab.gat.io.File; public class RemoteCopy { public static void main(String[] args) throws Exception { GATContext context = new GATContext(); URI src = new URI(args[0]); URI dest = new URI(args[1]); File file = GAT.createFile(context, src); // create file object file.copy(dest); // and copy it GAT.end(); }

11. 23.06.2006GAT11 File Write (Java-GAT) import java.io.PrintWriter; import org.gridlab.gat.GAT.*; import org.gridlab.gat.io.FileOutputStream; public class FileStreamSimple { public static void main(String[] args) { GATContext context = new GATContext(); URI src = null; FileOutputStream stream = null; PrintWriter p; src = new URI(args[0]); stream = GAT.createFileOutputStream(context, null, src); p = new PrintWriter(stream); String toBeStreamed = "hello world\n"; p.println(toBeStreamed); p.close(); GAT.end(); }

12. 23.06.2006GAT12 Job Submit (Java-GAT) public class SubmitJobToHost { public static void main(String[] args) { GATContext context = new GATContext(); Preferences prefs = new Preferences(); prefs.put("ResourceBroker.adaptor.name", "globus"); prefs.put("ResourceBroker.jobmanager", "sge"); URI exe = null; File outFile = null; File errFile = null; File pre1 = null; File pre1Dest = null; try { exe = new URI("file:////bin/hostname"); outFile = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl/out")); errFile = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl/err")); pre1 = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl//bin/echo")); pre1Dest = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl//home/rob/my_temp_file")); } catch (Exception e) { System.err.println("error: " + e); System.exit(1); } SoftwareDescription sd = new SoftwareDescription(); sd.setLocation(exe); sd.setStdout(outFile); sd.setStderr(errFile); sd.addPreStagedFile(pre1, pre1Dest); Hashtable hardwareAttributes = new Hashtable(); hardwareAttributes.put("machine.node", "fs0.das2.cs.vu.nl"); ResourceDescription rd = new HardwareResourceDescription( hardwareAttributes); JobDescription jd = null; ResourceBroker broker = null; try { jd = new JobDescription(sd, rd); broker = GAT.createResourceBroker(context, prefs); } catch (Exception e) { System.err.println("Could not create Job description: " + e); System.exit(1); } Job job = null; try { job = broker.submitJob(jd); } catch (Exception e) { System.err.println("submission failed: " + e); e.printStackTrace(); System.exit(1); }

13. 23.06.2006GAT13 GAT architecture API thin layer; only a frame. Adaptor implements “grid functionality” as “ordered” by the API. Engine mediates betwen API u. adaptor: the adaptors are called at runtime error tracing and “Fallback” (default local adaptors) CPI for Implementation of single adaptors.

14. 23.06.2006GAT14 GAT architecture GAT Adapter SGE PBS GTK4 Globus 2/3.x Unicore DRMAA Application Application layer GAT API GAT Engine GAT layer User Space „Grid“ Space

15. 23.06.2006GAT15 Implementation (Engine) C version fully implemented C++ wrapper fully implemented Python wrapper fully implemented JAVA version fully implemented

16. 23.06.2006GAT16 Implementation (Adaptors ) C-GAT Globus: gram, gridftp, RLS, gsiscp, gsissh Unicore: Job Submit, Job Monitoring DRMAA (Distributed Resource Management Application API) SGE (Sun Grid Engine) PBS (Portable Batch System)

17. 23.06.2006GAT17 Implementation (Adaptors ) Java-GAT Globus: With Java CoG Kit; Globus 3.y complete, GTK 4 up to now without WebServices. Unicore: In work SGE PBS

18. 23.06.2006GAT18 Implementation (Adaptors ) Java-GAT and C-GAT Local adpaptors: ssh, scp, ftp, sftp, File-Adapter: (cp, mv, read, write, etc...) Job-Adapter: fork, exec, auch über ssh...

19. 23.06.2006GAT19 File.copy: schematic run through File.copy(dest) FileCPI.copy(dest) Adapter1 copy Adapter2 copy Adapter3 copy

20. 23.06.2006GAT20 GAT API overview File operations (Monitoring and events) Resources, jobs (Utility classes: error handling, security, preferences) (Information exchange)

21. 23.06.2006GAT21 API sub systems File sub system GATFileGATFilestreamGATLogicalFile GATEndpointGATPipeListenerGATPipe Monitoring and event sub system GATRequestListenerGATRequestNotifierGATAction GATMetricListenerGATMetricGATMetricEvent Information exchange sub system GATAdvertisableGATAdvertService Resource management sub system GATSoftwareDescriptionGATResourceDescriptionGATResoure GATJobDescriptionGATResourceBrokerGATReservation GATJob Utility sub system GATSelfGATContextGATSecurityContext GATStatusGATPreferencesURL,Time,...

22. 23.06.2006GAT22 File sub system GATFile class GATFile Create Destroy Copy Move Delete IsReadable IsWritable GetLength LastWriteTime GetLocation GATObject

23. 23.06.2006GAT23 File sub system GATFileStream class GATFileStream Create Destroy Read Write Seek GATObject

24. 23.06.2006GAT24 File sub system GATLogicalFile class GATLogicalFile Create Destroy GetFiles GetLocations Remove RemoveFile AddFile AddLocation Replicate GATObject

25. 23.06.2006GAT25 Advert package Goal: Advertisement of important informations at a central place, e.g. job finished, job rc... Realized via an SQL adaptor in C-GAT.

26. 23.06.2006GAT26 Advert package GATAdvertService Add Delete AddMetadata GetMetadata GetAdertisable Find SetPWD GetPWD GATObject

27. 23.06.2006GAT27 Job management classes GATResourceBroker ReserveResource FindResources SubmitJob GATJob Unschedule CheckPoint CloneJob Migrate Stop GetJobDescription GetState GetJobID GetNativeID

28. 23.06.2006GAT28 Job management available job status GATJobState valuemeaning GATJobState UnknownStatus not retrievable GATJobState InitialAt initialization GATJobState Schedulede.g. queued GATJobState RunningJob is executing GATJobState StoppedJob read or stopped

29. 23.06.2006GAT29 Job management structures GATHardwareResourceDescription Machine type Memory CPU-Time Nodes needed GATSoftwareResourceDescription Executable Arguments Stdin Stdout Stderr Pre-Stage-Files Post-Stage-Files GATJobDescription

30. 23.06.2006GAT30 Job management example (PBS adaptor) GATHardwareResourceDescription machine.queue= destination (-q) jobname = jobname (-N) yeo = join (-j) memory.size = mem (-l) file.size = file (-l) cpu.walltime = walltime (-l) cpu.nodes = nodes (-l)

31. 23.06.2006GAT31 Job management example (PBS adaptor) GATSoftwareResourceDescription Stdin = stdin (exec < input) Stdout = stdout (-o) Stderr = stderr (-e) Executable = executable Arguments = arglist[]

32. 23.06.2006GAT32 Job management example (PBS adaptor) Realization in QSUB script machine.queue= new@url.org jobname= TestJob yeo= eo memory.size= 1G file.size= 2G cpu.walltime= 12:00:00 cpu.nodes= 8 stdin= input stdout= out.testjob stderr= err.testjob executable= /bin/prog arglist[0]= arg1 arglist[1]= arg2 arglist[2]= lastarg #PBS -q new@url.orgnew@url.org #PBS -l walltime=12:00:00, \ -mem=1G,file=2G,nodes=8@url.org #PBS -N testjob #PBS -o out.testjob #PBS -e err.testjob #PBS -j eo /bin/prog arg1 arg2 lastarg < input.

33. 23.06.2006GAT33 Job management lacks Problem: Defintions in HardwareResourceDescription quite arbitrarily. Solution: Orientation at the JSDL standard.

34. 23.06.2006GAT34 Job management lacks Problem: To less JOB status. Solution: Orientation at DRMAA.

35. 23.06.2006GAT35 GAT future C-GAT is now in maintenance mode Java-GAT is still further developed SAGA (Simple API for Grid Applications) is the upcoming GGF standard. First SAGA engine with GAT wrapper and GTK4 adaptor in autumn 2006

36. 23.06.2006GAT36 Usage example ProC MPA Garching

37. 23.06.2006GAT37 SAGA To be integrated: GAT, Java-CoG, DRMAA, RealityGrid, JSDL, GridRPC, OSGA-BES, GridCPR, gLite, HDF5 Developed by : GAT, RealityGrid UK Science, OMII Grid UK Science, CCT Louisana, VU Netherlands, NAREGI Japan, Globus/CoG, GGF DRMAA, GGF GridRPC Important: Please remarks needs on adaptors!

38. 23.06.2006GAT38 GAT users C-GAT MPA in Garching ProC-Workflows on the Grid LSU in Baton Rouge Chemistry project AMOLF NL Fourier Transform Mass Spectrometry (FTMS) Analyse. FTMS data are delivered to the Grid with JavaGAT (ssh, sftp, gridftp). Multimedian Project NL Start of parallel Jobs ZIB Berlin INRIA Frankreich Java-GAT SuperScalar (Univ. of Barcelona, ESP) SURA-Grid (63 partners, US) SCOOP project (LSU + 9 partners, US) UCOMS project (LSU + 4 partners, US) Cactus (LSU, US) Clusterix Project (PSNC, Univ. of Krakow, PL) Amolf (Vl-e, NL) Frank Seinstra (UvA, NL) Triana group (Cardiff, UK)

39. 23.06.2006GAT39 Links for GAT GAT overall: http://www.gridlab.org/WorkPackages/wp-1/ GAT CVS: cvs.gridlab.org cvsroot: pserver:readonly@cvs.gridlab.org:/cvs/gridlab Password: anon GAT sources: wp-1/Codes GAT documents: wp-1/Documents Only GATEngine: wp-1/Codes/GATEngine German GAT Mailing-Liste: GAT@d-grid.de (www.listserv.dfn.de)GAT@d-grid.de Download tarball: http://www.gridlab.org/WorkPackages/wp-1/gatreleases.html http://www.gridlab.org/WorkPackages/wp-1/adaptorreleases.html