GAT1 Alexander Beck-Ratzka OFFIS,

GAT2 GAT: Agenda Introduction –Why a new Grid-API? –What GAT offers, what not? –Examples –Architecture The single API groups –File (-stream) management, logical file management, –AdvertService management, –Job management Adaptors Conclusion

GAT3 Why a new Grid-API? GAT as a uniform API for the access to heterogeneous grid technologies / grid middleware. GAT is only a framework; the real grid operationen must be done by adaptors. GAT offers the possibility for accessing adpaptors. New grid technologies only needs to be coupled to GAT via adaptors -> No changes in the source sode necessary, even not If using new grid technologies. GAT enables a simple access to grid technologies. GAT uses the grid middleware, which is available at the moment, where is should be used, and this by only typing a single statement for GAT_JobSubmit or GATFileCopy...

GAT4 File copy: CoG/RFT package org.globus.ogsa.gui; import java.io.BufferedReader; import java.io.File; import java.io.FileReader; import java.net.URL; import java.util.Date; import java.util.Vector; import javax.xml.rpc.Stub; import org.apache.axis.message.MessageElement; import org.apache.axis.utils.XMLUtils; import org.globus.* import org.gridforum.ogsi.* import org.gridforum.ogsi.holders.TerminationTimeTypeHolder; import org.w3c.dom.Document; import org.w3c.dom.Element; public class RFTClient { public static void copy (String source_url, String target_url) { try { File requestFile = new File (source_url); BufferedReader reader = null; try { reader = new BufferedReader (new FileReader (requestFile)); } catch (java.io.FileNotFoundException fnfe) { } Vector requestData = new Vector (); requestData.add (target_url); TransferType[] transfers1 = new TransferType[transferCount]; RFTOptionsType multirftOptions = new RFTOptionsType (); multirftOptions.setBinary (Boolean.valueOf ( (String)requestData.elementAt (0)).booleanValue ()); multirftOptions.setBlockSize (Integer.valueOf ( (String)requestData.elementAt (1)).intValue ()); multirftOptions.setTcpBufferSize (Integer.valueOf ( (String)requestData.elementAt (2)).intValue ()); multirftOptions.setNotpt (Boolean.valueOf ( (String)requestData.elementAt (3)).booleanValue ()); multirftOptions.setParallelStreams (Integer.valueOf ( (String)requestData.elementAt (4)).intValue ()); multirftOptions.setDcau(Boolean.valueOf( (String)requestData.elementAt (5)).booleanValue ()); int i = 7; for (int j = 0; j < transfers1.length; j++) { transfers1[j] = new TransferType (); transfers1[j].setTransferId (j); transfers1[j].setSourceUrl ((String)requestData.elementAt (i++)); transfers1[j].setDestinationUrl ((String)requestData.elementAt (i++)); transfers1[j].setRftOptions (multirftOptions); } TransferRequestType transferRequest = new TransferRequestType (); transferRequest.setTransferArray (transfers1); int concurrency = Integer.valueOf ((String)requestData.elementAt(6)).intValue(); if (concurrency > transfers1.length) { System.out.println ("Concurrency should be less than the number" "of transfers in the request"); System.exit (0); } transferRequest.setConcurrency (concurrency); TransferRequestElement requestElement = new TransferRequestElement (); requestElement.setTransferRequest (transferRequest); ExtensibilityType extension = new ExtensibilityType (); extension = AnyHelper.getExtensibility (requestElement); OGSIServiceGridLocator factoryService = new OGSIServiceGridLocator (); Factory factory = factoryService.getFactoryPort (new URL (source_url)); GridServiceFactory gridFactory = new GridServiceFactory (factory); LocatorType locator = gridFactory.createService (extension); System.out.println ("Created an instance of Multi-RFT"); MultiFileRFTDefinitionServiceGridLocator loc = new MultiFileRFTDefinitionServiceGridLocator(); RFTPortType rftPort = loc.getMultiFileRFTDefinitionPort (locator); ((Stub)rftPort)._setProperty (Constants.AUTHORIZATION, NoAuthorization.getInstance()); ((Stub)rftPort)._setProperty (GSIConstants.GSI_MODE, GSIConstants.GSI_MODE_FULL_DELEG); ((Stub)rftPort)._setProperty (Constants.GSI_SEC_CONV, Constants.SIGNATURE); ((Stub)rftPort)._setProperty (Constants.GRIM_POLICY_HANDLER, new IgnoreProxyPolicyHandler ()); int requestid = rftPort.start (); System.out.println ("Request id: " + requestid); } catch (Exception e) { System.err.println (MessageUtils.toString (e)); }

GAT5 What GAT offers? (1) GAT is not a new grid middleware. GAT enables an easy to different grid infrastructures. GRAM Condor Unicore GridFTP... GAT is an OpenSource project.

GAT6 What GAT offers? (2) Applications call the GAT-API for grid operations.  Applications hav to be linked against the GAT. Applications are independend of the underlying grid infrastructure.  GAT Engine loads available adaptors at runtime  During a calls of the GAT-API the GAT engine decides, which adaptor performs the grid operation.  In error case during a „grid operation“, another adaptor will be accessed. default adaptors offer local operations  grid applications can be compiled, linked and tested without available grid services.  The same application can run in a „complete grid environment“: without a new build.

GAT7 What GAT offers? (3) The GAT-API doesn‘t change. Changes e.g. in Globus job submit are included in the GAT Globus adaptor. GAT offers reliability. If one grid service is not available, another grid service is used. GAT is much eaysier to install as Globus. GAT offers grid with a minimum effort for the end user.

GAT8 What GAT not offers? GAT doesn‘t replace any function of a grid middleware. Without an adaptor for a grid middleware GAT is useless for this middleware. GAT has no resource broker.

GAT9 Examples (Java-GAT) File operations Job submit

GAT10 File copy: Java-GAT import org.gridlab.gat.*; import org.gridlab.gat.io.File; public class RemoteCopy { public static void main(String[] args) throws Exception { GATContext context = new GATContext(); URI src = new URI(args[0]); URI dest = new URI(args[1]); File file = GAT.createFile(context, src); // create file object file.copy(dest); // and copy it GAT.end(); }

GAT11 File Write (Java-GAT) import java.io.PrintWriter; import org.gridlab.gat.GAT.*; import org.gridlab.gat.io.FileOutputStream; public class FileStreamSimple { public static void main(String[] args) { GATContext context = new GATContext(); URI src = null; FileOutputStream stream = null; PrintWriter p; src = new URI(args[0]); stream = GAT.createFileOutputStream(context, null, src); p = new PrintWriter(stream); String toBeStreamed = "hello world\n"; p.println(toBeStreamed); p.close(); GAT.end(); }

GAT12 Job Submit (Java-GAT) public class SubmitJobToHost { public static void main(String[] args) { GATContext context = new GATContext(); Preferences prefs = new Preferences(); prefs.put("ResourceBroker.adaptor.name", "globus"); prefs.put("ResourceBroker.jobmanager", "sge"); URI exe = null; File outFile = null; File errFile = null; File pre1 = null; File pre1Dest = null; try { exe = new URI("file:////bin/hostname"); outFile = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl/out")); errFile = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl/err")); pre1 = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl//bin/echo")); pre1Dest = GAT.createFile(context, prefs, new URI( "any://fs0.das2.cs.vu.nl//home/rob/my_temp_file")); } catch (Exception e) { System.err.println("error: " + e); System.exit(1); } SoftwareDescription sd = new SoftwareDescription(); sd.setLocation(exe); sd.setStdout(outFile); sd.setStderr(errFile); sd.addPreStagedFile(pre1, pre1Dest); Hashtable hardwareAttributes = new Hashtable(); hardwareAttributes.put("machine.node", "fs0.das2.cs.vu.nl"); ResourceDescription rd = new HardwareResourceDescription( hardwareAttributes); JobDescription jd = null; ResourceBroker broker = null; try { jd = new JobDescription(sd, rd); broker = GAT.createResourceBroker(context, prefs); } catch (Exception e) { System.err.println("Could not create Job description: " + e); System.exit(1); } Job job = null; try { job = broker.submitJob(jd); } catch (Exception e) { System.err.println("submission failed: " + e); e.printStackTrace(); System.exit(1); }

GAT13 GAT architecture API thin layer; only a frame. Adaptor implements “grid functionality” as “ordered” by the API. Engine mediates betwen API u. adaptor: the adaptors are called at runtime error tracing and “Fallback” (default local adaptors) CPI for Implementation of single adaptors.

GAT14 GAT architecture GAT Adapter SGE PBS GTK4 Globus 2/3.x Unicore DRMAA Application Application layer GAT API GAT Engine GAT layer User Space „Grid“ Space

GAT15 Implementation (Engine) C version fully implemented C++ wrapper fully implemented Python wrapper fully implemented JAVA version fully implemented

GAT16 Implementation (Adaptors ) C-GAT Globus: gram, gridftp, RLS, gsiscp, gsissh Unicore: Job Submit, Job Monitoring DRMAA (Distributed Resource Management Application API) SGE (Sun Grid Engine) PBS (Portable Batch System)

GAT17 Implementation (Adaptors ) Java-GAT Globus: With Java CoG Kit; Globus 3.y complete, GTK 4 up to now without WebServices. Unicore: In work SGE PBS

GAT18 Implementation (Adaptors ) Java-GAT and C-GAT Local adpaptors: ssh, scp, ftp, sftp, File-Adapter: (cp, mv, read, write, etc...) Job-Adapter: fork, exec, auch über ssh...

GAT19 File.copy: schematic run through File.copy(dest) FileCPI.copy(dest) Adapter1 copy Adapter2 copy Adapter3 copy

GAT20 GAT API overview File operations (Monitoring and events) Resources, jobs (Utility classes: error handling, security, preferences) (Information exchange)

GAT21 API sub systems File sub system GATFileGATFilestreamGATLogicalFile GATEndpointGATPipeListenerGATPipe Monitoring and event sub system GATRequestListenerGATRequestNotifierGATAction GATMetricListenerGATMetricGATMetricEvent Information exchange sub system GATAdvertisableGATAdvertService Resource management sub system GATSoftwareDescriptionGATResourceDescriptionGATResoure GATJobDescriptionGATResourceBrokerGATReservation GATJob Utility sub system GATSelfGATContextGATSecurityContext GATStatusGATPreferencesURL,Time,...

GAT22 File sub system GATFile class GATFile Create Destroy Copy Move Delete IsReadable IsWritable GetLength LastWriteTime GetLocation GATObject

GAT23 File sub system GATFileStream class GATFileStream Create Destroy Read Write Seek GATObject

GAT24 File sub system GATLogicalFile class GATLogicalFile Create Destroy GetFiles GetLocations Remove RemoveFile AddFile AddLocation Replicate GATObject

GAT25 Advert package Goal: Advertisement of important informations at a central place, e.g. job finished, job rc... Realized via an SQL adaptor in C-GAT.

GAT26 Advert package GATAdvertService Add Delete AddMetadata GetMetadata GetAdertisable Find SetPWD GetPWD GATObject

GAT27 Job management classes GATResourceBroker ReserveResource FindResources SubmitJob GATJob Unschedule CheckPoint CloneJob Migrate Stop GetJobDescription GetState GetJobID GetNativeID

GAT28 Job management available job status GATJobState valuemeaning GATJobState UnknownStatus not retrievable GATJobState InitialAt initialization GATJobState Schedulede.g. queued GATJobState RunningJob is executing GATJobState StoppedJob read or stopped

GAT29 Job management structures GATHardwareResourceDescription Machine type Memory CPU-Time Nodes needed GATSoftwareResourceDescription Executable Arguments Stdin Stdout Stderr Pre-Stage-Files Post-Stage-Files GATJobDescription

GAT30 Job management example (PBS adaptor) GATHardwareResourceDescription machine.queue= destination (-q) jobname = jobname (-N) yeo = join (-j) memory.size = mem (-l) file.size = file (-l) cpu.walltime = walltime (-l) cpu.nodes = nodes (-l)

GAT31 Job management example (PBS adaptor) GATSoftwareResourceDescription Stdin = stdin (exec < input) Stdout = stdout (-o) Stderr = stderr (-e) Executable = executable Arguments = arglist[]

GAT32 Job management example (PBS adaptor) Realization in QSUB script machine.queue= jobname= TestJob yeo= eo memory.size= 1G file.size= 2G cpu.walltime= 12:00:00 cpu.nodes= 8 stdin= input stdout= out.testjob stderr= err.testjob executable= /bin/prog arglist[0]= arg1 arglist[1]= arg2 arglist[2]= lastarg #PBS -q #PBS -l walltime=12:00:00, \ #PBS -N testjob #PBS -o out.testjob #PBS -e err.testjob #PBS -j eo /bin/prog arg1 arg2 lastarg < input.

GAT33 Job management lacks Problem: Defintions in HardwareResourceDescription quite arbitrarily. Solution: Orientation at the JSDL standard.

GAT34 Job management lacks Problem: To less JOB status. Solution: Orientation at DRMAA.

GAT35 GAT future C-GAT is now in maintenance mode Java-GAT is still further developed SAGA (Simple API for Grid Applications) is the upcoming GGF standard. First SAGA engine with GAT wrapper and GTK4 adaptor in autumn 2006

GAT36 Usage example ProC MPA Garching

GAT37 SAGA To be integrated: GAT, Java-CoG, DRMAA, RealityGrid, JSDL, GridRPC, OSGA-BES, GridCPR, gLite, HDF5 Developed by : GAT, RealityGrid UK Science, OMII Grid UK Science, CCT Louisana, VU Netherlands, NAREGI Japan, Globus/CoG, GGF DRMAA, GGF GridRPC Important: Please remarks needs on adaptors!

GAT38 GAT users C-GAT MPA in Garching ProC-Workflows on the Grid LSU in Baton Rouge Chemistry project AMOLF NL Fourier Transform Mass Spectrometry (FTMS) Analyse. FTMS data are delivered to the Grid with JavaGAT (ssh, sftp, gridftp). Multimedian Project NL Start of parallel Jobs ZIB Berlin INRIA Frankreich Java-GAT SuperScalar (Univ. of Barcelona, ESP) SURA-Grid (63 partners, US) SCOOP project (LSU + 9 partners, US) UCOMS project (LSU + 4 partners, US) Cactus (LSU, US) Clusterix Project (PSNC, Univ. of Krakow, PL) Amolf (Vl-e, NL) Frank Seinstra (UvA, NL) Triana group (Cardiff, UK)

GAT39 Links for GAT GAT overall: GAT CVS: cvs.gridlab.org cvsroot: Password: anon GAT sources: wp-1/Codes GAT documents: wp-1/Documents Only GATEngine: wp-1/Codes/GATEngine German GAT Mailing-Liste: Download tarball: