Using the GAT (GAT the semester project?) :-P Hartmut Kaiser

October, 17th 2005Using the GAT2 Semester Project Problem –You have an application which has to be run 1000 times with different input data –You have the 1000 different input files –You want to get back 1000 result files You want to distribute this to all available machines The data transfer should be as effective as possible

October, 17th 2005Using the GAT3 Semester Project - Roadmap Requirements analysis Problem decomposition Interface analysis System architecture design Tool selection Goal setting We will do that for the whole system today You‘ll be expected to do that for your subproject in the coursework

October, 17th 2005Using the GAT4 Requirements analysis Overall application needs to be able to –Discover and select resources and services –Copy files reliable using existing services –Submit jobs to selected resources –Decide about parameters for file transfer –Decide about how to distribute jobs to available resources –Figure out when to bring back result files –Correct things when something fails –Keep track what‘s going on –Be independent from the Grid architecture

October, 17th 2005Using the GAT5 Problem decomposition Service/resource discovery –Using information repositories -> MDS, IGRID, GPIR, local configuration files? Service/resource selection –Using which criteria? –Match with application requirements File transfer –Protocol/service selection, parameter selection, optimisation Job submission –Selection of method to use Information gathering –Notifications, polling?

October, 17th 2005Using the GAT6 Interface analysis Service/resource description –What information available/required? Job description –What needs to be specified? Notification description (do we need them at all?) –What notifications need to be defined?

October, 17th 2005Using the GAT7 Tool selection Libraries to use, depends on –Availability of libraries –Availability of services Programming language –C, C++ or Python Target operating system(s) –Portability Compilation and debug environments (compiler, debugger, tracing/logging libraries)

October, 17th 2005Using the GAT8 Libraries/Services to use Libraries –Database module –Web-service access –Logging of application activity Services –GT4 RFT –GPIR information service –Globus GRAM

October, 17th 2005Using the GAT9 Programming Language Since you want to use the GAT you have 3 alternatives C (+) Easy to learn and use (+) Lots of GAT examples (+) Complete documentation of C GAT API available (-) Low level, Error prone (-) Large code overhead for adaptor administration (-) Bad support for webservice access C++ (+) Very easy adaptor writing (+) Lots of GAT examples (+) Good library support (data structures as list, map etc.) (+/-) More complex to learn and use (-) Incomplete documentation of the C++ GAT API (-) Bad support for webservice access Python (+) Easy to learn and use (+) Very easy adaptor writing (+) Very good support for webservice access (-) Less GAT examples (-) Incomplete documentation of the Python GAT API

October, 17th 2005Using the GAT10 Target operating system(s) Linux (*nix) Mac OS (Darwin) Windows (unfortunately currently not fully supported)

October, 17th 2005Using the GAT11 Goal setting Create a Task farming manager –Usable for a class of applications: Single executable, non-MPI Reading all input from one or more files Producing one or more files in the output –Distribute tasks evenly on available resources –Stage in/out input and output files as effectively as possible (by selecting the appropriate transfer protcoll and parameters) –Keep track on the current status of all tasks, so the TFM may be restarted (no need to run it all the time)

October, 17th 2005Using the GAT12 System architecture design GAT Python/C++/C adaptor (resource and service discovery) Python/C++/C adaptor (file transfer) C adaptor (job submission) Task farming manager (Python/C++/C) Supporting libraries GT4 RFT ServiceGPIR ServiceGRAM Service

October, 17th 2005Using the GAT13 Example: Prequisites (Python) Python V2.4.3 – GAT (V1.7.1), GAT Python wrapper (V1.5.1) – SOAPpy (V0.11.6) – webserice bindings – GPIR and GT4 RFT running somewhere – (GPIR) – (GT4 RFT)

October, 17th 2005Using the GAT14 Submodules, project groups Group 1: Task farming manager –Application itself –Strategies of job distribution -> Cornelius, Krishnendu, Chirag Group 2: Resource discovery –Adaptor to connect to GPIR –Defining metadata for service/resource information -> Srikanth, Ibrahim, Rakesh Group 3: File transfer –Adaptor to connect to GT4 RFT –Optimisation of transfer parameters -> Memet, Santiago, Teresa

October, 17th 2005Using the GAT15 Semester Project -> Write a Grid based task farming manager using the GAT –Discover available resources to run jobs on Grid Information Service (GPIR) –Discover available file transfer and job submission services Grid Information Service (GPIR) –Stage input files, get back result files File Transfer Services (GT4 RFT) –Submit jobs Job Submission Service (GRAM) -> Write a GAT adaptor to connect to GPIR -> Write a GAT adaptor to use GT4 RFT ->Create your own webservice to receive notifications about job status?

October, 17th 2005Using the GAT16 Coursework Every of the project groups is expected to write a paper about their submodule containing: Elevator pitch Defining requirements, setting goals etc. Architecture diagram and description of the subprojects to deliver, containing submodules, their interfaces etc. (Due Wednesday October 26th)

October, 17th 2005Using the GAT17 The Elevator Pitch Write a short statement describing what your software will do, and why –Short is better than long –Functionality trumps technology (what and not how) –Solve a problem (explain the initial problem) –Pitch benefits not yourself –Figure out, what‘s important to your audience, address excatly that (cost, innovation, features, compatibility, or something else entirely).

October, 17th 2005Using the GAT18 Defining requirements Business requirements –High level objectives of the organisation or customer who requests the system User requirements –User goals or tasks that the users must be able to perform Functional requirements –Software functionality that must be built into the product to accomplish their tasks System requirements –Top-level requirements for the product