Workflow Management System based on Service Oriented Components for Grid Applications. Ju-Ho Choi Korea University, Seoul, Rep. of Korea

Distributed Supercomputing Lab in Korea University Contents Introduction Previous Work Requirements Grid Workflow System Implementation Conclusion

Distributed Supercomputing Lab in Korea University Introduction Grid Workflow init Input Data Divide job job1 job2 Output Proc.. job3 job4 job5 job6 display control Coordinated execution of multiple applications Abstract activity Control/Data flow Composition of activities Unreliable, Distributed, heterogeneous, huge-scaled computational and data resources We need a PSE which provides a transparent view in the aspect of Grid user Requirements ?

Distributed Supercomputing Lab in Korea University Previous Work – Grid/PSE Grid Very Complicate !! Distributed, heterogeneous, unreliable Various hardware, software, different administration … Grid middleware : Globus, OGSA, … Distributed resource management, information service, data management, single sign-on, Unfamiliar User environment Now being Changing & Improved Platform : Virtual homogeneous environment Need a familiar, easy, and, transparent PSE for user User Environment : MPI, NetSolve, Nimrod/G, … Grid Portal, GridPort, CoG, … User domain application : Cactus, SCIRun, … Simple job requesttext-based output Hotpage, ASC portal … Simple job, Not easy for complex job Specific domain tool

Distributed Supercomputing Lab in Korea University Previous Work – Workflow Model Workflow patterns in Business Workflow  Basic Patterns AND-Split/Merge, OR-Split/Merge, XOR-Split/Merge  Other Patterns N out of M join, deferred choice, arbitrary cycle Previous Grid Workflow System  GWSL, Triana, GridAnt, myGrid, Petrinet-based Grid Workflow (FhRG)  Execution of multiple App. on multiple resources  Composing of the existing applications  Providing Graphic User Interfaces  Simple Link element for control flow Common patterns : AND, XOR, LOOP arbitrary cycle (Petrinet-based)  Data link Node for File transfer

Distributed Supercomputing Lab in Korea University Requirements Transparency with abstract workflow model  Complicate Grid infrastructure to user  Various Grid middleware, their change and standardization  Execution of the existing applications and making a new application without suffering from the details of Grid  Parallel/distributed programming, data transfer, fault handling Advanced distributed/parallel workflow pattern  Previous patterns are too simple to describe Grid applications Insufficient expressiveness of simple link elements  New patterns to support various parallelism for easy parallel and distributed programming  Integration of Control and Data flow

Distributed Supercomputing Lab in Korea University Requirements Fault handling method  Faults from unreliable Grid resources  Three categories of fault Hardware fault : network or host problems Software fault : illegal access, user defined error, … Resource utilization fault  Allocation or reservation problem without hardware fault caused by dynamic property of Grid Efficient execution of workflow  Static/Dynamic information and Prediction CPU speed, memory size, file system size, … Load, available memory, usable bandwidth, …  Current Insufficient and unsatisfying dynamic information  semi-automatically or manually adapting dynamic Grid Fault handling, dynamic reallocation, runtime modification

Distributed Supercomputing Lab in Korea University Grid Workflow : Architecture Grid Workflow Controller Realizer Generallizer Resource Mapper A workflow instance A workflow instance Workflow instance Activity Controller GUI Editing & running workflows Information Service Resource management File transfer Abstract workflow Activity Listener Allocator Monitor Data Transfer Manager Activity Activity Repository GWDL (Grid Workflow Description Language) Parser Dynamic Manager Workflow Engine

Distributed Supercomputing Lab in Korea University Grid Workflow : Abstract Workflow EUserDefinedB AC EHardwareCrash EUserDefinedA B2B2 alloc T Data 1 Data 3 B S S Data 2 A start Data 1 A end B start Data 3 B endC endC start (a) (b) Grid Workflow  Activity  Control/ Data flow  Resource  exception Activity  Function  Input/ output data  Exception

Distributed Supercomputing Lab in Korea University GWDL Grid Workflow Description Language Data Model Activity Model Flow Model Resource Model Control flow Data flow Exception handling Predefined Activity allocation notifytimer Nodes Activity Node Group Node Resource specification Execution environment Exception type Data type Life cycle NotAvailable Allocation UserDefined Memory Storage NotExistData Function Loose-Coupled Tight-Coupled User defined Activity Input/output Data Exception Fail Time

Distributed Supercomputing Lab in Korea University Grid Workflow : Exceptions Predefined Exceptions  Efail : unknown Fail  ENotAvaliable : hardware fault  EAllocation : allocation problem  ETimeExpired : timer  ENoData : no usable input or output data  EMemory : insufficient memory  EStorage : insufficient storage User defined Exceptions  For notification of software fault

Distributed Supercomputing Lab in Korea University Grid Workflow : Patterns (a) Sequence (b) XOR-Split (c) XOR-Join (d) Loop (e) Multi-Split (f) Multi-Join (g) AND-Split (h) AND-Join (i) AND-Loop symbol (i') AND-Loop description (j) Queue (k) Wait (l) Node copy (m) Explicit termination (n) Pause/Continue (o) Group node

Distributed Supercomputing Lab in Korea University Workflow Model: Patterns Composition of basic patterns  Pipeline W Q n C n C m Q n

Distributed Supercomputing Lab in Korea University Workflow Model: Patterns Composition of basic patterns  Static/Dynamic Data distribution Static: AND-Split and AND-Join Dynamic: AND-Loop, Multi-Split and Queue M Q A … A … staticdynamic

Distributed Supercomputing Lab in Korea University Grid Workflow : Workflow Execution Creating Concrete workflow  Mapping activities to resources Statically or dynamically Automatically or manually  Activity parameters LC (Loose-coupled) type  For the existing applications  Executable file, addresses for input/output file transfer TC (Tight-coupled) type  For making new application  Executable file, identification number  Addresses for Input/output data: file, variables  Address for communication with workflow engine Runtime Modification  Pause and continue a workflow instance  To handle fault, unexpected situation, or manual reallocation.

Distributed Supercomputing Lab in Korea University Implementation Our prototype  Workflow engine and graphic workflow editor : Java XML parsing : Apache Xerces  TC type activity : C++ Making TC type activity template codes: Apache Xalan  Fault detection Throwing user defined exception State notifications of TC type activity Polling with Java CoG’s GRAM.ping() function or Timeout  Grid middleware Globus Toolkit 3.2 : GRAM, MDS, GridFTP  Client library of Grid Java CoG (Commodity of Grid)

Distributed Supercomputing Lab in Korea University Conclusion and Future work Analysis of Requirements for Grid Applications We proposed a new Grid workflow system  Abstract workflow model  Advanced workflow pattern for parallelism  Exception for handling fault on Grid  Run time modification Future works  Scalability experiment of our Grid workflow system  Appling our system to various Grid applications

Distributed Supercomputing Lab in Korea University Thanks for your attention