Triana Dr Matthew Shields, School of Computer Science, Cardiff University A Problem Solving Environment and Grid Workflow Tool

Course Outline Introduction History and Overview Tools and Toolboxes, Components and Groups How to compose a workflow Triana as a workflow editor

Course Outline cont. Service-oriented Components Web services as components Web service workflow Peer 2 Peer services as components Distributed service workflow Grid-oriented Components Grid file and job primitives as components Complex Grid workflow

Install Triana Unpack the triana.tar.gz to a location of your choice Set env $TRIANA (Unix) or %TRIANA% (Windows) to the unpacked directory Run $TRIANA/bin/triana (Unix) or %TRIANA/bin/triana.bat (Windows) Course website

About Triana Distributed Problem Solving Environment Workflow Based Multiple Component Domains Signal Processing (original domain) Image Processing Text Processing Audio Composite Components

Our Research Motivation Based on research within the Triana project: Triana - a vehicle for investigating research into distributed systems a workflow-based Problem Solving Environment Goals: To make the access of distributed services as seamless as possible To connect heterogeneous Grids To abstract the core capabilities needed for service-based computing (in P2P, Web services or Grid Computing)

Research Scope Triana is currently used in many projects GridOneD - Radio Astronomy DART - Distributed Audio Retrieval with Triana WHiP - Workflows Hosted in Portals Data-Mining Grid BioDiversity World See

Components and Definitions Component is unit of execution Components are defined in XML files: Naming information Input and output ports Parameter information Why Components? To simplify the application design process and to speed up application development

Triana Workflow Triana is inherently flow based Data flow - data arriving at component triggers execution Control flow - control commands trigger execution Decentralised execution Data or Control messages sent along communication “pipes” from sender to receiver causes receiver to execute Synchronous or Asynchronous messaging (Implementation dependant) Multiple inputs can block or trigger immediately (Component designer defined)

Taskgraph Internal object based workflow graph representation Taskgraph - DAG Tasks Connections External XML representation Simple XML syntax List of participating Task definitions Parent/Child connection Hierarchical (Compound components)

Simple Workflow Example Try It! $TRIANA/bin/triana

Example Continued Image is converted to Grey scale and then thresholded (set image less than 136 to black and everything higher to white). difference is then taken between this image and a image where the white lines are thinned out (i.e. detail is taken away). The resulting image is basically the detail of the white parts which were pruned by the ShrinkWhite unit which gives an outline of an image. Group the unit and save it as a new compound component

Coalescing Binary Search GEO 600 Coalescing Binary Search Algorithm implemented as a Triana workflow

Triana, the GAT and the GAP P2PSJXTA Web Services GAP Interface UDDI SOAP P2PS Discovery P2PS Pipes JXTA Discovery JXTA Pipes GAT Interface Condor Globus RLS Unicore PBS GridLab GRMS SGESSH WSRF LDR.NET Other.. GridFTP Grid Computing: Job Submission, File services A Graphical Grid Computing Environment or Portal Service Based Computing: Deployment, discovery and communication with distributed services e.g. P2P and (GSI) Web services Grid services

GAP Interface Motivation by GAT (GridLab.org) A Simple Service based API, for Service Deployment, Service Discovery Pipe Based Communication Static application interface with multiple middleware bindings P2PS (name…?) JXTA Web services P2PSJXTA Web Services GAP Interface UDDI SOAP P2PS Discovery P2PS Pipes JXTA Discovery JXTA Pipes

Service Oriented Comms network babelfish. altavista. com BabelFish en_fr hello bonjour Service Discovery Dynamic? Decentralized? Communication Message Format SOAP? Transport Protocol TCP? UDP? GAP

GAP Interface Simple Java API Peer-to-Peer Discovery Pipe-Based Communication Multiple Bindings JXTA P2PS Web Services (WSPeer)

P2P GAP Bindings Two Main Bindings Jxta legacy implementation, due to upgrade soon… P2PS Simple (lightweight) P2P System Uses in Triana, NRL (sensor nets) and Simulations (within NS- 2) Useful in highly dynamic environments Based on some Jxta principles i.e. pipes, rendezvous nodes, dynamic discovery, etc Capable of building small-world networks (centralised- decentralised)

P2PS Overview Discovery Service Broadcast/Locate Adverts + Queries in Discovery Subnet Pipe Service Connect Pipes using Endpoint Resolvers Rendezvous Service Send/Receive Adverts + Queries from known Rendezvous Peers Resolvers Pluggable Network Interfaces For Ns2 simulation environment TCP Resolver??? Resolver Discovery Service Rendezvous Service P2PS Peer Pipe Service UDP Resolver Network UDPTCP???

Service Creation, Discovery, and invocation Drag a Audio.Output.Play unit to a new scratchpad Right click on the Play unit and select create service Select Local Service [P2PS] and press Create Leave the default settings and click ok Use menu File -> New to create a new scratchpad Select menu Services -> Discover Services Type Play in the search name panel and click ok You should see a number of remote Play units Drag one of the remote Play units to the new scratch pad Drag a Audio.Input.LoadSound unit and connect to the play Download a sound file from the tutorial site Double click the LoadSound unit or right click and select properties, choose the sound file, click ok, click the play button.

Service Deployment Peer-to-peer services Distributing Audio !!! Generic example - can try and distribute using other networks Uses multicast or rendezvous to discover services Shows the basic interface to distribution Can publish compound/group componnents Publish as Web services Need a running UDDI server for service publication and discovery

WSPeer High Level Interface to Web Services Discovery Invocation Deployment Hosting Abstract from usual Web Service Discovery and Communication Mechanisms (i.e. UDDI and HTTP) P2PS Web Service Discovery? Uses Apache AXIS as SOAP Engine Extends Capabilities of Apache AXIS Stubless Invocation (including complex types) Non Standard Transports (i.e. P2PS)

WSPeer WSPeer – P2PS Application WSPeer – HTTP/UDDI deploypublishlocateinvoke UDDI HTTP Server deploy launch server publishlocate invoke deploy publish locate invoke

Web services Connecting Web services and integrating with other Triana units Xmethods Services -> Import Services Bible Verses Drag the readBible service to a new scratch pad Connect a Common.String.StringGen and a Common.String.StringViewer component Double click StringGen for the input window Psalm 119:105 Matthew 5:3-12,44-48 Matthew 5:3-12;John 1:1-5 Press Play, double click StringViewer to see the output Web Service Choreography

Multiple Services Xmethods - Services -> Import Services Urdu Translator Drag the Translate service to the scratch pad Drag a Common.Control.Duplicator unit the scratch pad Connect readBible to Duplicator connect one output from Duplicator to the StringViewer Connect the other output to the Translator service Drag a new StringViewer to the scratch pad and connect the output of the Translator to that Press play and double click the string viewers Try different inputs for StringGen

Web Services Resource Framework WSRF defines conventions for accessing stateful resources (WS-Resources) via Web service interactions Resources are refered to using EPR in message headers In Triana we refer to EPR/WS-addresses as a “context” for extendibility, i.e. WS-Security context Example using Globus 4 CounterService example Import erService?wsdl

The link between context and WSTypeGen is a “trigger” node to create a control dependency Right click on WSTypeGen1, select Node Editor Select “In Params” tab, click add, select “Trigger node only” select ok Double click the WSTypeGen unit, type a number into the “int” field To set the context on the Add service, double click or right click and select “Set Context” Press play Drag another WSTypeGen unit and Add service onto the scratch pad, set another integer and play again

GridLab GAT Application-level API (Gridlab.org) Abstract ‘Common’ Capabilities required by Grid Applications Resource Discovery Job Submission File Transfer Capabilities extracted through a number of Grid Application Scenarios. Makes life as a Grid programmer easier … Motivation for the GGF SAGA research group (Simple API for Grid Applications)

Using the GAT GATContext context = new GATContext(); Preferences prefs = new Preferences(); prefs.put("ResourceBroker.adaptor.name", "grms"); URI exe = new URI("file:////bin/hostname"); File outFile = GAT.createFile(context, prefs, new URI("file:///out") SoftwareDescription sd = new SoftwareDescription(); sd.setLocation(exe); sd.setStdout(outFile); ResourceDescription rd = new HardwareResourceDescription(); rd.put("hostname", "bouscat.cs.cf.ac.uk"); JobDescription jd = new JobDescription(sd, rd); ResourceBroker broker = GAT.createResourceBroker(context, prefs); Job job = broker.submitJob(jd);

Visual GAT GAT Primitives represented as Workflow Objects Job Tasks File Tasks Grid Data Flow implicit in Workflow Connections File Task -> File Task = Data Transfer File Task -> Job Task = Pre-Stage File File Task -> Local Task = File Read Grid Operations handled by GAT Adaptors Grid File -> Grid File = GridFTP HTTP File -> Local Task = HTTP Grid File -> GRMS Job = GRMS Pre-Stage

Grid Workflow in Triana Local Java Unit File (Local/Remote) Temporary File Job (Local/Remote) File WritePre-StagePost-StageFile Read

Visual GAT Alternative Grid Operations Expressed Explicitly GridFTP Tasks File Write Tasks File Read Tasks Not Intuitive Not Future Proof

File Input (HTTP) GridLab GAT File CPI GridFTP Adaptor HTTP Adaptor Data Movement Resource Broker CPI Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor www. gridlab. org Triana WaveToy.par

GridLab GAT File CPI GridFTP Adaptor HTTP Adaptor Data Movement Resource Broker CPI Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor www. gridlab. org Testbed Triana File Staging (GridFTP) WaveToy.par

Mercury GridLab GAT File CPI GridFTP Adaptor HTTP Adaptor Data Movement Resource Broker CPI Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor www. gridlab. org Testbed GRMS Data Mvment Testbed Triana Delphoi GAS iGrid Job Submission 1 WaveToy.par Cactus

GridLab GAT File CPI GridFTP Adaptor HTTP Adaptor Data Movement Resource Broker CPI Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor www. gridlab. org Testbed GRMS Data Mvment Portal Triana GAS Job Submission 2 Testbed Cactus Instant Mess MercuryDelphoi iGrid

GridLab GAT File CPI GridFTP Adaptor HTTP Adaptor Data Movement Resource Broker CPI Local Adaptor GRMS Adaptor GRAM Adaptor Local Adaptor www. gridlab. org Testbed Portal Triana Job Submission (GRAM) Testbed Cactus Instant Mess GRAM

GridMonSteer User Side Grid Side WSPeer User Interface Web Service GridMonSteer (gms) cactus phi_xy.jpg phi_yz.jpg submit Application (e.g. Triana) run monitor invoke using SOAP Resource Broker (e.g. GRMS) gms cactus phi_xy.jpgphi_yz.jpg rundir

Triana Grid Workflow Summary GridLab GAT Application-level API Abstracts Common Capabilities required by Grid Applications such as Triana Visual GAT GAT Primitives represented in Triana as Workflow Objects Grid Data Flow implicit in Workflow Connections GridMonSteer Generic Architecture for Monitoring and Steering Legacy Applications

Future Java GAT 2.0 (coming soon Vrije Universiteit) globus-4 web services, a windows shares adaptor (SAMBA/CIFS), an initial set of glite adaptors Mrs DART - Distributed Audio Retrieval with Triana Distribute music analysis workflows to P2P network to analyse user digital audio collections and make content based recommendations Alchemist Generic version of DART framework for distributed scientific search and analysis WHiP - Workflows Hosted in Portals Share workflows in collaborative Web portals for scientific collaboration and knowledge capture

Links Triana - WSPeer - WHiP - Mrs DART - Java GAT -