WS-VLAM Tutorial: Part III (part2): Learn how to WS-VLAM PYTHON API Adam Belloum

Outline Part I: Hands on the User Graphical Interface Part III: Port a Legacy application to WS-VLAM Part III: Learn how to use WS-VLAM API Notation (Title background): –Generic step –Tutorial specific step

Outline (Part III): Learn how to use WS-VLAM API Step 1: Define an Interface for your component Step 2: Create a description for your component Step 3: Define the component dependencies Step 4: JAVA Tutorial (create a WSVLAM JAVA component) Step 6: C/C++ Tutorial ( create a WSVLAM C/C++ component) Step 5: PYTHON Tutoria l ( create a WSVLAM PYTHON component) Step 6: Deploy the component/dependencies lib Step 7: Compose a workflow using the component Step 8: execute the workflow / monitor

Step 1: Define the component interface The Interface defines how WS-VLAM will interactwith your components – Parameter Interface: is use to interact at runtime with your module – Port interface: is used to stage in/out the data consumed/produced by your module Port in allows to stage in the consumed data Port out allows to stage out the produced data

Step 1: Example File Reader The file reader “ fileReader ” opens a file specified as a URI using the parameter Interface “ filename ” and sends the content through an output port “ port_out ”. – fileReader support many protocols: GSIFTP, ftp, http, file

Step 2.1: Create a description To create a description of your component : use CLAM workflow component configuration tool Action: Click on CLAM Button in the WS- VLAM menubar

Step 2.2: Create a description CLAM workflow component configuration tool willallow you to define: name of the component,default host, ports, parameters, classification, category Action: Enter the name of thecomponent Entre the name of thedefault host“ fs2.das3.science.uva.nl ”

Step 2.3: Create a description CLAM workflow component configuration tool allowsyou to define: name of the component, default host,ports, and parameters Action: Click on ports tab Click + to add new input or output port Type the name of theports exactly as you did it in config.xml

Step 2.4: Create a description To add a port a pop up window appears “ Add input Port ” or “ Add output port ” Action: Type the name of the ports exactly as you didit in source code Specify the type of the port either a simple data type, a file type, or Complex data type (see documentation formore details)

Step 2.5: Create a description Create a description of your component : use CLAM workflow component configuration tool Action: Click on save tocreated thedescription file“ component- name.xml ”

Step 2.6: Create a description Create a description of your component : use CLAM workflow component configuration tool Action: You will see in the filed file path the fullpath to the created

Step 2.7: Example File Reader Create a description of fileReader : use CLAM workflow component configuration tool The FileReader has One output: port-name = “port_out” and Type=“file” One parameter parameter-name = “filename” and Type=“String” Action: Perform the tasks described in step 2.1 to step 2.6 to create the fileReader.xml

Step 3.1: Define the component dependencies Dependencies are libraries & auxiliary softwarethat are required by your component but are not available standard Operating systems You have to list the name of these libraries & auxiliary software in a file called “dependencies” Every libraries & auxiliary software listed in the dependencies files have to be packaged as a“ pseudo-module ”

Step 3.2: Create a pseudo-module A pseudo module is just asimple software packagefor which you need tocreate a profile.sh in the root directory Using your favorite texteditor created a file callprofile.sh # ############ NOTE #################### # In the profile.sh export all the environment # variables needed to execute your package # # bash environment # # export VARIABLE_NAME=VARIABLE_VALUE # # Add your VARIABLES bellow the line ######################################

Step 5.1: PYTHON API From a developer ’ s point of view your components are applications that use a speciallibrary provided by the WS-VLAM framework:the “ vlport ” library

Step 5.2: PYTHON modules (pre-requisite) Before you build the wrapper make sure that the SWIG toolkit and Python are installed in your system. The SWIG version should be higher then To compile the python wrapper – the environment variable VLAM_INSTALL must be set. – Just type ‘‘make dist‘‘. – You will find the library in dist/lib subdirectory. – The wrapper consists of an ”adapter” shared library with thename ” vlport.so ” and a python module named ” vlport.py ”.

Step 5.3: Extend your PYTHON code 1. initialize– >>> Import vlport 2. create an instance of vlport application– >>>> vlapp=vlport.VLAppFactory.activate ([" inputPort1 "," inputPort2 "], [" outputPort1 ", " outputPort2 "]) 3. get references to input and output ports – >>> inputPort1 =vlapp.getInputPort(" inputPort1 ") >>> outputPort1 =vlapp.getOutputPort(" outputPort1 ") – Note: should provide the same names for the ports asspecified at the initialization stag

Step 5.4: API for writing to an output port writeInt (integer number) – writes an integer to the output port writeDouble (decimal number) – writes a decimal number to the output port » writeString (String) – writes a string to the output port which accepts strings » Write (String) – write a raw data from String to the port withoutserialization

Step 5.5: API for reading from in input port readInt () – reads an integer number from the input port » readDouble () – reads a decimal number from the input readString () – reads a string from input the port Write (String) » read ([size]) – Read at most size bytes from the port (less if theread encounters EOF before obtaining size bytes)

Step 5.6: API for reading/writing to/from parameter getParameter (" paramName ” ) – Reads the value of “ paramName ” setParameter (" paramName ", " paramValue ” ) – Writes the value of “ paramValue ” to “ paramName ”

Step 5.5: Example of PYTHON component The Example : shows the code for a module which open a file using “ inputPort ” and write is to the ” outputPort ”. The name of the file is read through eh parameter interface # include "vlapp.h" #!/usr/bin/python import vlport vlapp=vlport.VLAppFactory.activate([“inpputPort], ["outputPort"]) filename=vlapp.getParameter("fileName") file=open(filename) outputPort=vlapp.getOutputPort("outputPort") fileDaata = file.read() outputPort.write(fileData) =vlport.VLAppFactory.deactivate(vlapp)

Step 6.1: Deploy your workflow component Create a file“ main.sh ” : #!/bin/sh export PYTHONPATH=$VLAM_INSTALL/lib: \ $PYTHONPATH python $BASE_DIR/bin/fileReader $*

Step 6.2: package your component Create a directory with thefileReader name as the component Copy the files you havecreated: – dependencies (step 3.1) – main.sh (step 6.1) Copy any other files ordirectory related to yourcomponent: – bin/fileReader.py (step5.5) – lib/ Create a tar file of your component: (1) Created fileReader.tar $ tar cf fileReader.tar fileReader/ (1) Compress fileReader.tar $ gzip fileReader.tar

Step 6.3: Deploy your workflow component Test mode : you can test your component – Perform the stepsdescribed in the bluebox Shared mode : you can share your module withother users: – by moving the tar fileyou have created instep 6.2 to a component repository # On a machine where both GT4 & WS-VLAM # runtime system are installed # Create in your HOME directory called “.wsvlam” (1) copy fileReader..tar.gz to the default directory $ mkdir.wsvlam $ cd.wsvlam/ $ mkdir modules $ cd modules #untar the component tar file you have created # in step 4 (1)untar fileReader..tar.gz $ tar –zxvf fileReader.tar.gz

Step 7.1: Compose a workflow fileReader In the component list window you will see the created FileREader components as well as othercomponents Action: Click on the fileReaderin the list and drag anddrop in thecomposition panel

Step 7.2: Compose a workflow using LA An instance of fileReader will be instantiated Action: Click on the fileReader in the list and drag anddrop in the compositionpanel

Step 7.3: Compose a workflow using LA Instantiate the fileWriter component: This component will move the file produced by fileReaderto any remote location the user specifies Action: Click on the WriterReade r in the list and drag and drop in thecomposition panel

Step 7.4: Compose a workflow using LA Create a communication channel: This component will move the file produced by FileReaderto any remote location the user specifies Action: Create a communicationchannel by dragging between an output port ( read square) and input port ( blue square)

Step 7.5: customize the composed workflow In the step you should check if all the default valuesof the parameters of the workflow componentscomposing your workflow are set correctly Action: Click on the fileReade r in composition panel In the property window select the parameter tab Type in the URI to the fileyou want stage inFileReader support manyprotocols: GSIFTP, http,file, ftp

Step 7.6: customize the composed workflow In the step you should check if all the default valuesof the parameters of the workflow componentscomposing your workflow are set correctly Action: Click on the fileWriter in composition panel In the property window select the parameter tab Type in the URI to the file you want stage in fileWriter supportmany protocols: GSIFTP, http,file, ftp

Step 7.8: customize the composed workflow Input file: fileReader parameter fs2.das2.science.uva.nl: Type in the following URI for parameter in each fileReader gsiftp:// fs2.das.3science.uva.nl/home2/ / output file: fileWriter parameter fs0.das3.cs.vu.nl : Type in the following URI for parameter in each fileReader gsiftp:// fs0.das3.cs.vu.nl/home0/ /

Step 8: execute the composed workflow Now you are ready to execute the omnimatch workflow, which automatically stage in the data fileneeded for the omnimatch application, and stage outthe result files Action: Click on the run button

Some References 1.Science Z Zhao, A Belloum, M Bubak Editorial: Special section on workflow systems and applications in e-Future Generation Computer Systems 25 (5), A. Wibisono, D. Vasyunin, V. Korkhov, AS.Z. Belloum WS- VLAM: a GT4 based workflow management system, The 2 nd International Workshop on Scientific Workflows, In conjunction with ICCS 2007, Beijing, China, A.S.Z. Belloum, V. Korkhov, S. Koulouzis, M. A Inda, and M. Bubak Collaborative e-Science experiments: from scientific workflow to knowledge sharing JULY/AUGUST, IEEE Internet Computing, Ilkay Altintas, Manish Kumar Anand, Daniel Crawl, Shawn Bowers, Adam Belloum, Paolo Missier, Bertram Ludascher, Carole A. Goble, Peter M.A. Sloot, Understanding Collaborative Studies Through Interoperable Workflow Provenance, IPAW2010, Troy, NY, USA

Some References 5.A. Belloum, Z. Zhao, and M. Bubak Workflow systems and applications, Future Generation Comp. Syst. 25 (5): (2009) 6. Z. Zhao, A.S.Z. Belloum, et al., Distributed execution of aggregated multi domain workflows using an agent framework The 1st IEEE International Workshop on Scientific Workflows, Salt Lake City, U.SA, Zhiming Zhao, Adam Belloum, Cees De Laat, Pieter Adriaans, Bob Hertzberger Using Jade agent framework to prototype an e-Science workflow bus Authors Cluster Computing and the Grid, CCGRID 2007