1. WS-VLAM Tutorial: Part II: Port a Legacy application to WS-VLAM Adam Belloum

2. Outline Part I: Hands on the User Graphical Interface Part II: Port a Legacy application to WS-VLAM Part III: Learn how to use WS-VLAM API

3. Outline (Part II): Porting A Legacy application (LA) Step 1: Define an Interface for your LA Step 2: Create a description file for your LA Step 3: Define the LA dependencies Step 4: Pack the LA and dependencies lib Step 5: Deploy your LA and dependencies lib Step 6: Compose a workflow using the LA Step 7: customize the composed workflow Step 8: execute the workflow Step 9: monitor

4. Who is doing What? Step 1, Step 2, Step 3, Step 4, Step 5  performed by the application developers, requires –Insight about the application to be ported –Programming knowledge – bit of scripting and OS Step 6, Step 7, Step 8, Step 9  performed by Scientists, require: –Application domain knowledge

5. Step 1: Define the component interface The Interface will define how WS-VLAM willinteract with your components – Parameter Interface: is use to interact at runtime with your module – Port interface: is used to stage in/out the data consumed/produced by your module Port in allows to stage in the consumed data Port out allows to stage out the produced data

6. Step 1: Define the component interface Example Omnimatch application Ports – Input ports: psf [type file] Tomogram [type file] Template [type file] Mask [type file] – Output ports: Output_file.ccf [type file] Output_file.ang [type file]

7. Step 2: Create a description file for your LA Using your favorite texteditor created a file callconfig.xml config.xml has 4 sections – input – output – parameter – exec Note: – in the omnimatchexample there is noparameter section

8. Step 3: Define the component dependencies Dependencies are libraries & auxiliary softwarethat are required by your component but are not available standard Operating systems You have to list the name of these libraries & auxiliary software in a file called “dependencies” Every libraries & auxiliary software listed in the dependencies files have to be packaged as a“pseudo-module” (to be explained)

9. Step 3: Create a pseudo-module A pseudo module is just asimple software packagefor which you need tocreate a profile.sh in the root directory Using your favorite texteditor created a file callprofile.sh # ############ NOTE #################### # In the profile.sh export all the environment # variables needed to execute your package # # bash environment # # export VARIABLE_NAME=VARIABLE_VALUE # # Add your VARIABLES bellow the line ######################################

10. Step 4: package your workflow component Create a directory withthe same name as the component Copy the files you havecreated: – config.xml – dependencies Copy any other files ordirectory related to yourcomponent: – bin/ lib/ … – scripts … Create a tar file of your component: $ tar cf COMPONENT_NAME \ COMPONENT_NAME/ Compress the tar file you have created $ gzip COMPONENT_NAME.tar

11. Step 5: Deploy your workflow component Test mode : you can test your component – Perform the stepsdescribed in the bluebox Shared mode : you can share your modulewith other users: – by moving the tar fileyou have created instep 4 to a componentrepository # On a machine where both GT4 & WS-VLAM # runtime system are installed # Create in your HOME directory called “.wsvlam” $ mkdir.wsvlam $ cd.wsvlam/ $ mkdir modules $ cd modules #untar the component tar file you have created # in step 4 $ tar –zxvf COMPONENT_NAME.tar.gz

12. Step 6.1: Compose a workflow using LA Create a description of your component : use CLAM workflow component configuration tool Action: Click on CLAM Button in the WS- VLAM menubar

13. Step 6: Compose a workflow using the LA Composition panel Components Palette Module property panel monitoring panel

14. Step 6.2: Compose a workflow using the LA CLAM workflow component configuration tool willallow you to define: name of the component,default host, ports, and parameters Action: Enter the name of thecomponent“ omnimatch ” Entre the name of thedefault host“ fs2.das3.science.uva.nl ”

15. Step 6.3: Compose a workflow using the LA CLAM workflow component configuration tool allowsyou to define: name of the component, default host,ports, and parameters Action: Click on ports tab Click + to add new input or output port Type the name of theports exactly as you did it in config.xml

16. Step 6.4: Compose a workflow using the LA To add a port a pop up window appears “ Add input Port ” or “ Add output port ” Action: Type the name of the ports exactly as you didit in config.xml Specify the type of the port either a simple data type, a file type, or Complex data type (see documentation formore details)

17. Step 6.5: Compose a workflow using the LA For the omnimatch you need to define 4 input ports and 2 output ports (see omnimatch config.xml) Action: Repeat the samesteps 6.3 and 6.4for each port

18. Step 6.6: Compose a workflow using the LA Create a description of your component : use CLAM workflow component configuration tool Action: Click on save tocreated thedescription file“ omnimatch.xml ”

19. Step 6.7: Compose a workflow using the LA Create a description of your component : use CLAM workflow component configuration tool Action: You will see in the filed file path the fullpath to the created“ ominimathc.xml ”

20. Step 6.8: Compose a workflow using the LA Restart the WSVLAM GUI : use CLAM workflow component configuration tool Action: Click on the start a new instance in the WS-VLAM menu bar

21. Step 6.9: Compose a workflow using the LA In the component list window you will see the created omnimatch components as well as othercomponents Action: Click on theomnimatch in thelist and drag anddrop in thecomposition panel

22. Step 6.10: Compose a workflow using LA An instance of Omnimatch will be instantiated Action: Click on theomnimatch in thelist and drag anddrop in thecomposition panel

23. Step 6.11: Compose a workflow using LA Instantiate the fileReader component: this component will get the input file needed byominimatch from a remote distribute location and stageit in to be used by omnimatch Action: Click on the fileReader inthe list and drag and dropin the composition panel Create a communicationchannel by draggingbetween an output port( read square) and input port ( blue square)

24. Step 6.12: Compose a workflow using LA Ominimatch requires 4 input files, we need to instantiate 4 fileReaders Action: Repeat step 6.11 for eachfileReader

25. Step 6.13: Compose a workflow using LA Instantiate the fileWriter component: This component will move the file produced by omnimatchto any remote location the user specifies Action: Click on the WriterReaderin the list and drag anddrop in the compositionpanel Create a communicationchannel by draggingbetween an output port( read square) and input port ( blue square)

26. Step 6.14: Compose a workflow using LA Instantiate the omnimatch workflow component Action: Repeat step 6.13 for eachfileWriter

27. Step 7.1: customize the composed workflow In the step you should check if all the default valuesof the parameters of the workflow componentscomposing your workflow are set correctly Action: Click on the fileReader incomposition panel In the property windowselect the parameter tab Type in the URI to the fileyou want stage inFileReader support manyprotocols: GSIFTP, http,file, ftp

28. Step 7.1: customize the composed workflow For the tutorial all the input file are available from Fs2.das2.science.uva.nl: Type in the following URIfor parameter in each fileReader – gsiftp://fs2.das.3science.uva.nl/home2/vleroot/wsvlam/data/psf.em – gsiftp://fs2.das.3science.uva.nl/home2/vleroot/wsvlam/data/tomogram.em – gsiftp://fs2.das.3science.uva.nl/home2/vleroot/wsvlam/data/template.em – gsiftp://fs2.das.3science.uva.nl/home2/vleroot/wsvlam/data/mask.em

29. Step 7.2: customize the composed workflow For the tutorial all the output files can be saved from on fs2.das3.science.uva.nl : Type in the following URI for parameter in each fileWriter – gsiftp://fs2.das3.science.uva.nl/tmp/output-1 – gsiftp://fs2.das3.science.uva.nl/tmp/output-2 – You can also use your home directory to save theresults of das3:gsiftp://fs2.das3.science.uva.nl/home/ /output1 or 2 – Any other host where ftp server is running andyou right to save file on it

30. Step 8: execute the composed workflow Now you are ready to execute the omnimatch workflow, which automatically stage in the data fileneeded for the omnimatch application, and stage outthe result files Action: Click on the run button

31. Outline (Part III): Learn how to use WS-VLAM API Step 1: Define an Interface for your component Step 2: Create a description for your component Step 3: WS-VLAM API Step 4: Example of JAVA components Step 6: Example of C/C++ components Step 5: Example of PYTHON components Step 6: Deploy the component/dependencies lib Step 7: Compose a workflow using the component Step 7: execute the workflow / monitor

32. Some References 1.Science Z Zhao, A Belloum, M Bubak Editorial: Special section on workflow systems and applications in e-Future Generation Computer Systems 25 (5), 525-527 2.A. Wibisono, D. Vasyunin, V. Korkhov, AS.Z. Belloum WS- VLAM: a GT4 based workflow management system, The 2 nd International Workshop on Scientific Workflows, In conjunction with ICCS 2007, Beijing, China, 2007 3.A.S.Z. Belloum, V. Korkhov, S. Koulouzis, M. A Inda, and M. Bubak Collaborative e-Science experiments: from scientific workflow to knowledge sharing JULY/AUGUST, IEEE Internet Computing, 2011 4. Ilkay Altintas, Manish Kumar Anand, Daniel Crawl, Shawn Bowers, Adam Belloum, Paolo Missier, Bertram Ludascher, Carole A. Goble, Peter M.A. Sloot, Understanding Collaborative Studies Through Interoperable Workflow Provenance, IPAW2010, Troy, NY, USA

33. Some References 5.A. Belloum, Z. Zhao, and M. Bubak Workflow systems and applications, Future Generation Comp. Syst. 25 (5): 525-527 (2009) 6. Z. Zhao, A.S.Z. Belloum, et al., Distributed execution of aggregated multi domain workflows using an agent framework The 1st IEEE International Workshop on Scientific Workflows, Salt Lake City, U.SA, 2007 7.Zhiming Zhao, Adam Belloum, Cees De Laat, Pieter Adriaans, Bob Hertzberger Using Jade agent framework to prototype an e-Science workflow bus Authors Cluster Computing and the Grid, 2007. CCGRID 2007