1. Remote Api Tutorial
How to call WS-PGRADE workflows from remote clients through the http protocol?

2. Goals - Solutions
Submission a workflow from a http client (program environment) avoiding the interactive (human) Portal layer of the WS-Pgrade infrastructure. Solution: The remote api web service extension of the WS-Pgarde infrastructure will be used as application layer in order to communicate with the backend (gUSE) instead of the Portal.
The workflows to be submit should be modifiable by the caller.
As the WS-PGRADE workflow definitions contain own set of input files, first of all the input files are candidates to be changed but the files of job executable, and the command line parameters can be changed as well.
Solution: Manual change of files to be replaced and the creation of new association table (coded in the „portmapping.txt” file). The main workflow descriptor file <workflow>.xml should be modified only in the case when command line parameters belonging to the individual nodes(jobs) of the workflow must be modified.
On completion of a workflow submission session the common resources should be cleared. Solution: The servers side data belonging to the workflow (submission) will be cleared upon the successful execution of the „Get output” command of the client

3. Typical workflow development scenario in case of remote call
HTTP Client
Upd. user files
GRAPH Ed
<wf>.zip
<wf>.xml
tree of <wf> files
Copy (modify)
Browser
<wf>.xml
<inputs>.zip
<file_i>
portmapping.txt
Eventual update
Copy and rename
Orig user files
4. Submit the modified workflow
2. Download tested workflow
3. Reengineering of wf definition
1.Create and submit original workflow
Client side
Server side
WS PGRADE SERVER 2
Portal
gUSE
Remote API Serv.
DCI Bridge
WS PGRADE SERVER 1
Portal
gUSE
DCI Bridge
Set of computational resources

4. Summary of workflow development and remote submission process:
Download the original workflow from the WS-Pgrade gUSE server.
Make a little reengineering:
separate the structure description from the other parts
explore the user defined (not channel) input files connected to the numbered ports of the named jobs and the files of executable of the named jobs
Pack the needed files together and create a text file which describes the association of the files and the named jobs and ports
Create the needed script files needed to habitual submission process (call, observe, download (delete))
Execute the scripts on the client machine

5. Basic assumptions of remote call setup
There is a working WS-Pgrade workflow engine (server) where the Remote API Servlet* extension has been installed There is a client machine containing the description (structure, input files) of a workflow
The client can reach the server by the http protocol
There is a sever created password known by the client(s)
The client has the necessary proxy certificate file if the submissions involved in the workflow are directed to such resources which need certificate bound authorization *
(see installation file and documentation on Sourceforge for version <V> : Code:
Documentation: )

6. Part II: Step by step tutorial
The reconstruction and submission of a working „AutoDoc” PS workflow will be discussed in this tutorial. For the sake of simplicity in this case the selected submission resource does not require a certificate bound authorization and the original files will not be changed.

7. 1a. Analysis of the „AutoDoc” workflow
We have to search for the five „free” input files of the job „AutoGrid” and for the executables of the jobs „AutoGrid”, „AutoDock” and „Collector”
We have to search for the five „free” input files of the job „AutoGrid” and for the executables of the jobs „AutoGrid”, „AutoDoc” and „Collector”

8. 1b. Download the workflow „AutoDoc” from the server

9. Structure of the downloaded „AutoDoc” workflow
4. level: Enumeration of input ports of a given job
2. level: The needed „workflow.xml” file and the root of file structure
1. level: The compressed file
Standard name of job executable files
3. level: The list of jobs
„0” is the standard name of input file associated to the given port

10. 2 Reengineering Find the „workflow.xml” file
Rename, gather and compress the „0”named input port files, and the „execute.bin” named job executable files.
Make a file named „portmapping.txt” contains lines of the following syntax: <input_port_file_name>=<workflow_name>/<jobname>/<port_number> OR <job_executable_file_name>=<workflow_name>/<jobname> Notes: In section 3. use the new names defined in section2. The new file names should be different. As the separator of lines within the file „portmapping.txt” is the unix compatible linefeed character (´\n´) do not edit this file in Windows - or do it there with care.

11. Reengineering steps 1 and 2a (copy and rename)

12. Reengineering 2.2b Compress the found user defined workflow files
We have two of the three needed files: „inputs.zip” and „workflow.xml”
We have the two of the three needed files: „inputs.zip” and „workflow.xml”

13. 2.3 Reengineering: Creation of the „portmapping.txt” file

14. 3.1.a Preparation of submission Structure of the „curl” command
curl { –F <paramName>= } <URL and entry point of the requested service>
If the <paramValue> is preceeded by the charcter then it indicates the path of a file which must be transferred to the server and renamed as <parameterName>

15. 3.1.b The structure of the Api maps the submission session:
The leading workflow submission command („submit”) returns a process identifier, the other commands („info” – workflow status „detailsinfo” – detailed workflow status „download” - get workflow output after finish „stop” delete the suspended workflow „suspend” suspend the run of the workflow „rescue” continues the suspended workflow) are referencing the process identifier

16. Submit Command example: „LocalSubmit.sh”
„m” indicates the subcommand selector interpreted by the „RemoteServlet”
Workflow descriptor files
URL of server containing the requested service
Value of password

17. Server side placement of password

18. Status observation command example: „Status.sh”
Process identifier (result of the submit command) is expected here – in this case - as command line argument

19. Download Output command example: „GetOutput.sh”
The requested name of the file to be downloaded must be defined here.

20. Session summary Returned identifier (ID) ID used as argument
WF state is „running”
WF state is „finished”
„GetOutput” produces the „output.zip” file
„GetOutput” produces the „output.zip” file

21. Advanced features (1)
Problem of resources requireing certificates: Solution: the Submit command can be extended by a certs tag. curl F ….
where the zip file
enumerats files with the following names x509up.<VO_NAME> and the contents of the
„ x509up.<VO_NAME> ” files can be gained for example by the „ myproxy-logon” command issued on a Globus UIF machine: Example: myproxy-logon –s n40.hpcc.sztaki.hu –l MyAccountName –o x509up.seegrid

22. Advanced features (2)
A workflow definition may contain embedded workflow(s).
Lets remember the syntax of portmappinng file: <input_port_file_name>= <workflow_name>/<jobname>/<port_number> OR <job_executable_file_name>= <workflow_name>/<jobname>
The namespace separation permits the redefinition of files belonging to the eventual embedded workflows as well.