CLARIN tools for workflows Overview

Objective of this document  Determine which are the responsibilities of the different components of CLARIN workflows.  Explain what is expected from the CLARIN Workflow tools. Requirements.

Firstly, what is a workflow?  Workflow is a definition of a process in a certain language. It is done in a non-ambiguous way.  A workflow is only the definition of a process.  Since the workflow definition is made in a non- ambiguous way, any software can execute it expecting the same results. It means that workflows can be shared between different applications.  Workflow may be a XML file that a user can write without any other help.

Workflow tools  Since a workflow is only a process description, we need 2 other components. Workflow Engine : Application that process (executes) workflow descriptions. Workflow Editors : Application for creation of workflow descriptions.  Both applications can be in the same piece of software. Workflow Workbench.

Workflow Engines: Expectations  Engines are applications that execute workflows in the way they are described.  Many engines can be developed. Some with GUI for desktop computers some other hosted in servers with schedule features, etc...

Workflow Editor: Expectations  For advanced users of CLARIN, a text editor could be enough for a workflow creation.  But CLARIN objective is easiness. In this way to create a workflow in CLARIN must be really easy.  The editors in CLARIN must help as much as possible to the user. This is done, for instance, querying the registry and taking decisions behind the scenes.  Example: User connects two web services using first’s output with second’s input. Metadata stored in the registry specifies that this operation is not possible without a conversion. Also the registry contains info about converters available. The editor must be aware of that and introduce a conversion tool between the web services. This can be done without user’s interaction at all.  SMART editors are possible since the Registry will contain all the required metadata for taking decisions automatically.

Example  User wants to connect 2 web services. Web service X Web service Y  X’s output needs to be Y’s input.  But X’s output is not in the proper format for being Y’s input.  The workflow editor must be able to find a converters.

Example. Finding a converter  The editor needs to do the following tasks: Get information about web service X. Which format has X’s output? Get information about web service Y. Which format requires Y’s input? Find a converter. How can data in format X be transformed in format Y?  All this information must be stored and accessible in the registry

Example. Get information of X  It is required to query the registry. Still it is not decided how this query will be but intuitively it should look like:  specObject = GetWebServiceSpecification(X_persistent_id)  This call will return the specification where explains that X’s output format follows the standard A.

Example. Get information of Y  Like previously with X, the editor queries the Registry for Y’s specification.  specObject = GetWebServiceSpecification(Y_persistent_id)  This call will return the specification where explains that Y’s input format follows the standard B.

Example. Finding a converter  Now the editor needs to find a converter from standard A to standard B.  It will be done using the Registry as well. Also it is not developed yet but it will be something similar to: ResourceList = GetResourceList(“/standards/converters”) The editor has downloaded the list of converters (resources under the standards/converters branch). Now inspecting this list of resource descriptions, the editor will find the required converter (if any).

Example. Introduce the converter in the workflow  Before the insertion of the converter in the workflow, depending on the user’s profile: the user will be asked for confirmation or the user will be noticed of different converters if more than one is available or simply won’t be notified at all (novice users)  The editor introduces the converter in the workflow between X and Y.  The workflow description will keep this converter information. Engines executing this workflow will need to know about this conversion.

Workflow editor: Requirements  Graphical representation of all workflow description elements. (loops, if then clauses, parallelization, exceptions, etc…)  SMART editor.  Different user profiles (Advanced users will want to know more about automatic decisions taken by the workbench while novice users won’t want to decide anything)  Drag & Drop features.  Registry browsing

Workflow workbench.  The editor can help even more to the user if it is aware of the workflow execution results. That’s why it is recommended to develop a Workbench (Workflow Editor + Workflow Engine together).  Also for basic users it is easier when everything is centralized in just one application.

Workflow Workbench: Requirements  All Editor requirements  Debugging tools: breakpoints, Start-pause execution and inspection of intermediate steps.  Partial workflow execution and reuse of data obtained in previous executions. (To avoid repeating calls to web services while testing)  Output console

Very important. SMART applications  Different profiles in CLARIN applications. Don’t bother novice users with advanced questions.  Applications remember similar situations in the past. “You connected two web services similar to this one using converter AtoB. Do you want to do it again?”  Use of community intelligence stored in the registry. “250 users connected these 2 web services in other projects using converter AtoB, do you want to do it as well?”