Supported by EU projects 12/12/2013 Athens, Greece Open Data in Agriculture Hands-on with data infrastructures that can power your agricultural data products

Robert Lovas MTA SZTAKI, Hungary Tutorial on how to use from external tools or build new complex data processing workflows (example: data aggregation)

Slide 3 Intro: user scenario a user (information manager) wants to create a data set on- the-fly that will include specific filtered subsets of existing data sources, e.g. CIARD RING – E.g. covering a particular geographical region, time period, type of resources use agINFRA-powered RING panel (Drupal module) to select desired data sources and relevant metadata properties, facets, formats – aggregation workflow executed when grid resources are available, notifying user to collect produced data set when ready an advanced user wants to develop further the existing workflows or create new complex data processing workflows

Slide 4 Objectives This presentation aims to provide illustration how an agINFRA powered data aggregation works with simple but powerful interface to – select targets, execute complex workflow, and provide the downloadable outputs (data sets on the fly) overview on designing workflows for data processing with user access and documentation details

Slide 5 Targets from CIARD RING vs. manual settings

Slide 6 Checking progress (with details)

Slide 7 Execution finished  records can be downloaded

Slide 8 The background Steps: 1.Configuration 2.Starting aggregation (workflow submission through Remote API ) 3.Enacting the workflow (Init harvests, etc. doing one-by-one) 4.As a step of the workflow, uploading the harvested data to Grid Storage LFN (using Robot certificate) 5.Registering the dataset in agCrouchDB 6.Retrieving the HTTP location 7.Provide the downloadable datasets

Slide 9 Aggregation workflow

Slide 10 Workflow support with gUSE (

Slide 11 agINFRA Science Gateway for workflows Liferay-based gUSE/WS-PGRADE for creating and running workflows  – or on-demand deployment – agINFRA VO is accessible (ca CPU cores, 900 TB storage)

Slide 12 Example: REST-specific features in job settings

Slide 13 Documentation of gUSE Used from the GUI For workflows

Slide 14 New! Request form for a new instance of a workflow engine (+ workflows and other components) on the Cloud  public-aginfra

Thank you! Robert Lovas MTA SZTAKI