Nikos Manolis Agro-Know Technologies Tutorial on data aggregation and accessing datasets

Slide 2 of 63 There is a lot of data

Slide 3 of 63 Need for data aggregation and harmonization

Slide 4 of 63 Objectives This presentation aims to provide information on:  How to use a service for aggregating datasets  How to get already processed datasets  How to search processed datasets with a search API Educational – GLN API (21008 res) Bibliographic – ABN API ( res)

Slide 5 of 63 The agDataHarvester service Implements the OAI-PMH protocol to harvest metadata records from open data providers – REST-based API – Harvested dataset available through HTTP

Slide 6 of 63 agDataHarvester parameters { "document_type": "harvesting_target", "harvesting_target": { " name ":"Repository name", " description ":”Short Repository Description", " url ":"OAI-PMH target URL", " type ":"metadata format prefix", " frequency ":hours }

Slide 7 of 63 param.json { "document_type": "harvesting_target", "harvesting_target": { "name":"Indian Academy of Science", "description":"Indian Academy of Science", "url":" "type":"mets", "frequency":24 } curl -X POST ac.rs/agcouchdb curl -X POST param.json { "ok": true, "id": " 5c56a3fa18fa21d2a85fd63cc9eb78ac ", "rev": "1- 19ef df8f1695a32b53ecb963a" }

Slide 8 of 63 Get details on the dataset 5c56a3fa18fa21d2a85fd63cc9eb78ac

Slide 9 of 63 Get details on the dataset {" id ": " b52d79e4797e210c06e6a0aee ", "key": " b52d79e4797e210c06e6a0aee", "value": { "_id": " b52d79e4797e210c06e6a0aee", "_rev": "1-d55d7bc90d26db64dae328c9328e4e4a", "document_type": "harvesting_target", "harvesting_target": { "name": “ WorldBank ", "description": "The World Bank - Open Knowledge Repository", "url": "" "type": “mets", "frequency": 24 }, "document_publisher": { "address": " ", "author": "demo001", "utc_datetime": "Wed Dec 11 11:58: ", "utc_timestamp": }

Slide 10 of 63 The agWorkflow service dataset.process=agworkflow&dataset.type=oai_lom&dataset.accuracy=true I want all datasets with educational resources processed by the agINFRA powered aggregation workflow ! dataset.process=agworkflow&dataset.type=oai_agris&dataset.accuracy=true I want all datasets with bibliographic resources processed by the agINFRA powered aggregation workflow !

Slide 11 of 63 Is there a way to search on available datasets ?

Slide 12 of 63 Search API REST-based queries over harmonized information (result of metadata processing) Two data models supported – akif: describing educational resources for agriculture, – agrif: describing bibliographic resources for agriculture (mainly from FAO’s data),

Slide 13 of 63 Search options Simple search Searching within specific fields api/v1/akif/?languageBlocks.en.description=tomato Temporal Fetching specific items

Slide 14 of 63 Managing results Sorting results e.g ?q=*&sort_by=creationDate&sort_order=desc Facets e.g ?facets=set&facet_size=3 Pagination e.g ?q=sea&page_size=25&page=3 Full Documentation : :8080/search-api/

Nikos Manolis Agro-Know Technologies

Slide 16 of 63 … … demo001…demo005 // aginfra

Slide 17 of 63 View all harvested datasets