Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam The SEALS Platform for Scholarly Research Asun Gómez-Pérez Ontology Engineering Group Universidad Politécnica de Madrid Acknowledge to: Miguel Esteban Gutiérrez, Raul García-Castro, Nandana Mihindukulasooriya and the SEALS consortium

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam 2 Ed Hovy Requirements A kind of framework that graphically depicts the whole scholarly environment/workflow - on this, a set of markers, like Post Its notes, that each indicate a technology available for use at the relevant point - also on this, another set of Post Its notes that each indicate some kind of bottleneck or gap A kind of system that helps me to reproduce experiments that appear in research papers and compare my technology with other technologies Asun Gómez-Pérez Requirements

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam Scenario Tool1 Tool4 Tool5 Tool1 Tool4 Tool5    Tool1 Tool2 Tool1 Tool2 Tool3 Tool1 Tool4 Tool5 Tool1 Tool4 Tool5 Tool1 Tool2 Tool6 Tool2 Tool6 Tool7 Tool6 Tool7 Scholarly workflows How do I know which tools are appropriate?: Reproduce Experiments (unbiased, repeatable, …) Tools Test data Evaluations Results My tool My results

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam Heterogeneity in the tools to be compared Data Sets – Levels of complexity – Scalability – Synthetic, hand-crafted and real world data Tools – Execution environment – Functionalities – Criteria to evaluate: scalability, interoperability, etc. Metrics – Precision/recall, information losses of data, etc. Evaluations – Fully automatic – The user is in the loop – Hybrid approach – Extrinsic and intrisic Starting point – From scratch – Extend and adapt preliminar approaches

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam Test data selection Metrics selection Execution requirements selection Evaluation services implementation Interpretation selection Goals selection Tool features selection Tools selection Evaluation workflows implementation Assumptions selection Criteria selection Dimensions Evaluation design

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam SEALS platform for technology evaluation SEALS Platform Features 6 Open (everybody can use it) Scalable (to users, data size) Extensible (to more tests, different technology, more measures) Sustainable (beyond SEALS) Independent (unbiased) Repeatable (evaluations can be reproduced)

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam 7 Test Data Evaluations Tools The SEALS platform 7 Results Semantic Technology Recommendation Framework Results Comparisons

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam 8 SEALS Service Manager Runtime Evaluation Service SEALS Portal Test Data Repository Service Tools Repository Service Results Repository Service Evaluation Descriptions Repository Service Technology Providers Evaluation Organisers Technology Adopters Software agents SEALS Repositories SEALS Logical Architecture Virtualization: two execution environments

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam Evaluation Execution Process at 10,000ft 9 ER

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam Evaluation services 10 Exploit results Tools Test data Evaluations Results Reproduce experiments My tool My results My test data My results Tools Evaluations Use your dataset to evaluate others tool Tools Test data My evaluation My tool My results My test data Define your own experiment

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam 11 Achievements Specification of the Platform Architecture Workflows Implemented in BPEL Test Data Repositor y Service Tools Repositor y Service Results Repositor y Service Evaluation Descriptions Repository Service Implementation and Evaluation of Repositories SEALS entities (dataset, tools, evaluations and results) are described using Ontologies Semantic Technology Recommendation Framework Results Comparisons

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam 12 SEALS support to scholarly workflows 12 SEALS Service Manager Runtime Evaluation Service Test Data Repository Tools Repository Results Repository Evaluation Descriptions Repository SEALS Repositories Scholarly Workflow Repository Scholarly Portal Tool1 Tool2 Tool1 Tool2 Tool3 Tool1 Tool4 Tool5 Tool1 Tool4 Tool5 Tool1 Tool2 Tool6 Tool2 Tool6 Tool7 Tool6 Tool7 Tool1 Tool4 Tool5 Tool1 Tool4 Tool5   

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam Thank you for your attention!

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference th March Amsterdam A non exhaustive list of wishes Repositories – Adapt/Extend Data, tools, evaluation and result repositories – A scholarly workflow repository (workflow itself, inputs and expected outputs) Web application for WF visualization, including Post Its and recommendations Definition and execution of Evaluations – User friendly WF editors: BPEL4People – From Fully automatic to the user is in the loop Recommenders – Qualitiative models