Presentation is loading. Please wait.

Presentation is loading. Please wait.

UTPB: A Benchmark for Scientific Workflow Provenance Storage and Querying Systems Artem Chebotko Joint work with E. De Hoyos, C. Gomez, A. Kashlev, X.

Published byIbrahim Modrell Modified over 9 years ago

Similar presentations

Presentation on theme: "UTPB: A Benchmark for Scientific Workflow Provenance Storage and Querying Systems Artem Chebotko Joint work with E. De Hoyos, C. Gomez, A. Kashlev, X."— Presentation transcript:

1 UTPB: A Benchmark for Scientific Workflow Provenance Storage and Querying Systems Artem Chebotko Joint work with E. De Hoyos, C. Gomez, A. Kashlev, X. Lian, and C. Reilly Department of Computer Science University of Texas - Pan American 6th IEEE International Workshop on Scientific Workflows, June 24, 2012 Was Derived From

2 Provenance in eScience  Metadata that captures history of an experiment  Problem diagnosis  Result interpretation  Experiment reproducibility  Scientific Workflow Community Provenance Challenges  2006: understanding and sharing information about provenance representations and capabilities  2006: interoperability of different provenance  2009: evaluating various aspects of OPM  2010: showcase OPM in the context of novel applications  Open Provenance Model  W3C Provenance Working Group UTPB – University of Texas Provenance Benchmark

3 SWFMS and Provenance  Taverna  Kepler  View  VisTrails,  Pegasus  Swift  Galaxy  Triana  OPMProv  Karma  RDFProv  etc. UTPB – University of Texas Provenance Benchmark  Support provenance collection  Use proprietary of third-party systems to manage provenance  Differ in provenance models, provenance vocabularies, inference support, and query languages.

4 Provenance Management Requirements  Non-functional  Data storage and querying efficiency and scalability  Inference soundness and completeness  Functional  Support of a particular, provenance model, provenance vocabulary, query type, inference feature, visualization and analysis  No standard way to evaluate provenance systems with respect to these requirements UTPB – University of Texas Provenance Benchmark

5 Provenance System Benchmarking Challenges  Well-documented and easy-to-understand datasets  Provenance data in a range of sizes  Provenance data with predefined inferred results that are known to be correct and complete  Test queries  Performance metrics  Result interpretation  Existing empirical studies of provenance systems use ad-hoc benchmarks or benchmarks developed in other research domains (see the paper for details) UTPB – University of Texas Provenance Benchmark

6 Our Contributions  University of Texas Provenance Benchmark (UTPB)  http://faculty.utpa.edu/chebotkoa/utpb/ http://faculty.utpa.edu/chebotkoa/utpb/  Focus on scalability and inference  Flexible data generator  27 provenance templates  3 virtual workflows  3 workflow execution scenarios  3 provenance vocabularies  27 test queries in 11 categories  5 performance metrics UTPB – University of Texas Provenance Benchmark

7 Talk Outline  University of Texas Provenance Benchmark  UTPB Architecture  Provenance Templates  Provenance Generation  UTPB Queries  Performance Metrics  Interpretation of Benchmark Results  Summary and Future work UTPB – University of Texas Provenance Benchmark

8 UTPB Architecture UTPB – University of Texas Provenance Benchmark

9 UTPB Architecture UTPB – University of Texas Provenance Benchmark

10 Provenance Templates UTPB – University of Texas Provenance Benchmark

11 Provenance Templates  A provenance template is a document that serializes provenance of one workflow execution according to a particular provenance model and a provenance vocabulary.  Provenance templates make the benchmark extensible and thus adaptable to the changing requirements of the field.  UTPB currently supports:  1 provenance model (OPM)  3 virtual workflows  3 provenance vocabularies (OPMV, OPMO, OPMX)  3 workflow execution scenarios  1 x 3 x 3 x 3 = 27 provenance templates UTPB – University of Texas Provenance Benchmark

12 Virtual Workflow 1  Database Experiment  Processes: 7  Artifacts:14  Accounts: 2  Agents: 1 UTPB – University of Texas Provenance Benchmark

13 Virtual Workflow 2  Jeans Manufacturing  Processes: 13  Artifacts:18  Accounts: 3  Agents: 2  Several processes use and generate the same artifacts and are “executed” in parallel UTPB – University of Texas Provenance Benchmark

14 Virtual Workflow 3  French Press Coffee  Processes: 15  Artifacts:15  Accounts: 4  Agents: 0  Several branches with multiple processes are “executed” in parallel  Several processes trigger each other without the record of using or generating artifacts UTPB – University of Texas Provenance Benchmark

15 Provenance Vocabularies  Almost every existing scientific workflow management system defines its own proprietary model for provenance  Each model is serialized in some format, such as RDF, XML, or relational data, according to one or more predefined vocabularies or schemas.  Open Provenance Model (OPM) – a layer of interoperability  OPM Vocabulary  OPM Ontology  OPM XML Schema UTPB – University of Texas Provenance Benchmark

16 Workflow Execution Scenarios  successful execution  incomplete execution with an error  successful execution with materialized provenance inferences UTPB – University of Texas Provenance Benchmark

17 Provenance Generation UTPB – University of Texas Provenance Benchmark

18 Provenance Generation UTPB – University of Texas Provenance Benchmark

19 Provenance Generation UTPB – University of Texas Provenance Benchmark

20 Provenance Generation # Named graph: http://cs.panam.edu/utpb#opmGraph_C0_T0 @prefix opmv:. @prefix rdf:. @prefix rdfs:. @prefix utpb:. utpb:account_black_C0_T0 rdf:type. utpb:cuttingMachine_C0_T0 rdf:type opmv:Artifact. utpb:denim_C0_T0 rdfs:label "blue". utpb:andrey_C0_T0 rdf:type opmv:Agent. utpb:cutDenim_C0_T0 opmv:used utpb:cuttingMachine_C0_T0, utpb:cuttingPattern_C0_T0, utpb:denim_C0_T0. utpb:denimParts_C0_T0 opmv:wasGeneratedBy utpb:cutDenim_C0_T0. # Default graph. OPMV UTPB – University of Texas Provenance Benchmark

21 Provenance Generation # Named graph: http://cs.panam.edu/utpb#opmGraph_C0_T0 @prefix opmo:. @prefix opmv:. @prefix rdf:. @prefix rdfs:. @prefix owl:. @prefix utpb:. utpb:account_black_C0_T0 rdf:type opmo:Account. utpb:cuttingMachine_C0_T0 rdf:type opmv:Artifact. utpb:propertyDenim_C0_T0 opmo:key utpb:keyDenimType_C0_T0 ; opmo:value "blue". utpb:andrey_C0_T0 rdf:type opmv:Agent. utpb:used1_C0_T0 rdf:type opmo:Used ; opmo:effect utpb:cutDenim_C0_T0 ; opmo:cause utpb:cuttingMachine_C0_T0 ; opmo:role utpb:roleMachine_C0_T0 ; opmo:pname utpb:_used1 ; opmo:account utpb:account_black_C0_T0. utpb:wgb1_C0_T0 rdf:type opmo:WasGeneratedBy ; opmo:effect utpb:cutDenim_C0_T0 ; opmo:cause utpb:denimParts_C0_T0 ; opmo:role utpb:roleDenim_C0_T0 ; opmo:pname utpb:_wgb1 ; opmo:account utpb:account_black_C0_T0. # Default graph. OPMO UTPB – University of Texas Provenance Benchmark

22 Provenance Generation <utpb xmlns="http://openprovenance.org/model/opmx#" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> laser Cutting machine OPMX UTPB – University of Texas Provenance Benchmark

23 UTPB Queries UTPB – University of Texas Provenance Benchmark

24 UTPB Queries  27 Queries  11 Categories  Graphs  Dependencies  Artifacts  Processes  Accounts  Agents  Roles  Values  Cross-Graph Queries  Inferences  Application-Specific UTPB – University of Texas Provenance Benchmark

25 UTPB Queries UTPB – University of Texas Provenance Benchmark

26 UTPB Queries UTPB – University of Texas Provenance Benchmark

27 UTPB Queries TypeFormatSample Query EnglishFind all artifact derivation dependencies in a particular provenance graph SPARQLOPMV SELECT ?causeArtifact ?effectArtifact FROM NAMED WHERE { GRAPH utpb:opmGraph { ?effectArtifact opmv:wasDerivedFrom ?causeArtifact. } } SPARQLOPMO SELECT ?causeArtifact ?effectArtifact FROM NAMED WHERE { GRAPH utpb:opmGraph { ?wdf rdf:type opmo:WasDerivedFrom. ?wdf opmo:cause ?causeArtifact. ?wdf opmo:effect ?effectArtifact. } } XQueryOPMX declare default element namespace "http://openprovenance.org/model/opmx#"; { for $wdf in /utpb/opmGraph[@id="opmGraph_C0_T0"]/dependencies/wasDerivedFrom return {$wdf/effect}{$wdf/cause} } UTPB – University of Texas Provenance Benchmark

28 UTPB Queries effectArtifact causeArtifact --------------------------------------------- utpb:denimParts_C0_T0 utpb:denim_C0_T0 utpb:rawJeans_C0_T0 utpb:denimParts_C0_T0 utpb:rawJeans_C0_T0 utpb:sewingThread_C0_T0 utpb:washedJeans_C0_T0utpb:rawJeans_C0_T0 utpb:inspectedJeans_C0_T0utpb:washedJeans_C0_T0 utpb:buttonedJeans_C0_T0utpb:inspectedJeans_C0_T0 utpb:buttonedJeans_C0_T0utpb:buttons_C0_T0 utpb:qualityJeans_C0_T0utpb:buttonedJeans_C0_T0 utpb:jeans_C0_T0 utpb:qualityJeans_C0_T0 utpb:jeans_C0_T0 utpb:labels_C0_T0 utpb:inspectedJeans_C0_T0utpb:washedJeans_C0_T0 utpb:qualityJeans_C0_T0utpb:buttonedJeans_C0_T0 OPMV UTPB – University of Texas Provenance Benchmark

29 UTPB Queries effectArtifact causeArtifact --------------------------------------------- utpb:denimParts_C0_T0 utpb:denim_C0_T0 utpb:rawJeans_C0_T0 utpb:denimParts_C0_T0 utpb:rawJeans_C0_T0 utpb:sewingThread_C0_T0 utpb:washedJeans_C0_T0utpb:rawJeans_C0_T0 utpb:inspectedJeans_C0_T0utpb:washedJeans_C0_T0 utpb:buttonedJeans_C0_T0utpb:inspectedJeans_C0_T0 utpb:buttonedJeans_C0_T0utpb:buttons_C0_T0 utpb:qualityJeans_C0_T0utpb:buttonedJeans_C0_T0 utpb:jeans_C0_T0 utpb:qualityJeans_C0_T0 utpb:jeans_C0_T0 utpb:labels_C0_T0 utpb:inspectedJeans_C0_T0utpb:washedJeans_C0_T0 utpb:qualityJeans_C0_T0utpb:buttonedJeans_C0_T0 OPMO UTPB – University of Texas Provenance Benchmark

30 UTPB Queries <result xmlns="http://openprovenance.org/model/opmx#" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> … OPMX UTPB – University of Texas Provenance Benchmark

31 Performance Metrics UTPB – University of Texas Provenance Benchmark

32 Performance Metrics  Data loading time  Repository size  Query response time  Query soundness  Query completeness UTPB – University of Texas Provenance Benchmark

33 Interpretation of Benchmark Results UTPB – University of Texas Provenance Benchmark

34 Interpretation of Benchmark Results  Comparison across datasets of varying sizes  Comparison using a fixed dataset  Comparison across data serialized with different vocabularies (e.g., OPMV vs. OPMO)  Comparison across data managed using different technologies (e.g., RDF vs. XML)  Comparison across data of different provenance models (e.g., OPM vs. PROV-DM) – in the future UTPB – University of Texas Provenance Benchmark

35 Summary and Future Work UTPB – University of Texas Provenance Benchmark

36 Summary and Future Work  UTPB: A first formal benchmark for scientific workflow provenance management systems  Extensible with new provenance templates  Flexible data generation  Large selection of test queries  Well defined performance metrics  Future work  Benchmarking existing system using UTPB  Extending UTPB (functional requirements, PROV-DM, new metrics – query expressiveness) UTPB – University of Texas Provenance Benchmark

37 THANK YOU! Questions? UTPB – University of Texas Provenance Benchmark  UTPB website:  http://faculty.utpa.edu/chebotkoa/utpb/ http://faculty.utpa.edu/chebotkoa/utpb/  My contact information:  Artem Chebotko, Department of Computer Science, University of Texas – Pan American  chebotkoa@utpa.edu chebotkoa@utpa.edu  http://www.cs.panam.edu/~artem http://www.cs.panam.edu/~artem

Download ppt "UTPB: A Benchmark for Scientific Workflow Provenance Storage and Querying Systems Artem Chebotko Joint work with E. De Hoyos, C. Gomez, A. Kashlev, X."

Similar presentations

Ontology-Based Computing Kenneth Baclawski Northeastern University and Jarg.

Ontology-Based Computing Kenneth Baclawski Northeastern University and Jarg.
Dr. Leo Obrst MITRE Information Semantics Information Discovery & Understanding Command & Control Center February 6, 2014February 6, 2014February 6, 2014.

Dr. Leo Obrst MITRE Information Semantics Information Discovery & Understanding Command & Control Center February 6, 2014February 6, 2014February 6, 2014.
1 Ontolog OOR Use Case Review Todd Schneider 1 April 2010 (v 1.2)

1 Ontolog OOR Use Case Review Todd Schneider 1 April 2010 (v 1.2)
OLAC Metadata Steven Bird University of Melbourne / University of Pennsylvania OLAC Workshop 10 December 2002.

OLAC Metadata Steven Bird University of Melbourne / University of Pennsylvania OLAC Workshop 10 December 2002.
1 ICS-FORTH EU-NSF Semantic Web Workshop 3-5 Oct Christophides Vassilis Database Technology for the Semantic Web Vassilis Christophides Dimitris Plexousakis.

1 ICS-FORTH EU-NSF Semantic Web Workshop 3-5 Oct Christophides Vassilis Database Technology for the Semantic Web Vassilis Christophides Dimitris Plexousakis.
Introduction to Planets Hans Hofman Nationaal Archief Netherlands Prague, 17 October 2008.

Introduction to Planets Hans Hofman Nationaal Archief Netherlands Prague, 17 October 2008.
Meta Data Larry, Stirling md on data access – data types, domain meta-data discovery Scott, Ohio State – caBIG md driven architecture semantic md Alexander.

Meta Data Larry, Stirling md on data access – data types, domain meta-data discovery Scott, Ohio State – caBIG md driven architecture semantic md Alexander.
Brief Introduction to Provenance As data becomes plentiful, verifiable truth becomes scarce

Brief Introduction to Provenance "As data becomes plentiful, verifiable truth becomes scarce
Feedback on OPM Yogesh Simmhan Microsoft Research Synthesis of pairwise conversations with: Roger Barga Satya Sahoo Microsoft Research Beth Plale Abhijit.

Feedback on OPM Yogesh Simmhan Microsoft Research Synthesis of pairwise conversations with: Roger Barga Satya Sahoo Microsoft Research Beth Plale Abhijit.
Q UERY L ANGUAGE C ONSTRUCTS FOR P ROVENANCE Murali Mani, Mohamad Alawa, Arunlal Kalyanasundaram University of Michigan, Flint Presented at IDEAS 2011.

Q UERY L ANGUAGE C ONSTRUCTS FOR P ROVENANCE Murali Mani, Mohamad Alawa, Arunlal Kalyanasundaram University of Michigan, Flint Presented at IDEAS 2011.
Open Provenance Model Tutorial Session 6: Interoperability.

Open Provenance Model Tutorial Session 6: Interoperability.
A Stepwise Modeling Approach for Individual Media Semantics Annett Mitschick, Klaus Meißner TU Dresden, Department of Computer Science, Multimedia Technology.

A Stepwise Modeling Approach for Individual Media Semantics Annett Mitschick, Klaus Meißner TU Dresden, Department of Computer Science, Multimedia Technology.
Open Provenance Model Tutorial Session 2: OPM Overview and Semantics Luc Moreau University of Southampton.

Open Provenance Model Tutorial Session 2: OPM Overview and Semantics Luc Moreau University of Southampton.
WIMS 2014, June 2-4Thessaloniki, Greece1 Optimized Backward Chaining Reasoning System for a Semantic Web Hui Shi, Kurt Maly, and Steven Zeil Contact:

WIMS 2014, June 2-4Thessaloniki, Greece1 Optimized Backward Chaining Reasoning System for a Semantic Web Hui Shi, Kurt Maly, and Steven Zeil Contact:
Open Provenance Model Tutorial Session 7: Open Provenance Model Vocabulary.

Open Provenance Model Tutorial Session 7: Open Provenance Model Vocabulary.
ICS-FORTH May 23, 2009 1 An Ontological Approach to Digital Preservation Metadata Martin Doerr Foundation for Research and Technology - Hellas Institute.

ICS-FORTH May 23, An Ontological Approach to Digital Preservation Metadata Martin Doerr Foundation for Research and Technology - Hellas Institute.
An Introduction to Semantic Web Portal

An Introduction to Semantic Web Portal
1 UIM with DAML-S Service Description Team Members: Jean-Yves Ouellet Kevin Lam Yun Xu.

1 UIM with DAML-S Service Description Team Members: Jean-Yves Ouellet Kevin Lam Yun Xu.
KOM, SEKE, June 20, 2004 Representing Chains of Custody Along a Forensic Process: A Case Study on Kruse Model Tamer Fares Gayed, UQAM Hakim Lounis, UQAM.

KOM, SEKE, June 20, 2004 Representing Chains of Custody Along a Forensic Process: A Case Study on Kruse Model Tamer Fares Gayed, UQAM Hakim Lounis, UQAM.
27 January 2005 1 Semantically Coordinated E-Market Semantic Web Term Project Prepared by Melike Şah 27 January 2005.

27 January Semantically Coordinated E-Market Semantic Web Term Project Prepared by Melike Şah 27 January 2005.

Similar presentations

Ads by Google