1. Method for developing the benchmark suites. October 10th 2005 1 © Raúl García-Castro Method for developing the benchmark suites Raúl García-Castro October 10th, 2005 Interoperability Working Days October 10th-11th, 2005

2. Method for developing the benchmark suites. October 10th 2005 2 © Raúl García-Castro rdfs:Resource rdfs:Containerrdf:Listrdf:Propertyrdfs:Class rdf:Statement rdfs:Datatype rdfs:Literal rdf:XMLLiteralrdf:Bagrdf:Seqrdf:Alt rdfs:ContainerMembershipProperty rdfs:member rdfs:seeAlso rdfs:isDefinedBy rdfs:value “property” rdfs:label rdfs:comment rdf:subject rdf:predicate rdf:object rdf:type rdfs:subclassOf rdfs:domain rdfs:range rdfs:subPropertyOf rdf:first rdf:rest RDF(S) Knowledge Model

3. Method for developing the benchmark suites. October 10th 2005 3 © Raúl García-Castro Goals Goal of the method: To obtain all the possible relations between the components of the RDF(S) knowledge model. What is a relation? component1 component2 relation1 Instances of class1 can be related to instances of class2 using the property property1. Goal of the benchmark suite: To test all the possible combinations of components of the RDF(S) knowledge model. class1 class2 property1 In RDF(S):

4. Method for developing the benchmark suites. October 10th 2005 4 © Raúl García-Castro Relation example rdfs:Resource rdfs:Literal rdfs:label Instances of rdfs:Resource can be related to instances of rdfs:Literal using the property rdfs:label. But also… rdfs:Class rdfs:Literal rdfs:label rdfs:Resource subclass rdfs:Resource rdfs:XMLLiteral rdfs:label rdfs:Literal subclass rdfs:label rdfs:domain: rdfs:Resource rdfs:range: rdfs:Literal

5. Method for developing the benchmark suites. October 10th 2005 5 © Raúl García-Castro Relation cardinalities As RDF(S) doesn’t constraint the cardinalities of its properties: rdfs:Resource rdfs:Literal rdfs:label * An instance of rdfs:Resource can be related to several instances of rdfs:Literal using the property rdfs:label. Several instances of rdfs:Resource can be related to an instance of rdfs:Literal using the property rdfs:label.

6. Method for developing the benchmark suites. October 10th 2005 6 © Raúl García-Castro Types of benchmarks b) Import all the possible combinations of two components with a property a) Import single components c) Import combinations of more than two components that usually appear together in RDF(S) graphs d) RDF(S) graphs with the different variants of the RDF/XML syntax rdfs:Resource rdf:Propertyrdfs:Class rdf:Statement rdfs:Literal rdfs:label rdfs:comment rdf:subject rdf:predicate rdf:object rdf:type rdfs:domain rdfs:range rdfs:Resource rdf:Propertyrdfs:Classrdf:Statement rdfs:Literal rdfs:label rdfs:comment rdf:subject rdf:predicate rdf:object rdf:type rdfs:domain rdfs:range rdfs:Resource rdf:Property rdfs:Classrdf:Statementrdfs:Literal rdfs:label rdfs:comment rdf:subject rdf:predicate rdf:object rdf:type rdfs:domain rdfs:range =

7. Method for developing the benchmark suites. October 10th 2005 7 © Raúl García-Castro a) Benchmarks that import single components Two benchmarks defined for each component. Using rdfs:Class as an example: Import a single component. Import several components. Class1 Class2 rdfs:Class

8. Method for developing the benchmark suites. October 10th 2005 8 © Raúl García-Castro b) Benchmarks that import all the possible combinations of two components with a property Step 1 Identify the possible relations of the component with others RDF(S) properties whose domain is the component RDF(S) properties whose domain is superclass of the component User defined properties whose domain is the component rdfs:Class rdfs:subclassOf rdfs:Resource rdfs:Class rdfs:Literal rdfs:member rdfs:seeAlso rdfs:isDefinedBy rdfs:value rdfs:label rdfs:comment rdf:type rdfs:Class “property1”

9. Method for developing the benchmark suites. October 10th 2005 9 © Raúl García-Castro b) Benchmarks that import all the possible combinations of two components with a property Step 2 For each relation, identify the possible ranges that the property can have and assign cardinalities. The ranges are: Components defined in the RDF(S) specification as the range of the property If rdfs:Class is range, the RDF(S) predefined instances of rdfs:Class An unknown component not defined in the graph (although being a resource) Subclasses of components defined in the RDF(S) specification as the range of the property rdfs:Class * rdfs:subClassOf * rdfs:Class * rdfs:subClassOf * rdfs:Datatype rdfs:Class * rdfs:subClassOf 1 rdfs:Clas s * rdfs:subClassOf 1 rdfs:Resourc e rdfs:Class * rdfs:subClassOf * unknown …

10. Method for developing the benchmark suites. October 10th 2005 10 © Raúl García-Castro b) Benchmarks that import all the possible combinations of two components with a property Step 3 a) The assigned cardinalities define the number of benchmarks that will be performed for each relation. C1 1 property 1 C2 C1 1 property * C2 C1 * property 1 C2 I1I2 property I1I2 property I1 I2 property I3 I1I2 property I1 I2 property I3

11. Method for developing the benchmark suites. October 10th 2005 11 © Raúl García-Castro b) Benchmarks that import all the possible combinations of two components with a property Step 3 b) The assigned cardinalities define the number of benchmarks that will be performed for each relation. C1 * property * C2 I1I2 property I1 I2 property I3 I1 I2 property I3 C1 * property * C1 I1 property rdfs:Property * rdfs:subPropertyOf * rdfs:Property I1I2 property I3 property rdfs:Class * rdfs:subClassOf * rdfs:Class

12. Method for developing the benchmark suites. October 10th 2005 12 © Raúl García-Castro c) Benchmarks that import combinations of more than two components that usually appear toghether in RDF(S) graphs Properties that have both domain and range (rdf:Property with rdfs:domain and rdfs:range). Statements that have subject, predicate and object (rdf:Statement with rdf:subject, rdf:predicate and rdf:object). Definitions of lists (rdf:List with rdf:first, rdf:rest and rdf:nil). C1 1 P1 + C2 C1C3 1 P2 + I1I2 P1 I1I3 P2 I1 I2 P1 I3 I1I4 P2 I1 I3 P2 I4 I1I2 P1 I1 I2 P1 I3 I1 I4 P2 I5 C1 1 P1 + C2 C1C2 1 P2 + I1 I2 P1 I1 P2

13. Method for developing the benchmark suites. October 10th 2005 13 © Raúl García-Castro d) Benchmarks that import RDF(S) graphs with the different variants of the RDF/XML syntax URI references: Absolute Relative to a base URI Transformed from rdf:ID attribute values Relative to an ENTITY declaration Language identification attributes in tags (xml:lang) Abbreviations of: Empty nodes Multiple properties Typed nodes String literals Blank nodes Containers Collections Statements Literal 1 Literal 1 =

14. Method for developing the benchmark suites. October 10th 2005 14 © Raúl García-Castro Pruning the benchmark suite The resulting benchmark suite can be HUGE! It was pruned according to its intended use and to the kind of tools that it is supposed to evaluate. We only considered the components of the knowledge model of RDF(S) that are most frequently used for modelling ontologies in ontology development tools: Classes Instances Properties Literals Class hierarchies Property hierarchies The resulting benchmark suite contains 72 benchmarks. http://knowledgeweb.semanticweb.org/benchmarking_interoperability/rdfs_import_benchmark_suite.html

15. Method for developing the benchmark suites. October 10th 2005 15 © Raúl García-Castro Benchmark definition IdentifierI09 DescriptionImport one class that is subclass of several classes Graphical representation RDF/XML file<rdf:RDF xmlns="http://www.w3.org/2000/01/rdf-schema#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">

16. Method for developing the benchmark suites. October 10th 2005 16 © Raúl García-Castro RDF(S) Export Benchmark Suite Followed the same method. Instead of the RDF(S) knowledge model, we took a common subset of the knowledge models of all the tools. We only considered: Classes Instances Object properties Datatype properties Literals Class hierarchies The resulting benchmark suite contains 52 benchmarks.

17. Method for developing the benchmark suites. October 10th 2005 17 © Raúl García-Castro Benchmark definition IdentifierE09 DescriptionExport one class that is subclass of several classes Graphical representation WebODE’s instantiationExport one concept that is subclass of several concepts Protégé’s instantiationExport one class that is subclass of several classes ……

18. Method for developing the benchmark suites. October 10th 2005 18 © Raúl García-Castro Method for developing the benchmark suites Raúl García-Castro October 10th, 2005 Interoperability Working Days October 10th-11th, 2005