Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Raúl García-Castro Asunción Gómez-Pérez September 27th, 2004 Benchmarking Methodology

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Ontology Technology/Methods EvaluationBenchmarking Desired attributes Weaknesses Comparative analysis... Continuous improvement Best practices Measurement Experimentation What has been done? in D Survey of Scalability Techniques for Reasoning with Ontologies Overview of benchmarking, experimentation, and measurement State of the Art of Ontology-based Technology Evaluation Recommendations

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez What are we doing? T Benchmarking methodology, criteria, and test suites General evaluation criteria: Interoperability Scalability Robustness Benchmark suites for: Interoperability Scalability Robustness Benchmarking supporting tools: Workload generators Test generators Monitoring tools Statistical packages... Benchmarking results: Comparative analysis Compliance with norms Weaknesses Recommendations on tools Recommendations on practices Benchmarkin g Methodology Ontology tools: Ontology building tools Annotation tools Querying and reasoning services Semantic Web Services technology GOAL: Provide a framework for benchmarking activities in WP 2.1

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Experiment results: test 1 test 2 test 3... Experiment results: test 1 test 2 test 3... What will be done? T 2.1.6: Benchmarking of ontology building tools Tools/Partners:... Benchmarking results: Comparative analysis Compliance with norms Weaknesses Recommendations on tools Recommendations on practices Benchmark suites: RDF(S) Import capability OWL Import capability RDF(S) Export capability OWL Export capability... Interoperability Do the tools import/export from/to RDF(S)/OWL? Are the imported/exported ontologies the same? Is there any knowledge loss during import/export?... UPM Experiment results: test 1 test 2 test 3... NO OK Benchmarkin g ontology building tools Benchmarking supporting tools: Workload generators Test generators Monitoring tools Statistical packages...

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Benchmarking methodology Processes InputsOutputs Task 1Task n... Plan 1 Goals identification 2 Subject identification 3 Management involvement 4 Participant identification 5 Planning and resource allocation 6 Partner selection Experiment 7 Experiment definition 8 Experiment execution 9 Experiment results analysis Improve 10 Report writing 11 Findings communication 12 Findings implementation 13 Recalibration Methodology processes Methodology: Benchmarking process is: Planned Collaborative More Semantic Web oriented More KW oriented

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Benchmarking methodology Plan 1.- Benchmarking goals identification Goals depend on the organisation’s vision, objectives, and strategies. 2.- Benchmarking subject identification 3.- Management involvement Inform the organisation's management about the benefits of the benchmarking study and its costs. Management support is needed to proceed and when implementing changes based on the benchmarking. 4.- Participant identification Identify and contact the members of the organisation that are involved with the selected tool. Select and train the members of the benchmarking team. 5.- Benchmarking planning and resource allocation The planning must consider time and resources. The planning must be integrated into the organisation's planning. Analyse the current tools in the organisation. Select, understand, and document the tool whose improvement would significantly benefit the organisation, according to: end user needs or expectations, organisational goals, etc. 6.- Benchmarking partner selection Identify, collect, and analyze information about the tools that are considered the best. Select the tools to benchmark with and make contact with someone in their organisations. The partner organisations may not belong to KW. Not all ‘best in class’ tools are developed by KW partners.

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Benchmarking methodology Experiment 7.- Experiment definition 8.- Experiment execution 9.- Experiment results analysis Determine the experimentation plan and method. Define the experiment that will be performed. The experiment must collect not just the data on the performance of the tools but the reasons of this performance. Communicate the partners the experimentation plan and method and agree on it. Perform the experiment according to the experimentation plan and method. The collected data must be documented and prepared for analysis. Compare the results obtained from the experiments and the practices that lead to these results. Document findings in a report, including the best practices found (if any).

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Benchmarking methodology Improve 10.- Benchmarking report writing The benchmarking report must provide an understandable summary of the benchmarking study with: An explanation of the benchmarking process followed. The results and conclusions of the experiments. The recommendations on improving the tools Benchmarking findings communication Findings must be communicated to all the organisation (including identified participants) and to the benchmarking partners. Collect and analyze any feedback received Benchmarking findings implementation Define a planning for the implementation of the benchmarking findings. Implement the necessary changes in order to achieve the desired results. Periodically monitor the benchmarked tool Recalibration Recalibrate the benchmarking process using the lessons learnt. The benchmarking process should be repeated forever in order to obtain a continuous improvement.

Benchmarking Methodology. Sep 27th, 2004 © R. García-Castro, A. Gómez-Pérez Raúl García-Castro Asunción Gómez-Pérez September 27th, 2004 Benchmarking Methodology