Towards Evaluation of P2P-based DKMS AMKM-2003 Christoph Tempich, M. Ehrig, C. Schmitz, S. Staab, J. Tane University Karlsruhe

Agenda Introduction Evaluation process Test data Current status and future work Conclusion

Introduction Evaluation of algorithms necessary Distributed Knowledge Management No. of Conceptualizations No. of Documents No. of Domains Unklarheiten über die No. von konzeptualisierungen, was sind domains, Input Data. Evaluation of algorithms necessary

Peers, Documents, nm Ontologies Evaluation Input Peers, Documents, nm Ontologies nm Ontologies n1 Ontologies 1m Ontologies disjoint overlapping 11 Ontologies Number of domains to conceptualize Number of conceptualizations for one domain I want to introduce the notion of Ontologies briefly. shared spezification of a conceptualization. We have a hierachy of concepts. concepts a speziefied by properties. These properties may refer to other concepts or just be attributes. System based evaluation Representation in RDF(S) / OWL

Evaluated Methods Alignment of ontologies Merging of ontologies Creation of ontologies Semantic routing while there are many dimensions we want to focus on

Information Retrieval Evaluation results Relevance Precision Recall No. of Hops Network load Reliability Information loss Information Retrieval Peer-to-Peer Ontology specific

Test data Corpus name no. doc text type Reuters 21678 21.578 yes classification DMOZ 190.000.000 hierarchical Medline 1.141.893 no hierarchical ontology self parameter pure ontology

Current status Implementation of Test environment Inclusion into KAON tool suit Testing of routing algorithms

Future work Test framework Enabling techniques like Creation of a standard test ontology Enabling techniques like merging algorithms semantic routing …

Conclusion Evaluation is important for DKMS Combination of evaluation measures from different disciplines Test dataset must be created

Thank you! Acknowledgements: SWAP

Evaluation Principles User - based vs. System based Combination of Techniques from Information Retrieval Ontology specific Peer-to-Peer

Evaluation function n > 1 problem of alignment information loss