1. Towards Evaluation of P2P-based DKMS
AMKM-2003
Christoph Tempich, M. Ehrig, C. Schmitz, S. Staab, J. Tane
University Karlsruhe

2. Agenda Introduction Evaluation process Test data
Current status and future work
Conclusion

3. Introduction Evaluation of algorithms necessary
Distributed Knowledge Management
No. of Conceptualizations
No. of Documents
No. of Domains
Unklarheiten über die No. von konzeptualisierungen, was sind domains,
Input Data.
Evaluation of algorithms necessary

4. Peers, Documents, nm Ontologies
Evaluation Input
Peers, Documents, nm Ontologies
nm Ontologies
n1 Ontologies
1m Ontologies
disjoint
overlapping
11 Ontologies
Number of domains to conceptualize
Number of conceptualizations for one domain
I want to introduce the notion of Ontologies briefly. shared spezification of a conceptualization. We have a hierachy of concepts. concepts a speziefied by properties. These properties may refer to other concepts or just be attributes. System based evaluation
Representation in RDF(S) / OWL

5. Evaluated Methods Alignment of ontologies Merging of ontologies
Creation of ontologies
Semantic routing
while there are many dimensions we want to focus on

6. Information Retrieval
Evaluation results
Relevance
Precision
Recall
No. of Hops
Network load
Reliability
Information loss
Information Retrieval
Peer-to-Peer
Ontology specific

7. Test data Corpus name no. doc text type Reuters 21678 21.578 yes
classification
DMOZ
hierarchical
Medline no
hierarchical ontology
self
parameter
pure ontology

8. Current status Implementation of Test environment
Inclusion into KAON tool suit
Testing of routing algorithms

9. Future work Test framework Enabling techniques like
Creation of a standard test ontology
Enabling techniques like
merging algorithms
semantic routing …

10. Conclusion Evaluation is important for DKMS
Combination of evaluation measures from different disciplines
Test dataset must be created

11. Thank you!
Acknowledgements:
SWAP

12. Evaluation Principles
User - based vs. System based
Combination of Techniques from
Information Retrieval
Ontology specific
Peer-to-Peer

13. Evaluation function
n > 1
problem of alignment
information loss