Creating Creating, Maintaining Integrating Maintaining, and IntegratingUnderstandable Knowledge Bases Richard FikesDeborah McGuinnessSheila McIlraith Jessica Jenkins Steve Wilder Kengo Ishii Gleb Frank Yulin Li Honglei Zeng Knowledge Systems Laboratory Stanford University 1/24/01

Knowledge Systems Laboratory, Stanford University2 Forming Understandable Knowledge knowledge evolution  Knowledge formation requires knowledge evolution  KBs require multiple developmental steps to become useful KB evolution  KSL is building tools to support KB evolution >KB diagnostics –Bugs, missing knowledge, heuristic warnings, architectural advice >KB explanation –Customized to individual users and tasks >KB merging –Consistency checking using a hybrid reasoner (JTP) >KB modularization –To produce reusable KB building blocks  Knowledge formation requires expressive KR languages  KSL is extending current representation formalisms >Defaults, KB partitioning, perspectives, …

Knowledge Systems Laboratory, Stanford University3Chimaera  A Knowledge Evolution Environment  Tools for KB diagnosis and merging  Available as a Web service  www-ksl-svc.stanford.edu  Usable from a Web browser  Online user manual, tutorial, and demonstration movie  Performs KB diagnostics in batch mode  Uploads and analyzes user’s KB  Accepts KBs in MELD, KIF, OKBC, RDF, XML, DAML, …  Provides results as HTML pages linked to frames and axioms  Provides user selectable set of diagnostic tests  Analyzes both the structure and content of a KB  Uses reasoners to analyze content  Currently runs 28 diagnostic tests

Knowledge Systems Laboratory, Stanford University4 Recent Publications and Presentations  AAAI 2000  Deborah McGuinness, Richard Fikes, James Rice, and Steve Wilder; “The Chimaera Ontology Environment”; Intelligent Systems Track, Seventeenth National Conference on Artificial Intelligence; Austin, Texas; July 30 - August 3,  ICCS 2000  Deborah McGuinness; “Conceptual Modeling for Distributed Ontology Environments”; Eighth International Conference on Conceptual Structures: Logical, Linguistic, and Computational Issues; Darmstadt, Germany; August 14-18,  Invited Talks  Deborah McGuinness; “Ontology Environments”; >Autumn School for Cognitive Science – Freiburg, Germany, Sept., 2000 >Free University of Amsterdam (Vrij) – Sept >Sun Microsystems – Palo Alto, CA – Dec >National Center for Atmospheric Research – Boulder, CO, coming Feb 2001

Knowledge Systems Laboratory, Stanford University5 Classification of Diagnostic Results  Errors  Logical inconsistencies E.g., contradictory type constraints  Content structure errors E.g., terms used but not defined  Anomalies  Missing information E.g., type constraints  Redundancies E.g., redundant superclass and type links  Extraneous structure or content E.g., terms defined but not used  Summaries E.g., counts of term references  Suggestions E.g., use consistent naming conventions

Knowledge Systems Laboratory, Stanford University6 Diagnose Both Frames and Axioms  Examples of frame-oriented diagnostics  Local constraint contradicts inherited constraint (error)  Object an instance of disjoint classes (error)  Cyclic subclass links (anomaly)  Class with a single subclass (anomaly)  Object an instance of a non-leaf class (anomaly)  Class contains no local information (anomaly)  Include disjointness statements about class siblings (suggestion)  Examples of axiom-oriented diagnostics  Quantified variable not in the body of the axiom (anomaly)  Variable in implication antecedent but not in consequent (anomaly)  Illegal number of arguments for implication or negation (error)  Conjunction or disjunction with only one argument (anomaly)  Suggest breaking up exceptionally long axioms (suggestion)

Knowledge Systems Laboratory, Stanford University7 Next Steps: Repair Dialogues  Provide interactive follow-up to diagnostics  Identify KB content on which diagnosis result is based  Suggest repairs or repair strategies  Guide user through repair procedure  Examples  Class is a direct subclass of “THING” >Provide direct subclasses of THING as candidate superclasses >Step down through the class hierarchy  Class has redundant superclass links >Present the redundant links >Suggest removal of link(s) to most general classes  Type, cardinality, or bounds conflict >Present conflicting constraints >Suggest changing local conflicting constraint(s)  Missing information >Initiate acquisition dialogues for missing information

Knowledge Systems Laboratory, Stanford University8 Next Steps: Acquisition Dialogues  Chimaera notes missing information about “parent” of “Person”  User requests that Chimaera initiate an acquisition dialogue  Chimaera responds by asking questions:  “How many parents must a person have?” >n>n >At least n >At most n >Any number >Don’t know  “What kind of an object is a parent of a person?” >The parent is a …  Assume the SME responds:  “The parent is a person and a cat”  Chimaera might respond:  “A person cannot also be a cat. Is a parent of a person always a person?”

Knowledge Systems Laboratory, Stanford University9 Next Steps: “Background” Analysis  Reasoning tests that may take substantial time  Performed in background  Results incrementally posted on Web page  Result summaries sent to user via when ready  Example tests  Redundant axioms that are inferred by the KB (anomaly)  Inconsistent axioms whose negations are inferred by the KB (error)  Determine which relations in KB are primitive and non-primitive (summary) >Show relations on which each non-primitive relation depend  Determine classes that are disjoint (suggest adding results to KB)  Derive subclass and instance of links (suggest adding links to KB) I.e., classification and recognition  Suggest reordering of an implication’s antecedents based on number of inferable instances of each antecedent (suggestion)

Knowledge Systems Laboratory, Stanford University10 Contributions To SRI Team  Defaults  Delivered design document, June 2000  Providing design support for defaults in the Summer 2001 KB  Partition-Based Logical Reasoning  Techniques for answering queries from large KBs  Working jointly with Eyal Amir in John McCarthy’s group  Knowledge Base Diagnostics  SRI has provided sample RKF KBs  KSL has diagnosed the sample KBs and obtained feedback from SRI  KSL to provide a KB diagnosis service on an ongoing basis >SRI team to provide evaluation feedback on the diagnosis service  KSL to develop strategies for KB repair dialogues  KSL to develop strategies for incremental diagnosis

Knowledge Systems Laboratory, Stanford University11 KB Diagnosis Component Evaluation  Evaluation methodology  Obtain structured feedback from KB developers >“Check Box” feedback on individual diagnosis results >Follow-up questions on a sampling of diagnosis results >Summary assessments of overall value of diagnosis  Record and analyze repair dialogue use  Sample “Check Box” questions for KB developer  Does this diagnostic provide information you did not know? [yes no]  Does this diagnostic provide information you need to know? [ ]  Are you going to change the KB in response to this diagnostic? [yes perhaps no]  How difficult would it be to obtain this information in some other way? [ ]  Is the diagnostic understandable? [yes mostly marginally no]  Was running these diagnostics worth the time and effort? [ ]

Knowledge Systems Laboratory, Stanford University12 KB Diagnosis Component Evaluation  Sample follow-up questions  Check box question: Is the diagnostic understandable? [yes mostly marginally no]  Follow-up question: What would have made the diagnostic more understandable?  Check box question: Does this diagnostic provide information you need to know? [ ]  Follow-up question: What property or capability of the KB did this diagnostic enable you to improve?  Sample summary assessment questions  What diagnostic information about this KB would you like to have that is not provided by Chimaera?  What would make running these diagnostics more worthwhile?

Knowledge Systems Laboratory, Stanford University13Summary ChimaeraKB evolution  KSL is building the Chimaera tool suite to support KB evolution  Our current focus is on diagnosing KBs  Providing a KB diagnosis Web service  Finding errors and anomalies in both structure and content  Providing advice to KB authors  Using reasoners to provide sophisticated diagnostics  Developing KB repair and acquisition dialogues  We are providing support to the SRI team in multiple ways  Defaults  Partition-Based Logical Reasoning  KB Diagnostics  We will do a component evaluation experiment  To evaluate Chimaera’s KB diagnostics  Based on structured feedback from KB developers and repair dialogue use