1. Justification/ExplanationEvaluation Breakout Session 6/13/02 Stefano BertoloRichard Fikes AQUAINT PI Meeting Monterey, California June 11-13, 2002

2. Knowledge Systems Laboratory, Stanford University2 Straw Man Proposal  General Evaluation Principles  Scope of the evaluation  Independence of correctness and justification  Required Characteristics  Accountability  Meaningful ranking of justifications  Understandability of justifications  Desirable Characteristics  Natural language presentation  Justification clustering  Justification persistence  Agent-accessible API

3. Knowledge Systems Laboratory, Stanford University3 General Evaluation Principles  Scope of the evaluation  Evaluating the quality of the justification(s) the system provides in support of the answer(s) it has returned for a given question  Not evaluating answers  Would be an add-on to other evaluations  Independence of correctness and justification  Evaluate justifications whether or not the answer they justify is correct

4. Knowledge Systems Laboratory, Stanford University4 Required Characteristics  Source Identification  A justification must identify the sources on which it depends  If a justification has multiple "steps" (where the meaning of "step" is system-dependent), it must identify the sources on which each step depends  Understandability  Justifications should be easily and quickly understandable  Understandability will be assessed by a panel of human evaluators  The modality of the presentation is left undetermined and need not be fluent English  Interpretation of question  Must provide understandable description of system’s interpretation of the question

5. Knowledge Systems Laboratory, Stanford University5 Evaluation Principles  What are we evaluating?  Quality of description of system’s rationale?  Quality of system’s rationale?  Quality of description  Query: Is aluminum a metal? Answer: Yes  Justification: >Source: National Enquirer …    Justification: >Text: “Aluminum is not a metal.” >Source: … >Method: Match words in query to words in sentence in source

6. Knowledge Systems Laboratory, Stanford University6 Evaluation Principles  What are we evaluating?  Quality of description of system’s rationale?  Quality of system’s rationale?  Quality of rationale  Query: Is aluminum a metal? Answer: Yes  Justification: >Text: “Aluminum is not a metal.” >Source: … >Method: Match words in query to words in sentence in source  Justification: >Source: National Enquirer …  

7. Knowledge Systems Laboratory, Stanford University7 Evaluation Principles  What are we evaluating?  Quality of description of system’s rationale?  Quality of system’s rationale?  Quality of rationale  How well does the justification support the answer?  General criteria: >Relevancy of sources >Quality of sources >Strength of inferential links Yes Yes

8. Knowledge Systems Laboratory, Stanford University8 Next Steps  Write proposal for 2003 evaluation  Determine likely participants in an evaluation  Establish an e-mail list for proposal discussions Send subscribe message to Stefano Send subscribe message to Stefano