1. Empirical Studies of Knowledge Acquisition - or - Natasha and Mark do time at Leavenworth
Natasha Fridman Noy
Mark A. Musen
Stanford University
Stanford, California USA

2. Overview Protégé-2000 version 1.0 DARPA’s HPKB program
Empirical evaluation of Protégé-2000
Where to we go from here?

3. Generations of Protégé systems at SMI
PROTÉGÉ LISP-Machine system for rapid knowledge acquisition for clinical-trial specifications
PROTÉGÉ-II NeXTSTEP system that allowed independent ontology editing and selection of alternative problem-solving methods
Protégé/Win Finally, a Protégé system for the masses ...
Protégé/Java (a.k.a. Protégé-2000) The subject of this talk ...

4. Protégé/2000
Represents the latest in a series of interactive tools for knowledge-system development
Facilitates construction of knowledge bases in a principled fashion from reusable components
Allows a variety of “plug ins” to facilitate customization in various dimensions
Still needs a better name ...

5. Knowledge-base development with Protégé/2000
Build a domain ontology (a conceptual model of the application area)
Custom-tailor GUI for acquisition of content knowledge
Elicit content knowledge from application specialists
Map domain ontology to appropriate problem solvers for automation of particular tasks

6. Building knowledge bases: The Protégé methodology
Domain ontology
to provide domain
of discourse
Knowledge-acquisition tool
for entry of detailed content

7. Protégé/2000 Ontology-editing tab
Add additional
constraints on classes and
attributes
Developer can see knowledge
organization clearly
Easy
to edit attributes and facets
Classification
problems become
viewable

8. Generation of usable domain-specific KA tools
Protégé/2000 system
takes as input a domain ontology
generates in real time a graphical KA tool
Developers
Tweak KA tool appearance by using direct-manipulation layout-editing facilities
Add custom user-interface widgets when complexity of domain warrants more specialized visual metaphors

9. A great case for customized widgets: monitoring nuclear power plants

12. Some Advances in Protégé/2000
Much improved
editing of ontologies
creation and customization of knowledge-acquisition tools
adaptation of system to new requirements
But still no automated support for mapping of knowledge bases to problem-solving methods—yet!
No more shuffling among different development tools!

13. Protégé-2000 adopts the OKBC knowledge model
Protégé-2000 knowledge-bases are OKBC-compliant
Protégé-2000 is not OKBC-generic
There are some OKBC knowledge bases that Protégé-2000 cannot handle
It’s very close, though!
Differences are required to ease KA
Instances are instances of exactly one class

14. The race to develop plug-ins
GUI widgets for
tables
diagrams
animation
File I/O plug ins for interoperability with databases, other knowledge-based systems
Tab plug-ins for embedded applications

15. Swapping components Each of the Protégé-2000 major components can be
swapped out and replaced with a different one
Knowledge
model
Storage
User
interface

16. Protégé-2000 plug-ins Will revolutionize development of KA tools
Allow nearly every aspect of the system to be modifiable in a well-defined manner
Allow multiple groups each to develop special-purpose plug ins for their own purposes
Will lead to libraries of plug-ins to allow KA systems to be adapted in radical ways
Are already being developed by a widely distributed user community!

17. But how do we know we’re making progress?
Most KA systems are never evaluated
There are no well-established evaluation approaches
There are no benchmarks for comparison
Most KA-tool users do not want to participate in evaluation experiments
They have their own work to do
Evaluation is time-consuming

18. Sisyphus experiments Have been organized by KA community
Have involved shared tasks
Office assignment
Elevator configuration
Rock and mineral classification
Have done a better job of allowing comparison of knowledge-system architectures than of KA techniques

19. What is needed
Empirical studies of subject-matter experts entering “real” knowledge
Metrics for assessing
Quality of entered knowledge
Quantity of entered knowledge
Usability of KA tools
Environments where subject-matter experts can allocate the necessary time for these kinds of studies

20. We found a captive audience in Kansas ...

21. What the rest of the talk is about
High-Performance Knowledge Bases Program
Empirical evaluation of knowledge-based systems
Why and How?
How we
designed,
conducted,
evaluated a usability experiment
Extensions to Protégé
Experiment results

22. High-performance knowledge bases (HPKB) program
Enable developers to construct large knowledge bases
Reuse the knowledge
in multiple applications
with diverse problem-solving methods
in rapidly changing environments
Foster collaboration among multiple teams of technology developers and integrators

23. Two challenge problems
Crisis management challenge problem
Managing and understanding information before confrontation
Building systems to help warning analysts and policy makers
Battlespace challenge problem
Analyzing courses of actions for conformance with principles of warfare, resource allocation, feasibility and so on

24. Why does SMI care about HPKB
Research challenges common to both:
collaboration and knowledge sharing
management of large knowledge bases
knowledge-base development by subject-matter experts (SMEs) who are not experts in knowledge engineering
empirical evaluation of the tools and knowledge bases
Tools developed for HPKB are also applied in medical domains

25. Evaluating artificial-intelligence systems
“Studying AI systems is not very different from studying moderately intelligent animals such as rats”
— Paul R. Cohen, “Empirical Methods for Artificial Intelligence”

26. Designing an experiment
Formulate a hypothesis
What are we testing?
Determine what exactly affects performance
Remove various factors from the system and compare results
Create conditions for controlled experiment
Script sessions
Design tasks carefully

27. Knowledge-acquisition experiment
Evaluate how subject-matter experts (in this case, military experts) can use Protégé to develop and maintain knowledge bases

28. The problem Knowledge is not static The world changes
What we know about the world changes

29. Large-scale changes in military doctrine
From presentation by COL Mike Smith (

30. Domain experts need to interact with knowledge bases
Understand the knowledge base
Know what it contains (and what it doesn’t)
Perform quality control
Remove or change outdated knowledge
Acquire new knowledge
Extend the knowledge base to cover new areas of expertise

31. Specific goals for the experiment
Hypothesis 1
Subject-matter experts can use Protégé-2000 effectively for knowledge acquisition
Hypothesis 2
Highly custom-tailored tools for the specific domain improve knowledge-acquisition rate and quality

32. Domain: Opposing-force unit organization
Source:
Opposing Force (OPFOR) Battle book: force structure for opposing force
Why this domain?
The OPFOR information is used by intelligence analysts in planning battles
The OPFOR information is changing and needs to be verified and updated by intelligence analysts

33. Information represented in the knowledge base

34. Protégé-2000

35. HPKB tab

36. Purpose of the experiment
Compare Protégé-2000 and HPKB Tab
Protégé-2000
general-purpose tool for knowledge-base design and maintenance
allows automatic generation of forms for browsing and entering knowledge
HPKB Tab
Battlespace-analysis-specific addition to Protégé to collect unit-related information

37. Experiment methodology
Ablation experiment

38. Experiment time line Group 1 Group 2 Day 1 (use Protégé-2000)
Day 1 (use HPKB tab)
Morning: training session
Afternoon: experiment 1
Morning: training session
Afternoon: experiment 1
Compare
Protégé-2000
to HPKB tab
Day 2 (use HPKB tab)
Day 2 (use Protégé-2000)
Morning: training session
Afternoon: experiment 2
Morning: training session
Afternoon: experiment 2
Day 3 (use HPKB tab)
Day 3 (use Protégé-2000)
Test retention
of skills
Afternoon: experiment 3
Afternoon: experiment 3

39. Tasks Task design Tasks included
Seven tasks each day – from easy to more difficult
Each task starts on a new version of ontology
Sets of tasks for all three days are similar
Tasks included
Verifying what is in the knowledge base
Correcting the wrong information
Making information more specific
Creating new classes of units

40. Example of a task (task 4)
Verify that all Artillery subunits of Mechanized Infantry Brigade (IFV)(DIV) have their organization chart specified.
You need to verify that each artillery unit mentioned in the chart for Mechanized Infantry Brigade (IFV)(DIV) has its own chart defined. All subunits of other types are now fully specified and you do not need to verify this fact. Only study the artillery subunits. For each artillery unit that does not have the chart defined, or does not have it checked (that is, it may be not fully specified), create or complete the chart.

41. Preparing for evaluation
For each task, define a set of evaluation criteria in advance
What constitutes a correct answer?
What to do if there is more than one answer ?
What do we measure?
Logging capability
Keep logs of all steps for each user
Still hard to measure quality – some of the analysis had to be done manually
Usability questionnaires

42. Evaluation criteria Knowledge-acquisition rate Ability to find errors
Quality of knowledge entry
Subjective opinion

43. Evaluating quality of knowledge entry
How many errors SMEs found in the knowledge base
How many wrong steps SMEs took (vs. correct steps)
How many terms SMEs correctly added to the knowledge base
Have the SMEs noticed their errors themselves and where they able to recover
How long did it take for a user to recover from an error

44. Knowledge-acquisition rate (Days 1-3)2 4 6
HPKB Tab outperforms Protégé-2000 by 43%

45. KA rate improves substantially with learning

46. Knowledge base verification: finding errors
93% of errors
found
Knowledge base contained a small number of errors for each task.
The subjects had to find all the errors.
On average, the subjects using HPKB tab performed
26% better than the subjects using Protégé-2000

47. Quality of knowledge entry: wrong steps versus correct steps

48. Removing the “hangover effect”
Wrong steps: 1%

49. Task 6: enter a large amount of data

50. Error recovery rate Average number of steps to recover
from an error: 3.5

51. Creating new classes 14 new classes to create Observations
All the classes were placed in correct places
On the first two days subjects created additional categories to hold groups of similar classes
Explored (and changed) the hierarchy on their own

52. Retention of skills experiment: knowledge-acquisition rate

53. Retention of skills experiment
Results
Number of errors found
increased to 81% with Protégé
was 72% with HPKB Tab
Correctness
93% of the steps were correct

54. User satisfaction

55. Testing the hypothesis: Protégé-2000 versus HPKB tab
KA rate is 43% higher with HPKB tab
On the first day, the quality of knowledge entry is significantly better with HPKB tab

56. Summary of results Very small amount of training
No help at all on day 3
Knowledge-acquisition rate improves substantially with learning
Subjects found up to 93% of errors
Very low error rate: 6 % (almost 1% with HPKB Tab if you discount hangover effect)
One week later: still works….

57. Lessons learned Preparation, preparation, preparation
Do not expect anything
What you think is going to be hard is actually easy
What you think is easy, turns out to be hard
Dry-run is very important
Test the tasks
Test the software
Test the metrics collection mechanism

58. Lessons learned (2) Do not under-estimate the human factor
You need to break the ice
Design a valid experiment
“Our system does 5 apples per millennium”
Carefully designed tasks
Scripts for training sessions

59. Lessons learned (3) Leavenworth is not as bad as you would expect
Or is it?