1. Neil A. Ernst, Margaret-Anne Storey, Polly Allen, Mark Musen
Addressing cognitive requirements in knowledge engineering with Jambalaya
Neil A. Ernst, Margaret-Anne Storey, Polly Allen, Mark Musen

2. Outline
introduction: the increasing need for cognitive support in knowledge engineering
identifying tasks requiring cognitive support
survey
literature review
contextual inquiries
list of tasks
Jambalaya demonstration
discussion of non-functional goals for cognitive support
note that for the discussion at the end, I’d like to know if the requirements seem accurate, complete, different than others

3. Motivation/Background
Knowledge engineering is at something of a crossroads
Semantic Web initiative will see more ontologies and intelligent apps built, some (most?) by non-experts
byproduct: need for better metaphors for ontology modeling
not a new problem really, just not looked at much
better metaphors will need to consider techniques from information visualization
amplifies human ability to reason and conceptualize
early tools included this idea:
concept maps, SemNet, KEATS, CODE4
our work focuses on the modeler, not end-users or designers
code4 – separation of interface and model

4. Requirements Analysis
Why? First approach produced a tool, Jambalaya, which has seen little adoption or evaluation
Approach: combination of qualitative techniques
Literature review
Tallis et al – describe problems users encounter when doing KA tasks
Blythe et al – describe concerns users have adding new knowledge to a system
Ng 2000 – implemented a tool for browsing a DL ontology
Clark et al – non-KE able to enter knowledge accurately
problem: every domain has different challenges and users. We need to start synthesizing this work and finding commonalities.
quantitative techniques didn’t really work
tallis – browsing, checking for errors, adding knowledge
blythe – where to start, where to go next, get lost
ng – problems with implementation and scalability

5. Requirements Analysis (2)
User survey
Posted a survey to rdf-interest and protege-discussion about the uses of ontologies and information visualization
Available at
Some results:
large number of domains represented
most ontologies still small, but a significant number of large (>1000 frames),
Variety of visualization features requested, and variety of tools in use

6. Requirements analysis (3)
Contextual inquiries
two separate studies of large-scale knowledge engineering projects at the NCI (cancer research) and FMA (anatomy modeling)
observed ‘typical’ users to gain insight into what tasks are hard to do with current tools
interviewed managers and chief modelers to determine what their goals were, and where they saw visualization fitting in.
Some results:
need for custom-built reporting tool to augment existing system
requests for better navigation and search facilities
need for better support for collaboration

7. Proposed tasks requiring support
Using the contextual inquiry, survey, and lit review, we came up with a list of KE tasks we think need support in KE tools
facilitate navigation
overviews
slot-based browsing
view query results
save, annotate, and share views
support modeling
graphical editing techniques
editing navigation
support reuse
support verification
identify incoming relationships
incremental navigation
support complex relationships
in our paper we identify from which research techniques these were obtained
discussed at length in paper submitted to IJHCS

8. Jambalaya and its feature set
Based on the requirements/tasks identified, we evaluated Jambalaya’s functionality
Tools for manipulating a graph
Treemap view
Bookmarks
Search
Scripting
Views on the graph
Nested
Tree
Spring
Treemap
Brief demo of sample domain
focus should be on views not in other tools, and requirements (not the tool itself 5 min max) mature tool, open, extensible

9. Discussion
We suggest using non-functional design goals to examine what level of cognitive support is needed for e.g.
Customizability to support different domains and users
Usability problems still getting in the way of evaluation
Learnability of necessary functions often a problem
Examine issues centred around presenting the best metaphor in a particular circumstance
Expressivity: being able to reprsenet what’s in the model
Scalability: handling meaningful ontologies well
These goals are necessarily a series of trade-offs
Evaluation still difficult and under-emphasized

10. Summary of Workshop Discussion
Themes:
Semantic Web and inserted assertions/inclusions – information sharing
make life easier for users (= cheaper and more effective)
tools should be relevant to an example/domain not vice versa
deal with distributed information and 3rd party sources
query formulation, presentation, and manipulation
present context of navigation/area of interest
editing (need to reexamine)
meaningful products needed
more end-user focus (esp. tools e.g. Protégé vs IE (w/o plugins) (address previous topic)
Web services domain needs help (w. execution models) and popular