Neil A. Ernst, Margaret-Anne Storey, Polly Allen, Mark Musen Addressing cognitive requirements in knowledge engineering with Jambalaya Neil A. Ernst, Margaret-Anne Storey, Polly Allen, Mark Musen

Outline introduction: the increasing need for cognitive support in knowledge engineering identifying tasks requiring cognitive support survey literature review contextual inquiries list of tasks Jambalaya demonstration discussion of non-functional goals for cognitive support note that for the discussion at the end, I’d like to know if the requirements seem accurate, complete, different than others

Motivation/Background Knowledge engineering is at something of a crossroads Semantic Web initiative will see more ontologies and intelligent apps built, some (most?) by non-experts byproduct: need for better metaphors for ontology modeling not a new problem really, just not looked at much better metaphors will need to consider techniques from information visualization amplifies human ability to reason and conceptualize early tools included this idea: concept maps, SemNet, KEATS, CODE4 our work focuses on the modeler, not end-users or designers code4 – separation of interface and model

Requirements Analysis Why? First approach produced a tool, Jambalaya, which has seen little adoption or evaluation Approach: combination of qualitative techniques Literature review Tallis et al. 1999 – describe problems users encounter when doing KA tasks Blythe et al. 2001 – describe concerns users have adding new knowledge to a system Ng 2000 – implemented a tool for browsing a DL ontology Clark et al. 2001 – non-KE able to enter knowledge accurately problem: every domain has different challenges and users. We need to start synthesizing this work and finding commonalities. quantitative techniques didn’t really work tallis – browsing, checking for errors, adding knowledge blythe – where to start, where to go next, get lost ng – problems with implementation and scalability

Requirements Analysis (2) User survey Posted a survey to rdf-interest and protege-discussion about the uses of ontologies and information visualization Available at http://shrimpviews.org/jambalaya/user-survey.htm Some results: large number of domains represented most ontologies still small, but a significant number of large (>1000 frames), Variety of visualization features requested, and variety of tools in use

Requirements analysis (3) Contextual inquiries two separate studies of large-scale knowledge engineering projects at the NCI (cancer research) and FMA (anatomy modeling) observed ‘typical’ users to gain insight into what tasks are hard to do with current tools interviewed managers and chief modelers to determine what their goals were, and where they saw visualization fitting in. Some results: need for custom-built reporting tool to augment existing system requests for better navigation and search facilities need for better support for collaboration

Proposed tasks requiring support Using the contextual inquiry, survey, and lit review, we came up with a list of KE tasks we think need support in KE tools facilitate navigation overviews slot-based browsing view query results save, annotate, and share views support modeling graphical editing techniques editing navigation support reuse support verification identify incoming relationships incremental navigation support complex relationships in our paper we identify from which research techniques these were obtained discussed at length in paper submitted to IJHCS

Jambalaya and its feature set Based on the requirements/tasks identified, we evaluated Jambalaya’s functionality Tools for manipulating a graph Treemap view Bookmarks Search Scripting Views on the graph Nested Tree Spring Treemap Brief demo of sample domain focus should be on views not in other tools, and requirements (not the tool itself 5 min max) mature tool, open, extensible

Discussion We suggest using non-functional design goals to examine what level of cognitive support is needed for e.g. Customizability to support different domains and users Usability problems still getting in the way of evaluation Learnability of necessary functions often a problem Examine issues centred around presenting the best metaphor in a particular circumstance Expressivity: being able to reprsenet what’s in the model Scalability: handling meaningful ontologies well These goals are necessarily a series of trade-offs Evaluation still difficult and under-emphasized

Summary of Workshop Discussion Themes: Semantic Web and inserted assertions/inclusions – information sharing make life easier for users (= cheaper and more effective) tools should be relevant to an example/domain not vice versa deal with distributed information and 3rd party sources query formulation, presentation, and manipulation present context of navigation/area of interest editing (need to reexamine) meaningful products needed more end-user focus (esp. tools e.g. Protégé vs IE (w/o plugins) (address previous topic) Web services domain needs help (w. execution models) and popular http://www.cs.uvic.ca/~nernst/vike