1. Evaluating Long-Term Use of the Gnowsis Semantic Desktop for PIM
Leo Sauermann
Dominik Heim
Knowledge Management Lab Kaiserslautern

2. Long-Term Evaluation of Semantic Desktop
we built Semantic Desktop
Evaluation 2006:
scientific question: is this good for PIM?
answer: probably
2008: oh, 2 still use it
scientific question: why, what is it good for?
answer: PIM
Results
Ways to do it better
sauermann,heim: semantic desktop long-term evaluation

3. Semantic Desktop
sauermann,heim: semantic desktop long-term evaluation

4. Information is chaotic
the CID project
Thousands of files, in hundreds of folders, in many applications and formats
Hard-disk space cheap as air, no one will delete these files
No desktop standard to represent ideas and align to files
Documents
Papers
Projects
CID
Research
files s
Inbox
Todo
SAP
CID-proj
Karlsruhe
contacts
Claudia Stern
Dirk Hagemann
Klaus Nord
sauermann,heim: semantic desktop long-term evaluation

5. The Personal Information Model PIMO
Ontologies
Person
represents personal mental model
formal: classes, concepts, relations in RDFS
documents, s, websites are “tagged” with concepts
extensible with
organizational knowledge
ontologies
extracted data
The CID Project
Topic
Project
Document
Management
CID
CID slides
Claudia
Karlsruhe
Person
Dirk
Location
Documents
Papers
Projects
CID
Research
Files s
Inbox
Todo
SAP
CID-proj
Karlsruhe
Contacts
Claudia Stern
Dirk Hagemann
Klaus Nord
Data
is a
Relation
Annotation
PIMO - a Framework for Representing Personal Information Models Sauermann, van Elst, Dengel (2007)
sauermann,heim: semantic desktop long-term evaluation

6. Semantic Wiki, Drop-Box, Thing Editor, Miniquire
Applications
Semantic Wiki, Drop-Box,
Thing Editor, Miniquire
sauermann,heim: semantic desktop long-term evaluation

7. Miniquire Sidebar Search PIMO Ontology viewer and editor Filtering
keeping orientation
sauermann,heim: semantic desktop long-term evaluation

8. PIMO Editor Relations to other things
wiki name = short unique name for things
semantic wiki text
Link to filesystem,
, outlook entry,
anything integrated through Aperture.sf.net
Text-Attributes:
key/value pairs for datatype attributes
sauermann,heim: semantic desktop long-term evaluation

9. Application: Drop Box Helps filing information
uses PIMO structures concepts and folders
Knows the users model and is trained by using it
process flow
files are stored into a Drop-Box folder
files are text-analysed and possible target folders are suggested
Drop-Box user interface shows
user selects a folder, classify
files are moved and classified
Learns from the user
doc: … EPOS …
Project:Epos EPOS, DFKI, Maus, …
PIMO of Paul =
Sauermann, L.; Grimnes, G.A.; Kiesel, M.; Fluit, C.; Maus, H.; Heim, D.; Nadeem, D.; Horak, B. & Dengel, A. Semantic Desktop 2.0: The Gnowsis Experience
ISWC 2006
sauermann,heim: semantic desktop long-term evaluation

10. Personal Semantic Wiki for note-taking
Terms
Types
Relation types
Relations
RDF
Nepomuk
Nepomuk is an EU project . It’s topic is the Social Semantic Desktop .
DFKI coordinates .
Malte Kiesel, Kaukolu: Hub of the Semantic Corporate Intranet ESWC 2006
Sauermann, L. The Gnowsis-Using Semantic Web Technologies to build a Semantic Desktop Diploma Thesis, Technical University of Vienna, 2003
sauermann,heim: semantic desktop long-term evaluation

11. Thing = RDF resource (Instance of a Class) = Wiki page
Questions from reviewers: how is it different from YetAnotherSemanticWiki?
Thing = RDF resource (Instance of a Class)
= Wiki page
= Wiki name, Tag name
= File Folder
= a File/ /Website/Contact…
unifies Ontologies, Tagging, Wiki
curiosity: was there another Semantic Wiki before?
today, are there others unifying Wikis?
Sauermann The Gnowsis-Using Semantic Web Technologies to build a Semantic Desktop
sauermann,heim: semantic desktop long-term evaluation

12. Personal Information Management PIM
Book, wikipedia Article, Series of Workshops, Jaime Teevan, William Jones
Support for three activities in the personal space of information:
filing
finding
thinking
Science: mostly field studies, sometimes tools
sauermann,heim: semantic desktop long-term evaluation

13. How can I evaluate this
sauermann,heim: semantic desktop long-term evaluation

14. 2006 Usability&Usefulness Evaluation
Preliminary work: Two studies 2005 to warm up
July2006: two month study with 8 participants
age 25-40, 1female, all DFKI, 6 in our lab
2 Students, 6 researchers, all knowledge workers
sauermann,heim: semantic desktop long-term evaluation

15.
sauermann,heim: semantic desktop long-term evaluation

16. 2006 Usability&Usefulness Evaluation
Procedure
Installation and introduction tutorial w. researcher
Individually walked through steps
Team approach: Weekly show and tell how they used the software
Activity Log file
Final on-workplace contextual inquiry
All this talking: find out for what they use it.
Morse, E.L: Evaluation Methodologies for Information Management Systems D-Lib Magazine, 2002
sauermann,heim: semantic desktop long-term evaluation

17. 2006 Results Expectation Questionnaire: Good/Bad/most used
power users: most uses wikis before, half desktop search
Good/Bad/most used
type-ahead function in wiki was a winner
Miniquire, Thing Editor, DnD, Starring, installer
Bad: Switching between web wiki frontend and Java Swing application
Software bugs cut away half of the study ( )
Said they liked wiki, dropbox, but did not use much
Ontology work
did scarcely create classes
used for project and event management, note keeping
underlying reasons missing, gathered numbers did not help
sauermann,heim: semantic desktop long-term evaluation

18. 2008 Methodology Only contextual inquiry with statistical backup
2 users: KM workers, 2 years usage, friends of author: is it good? for what did you use it?
Procedure
setup video camera recording screen, use written guide
warm up “whats your name”, “For what did you use it last?”, “Show me Classes, things, wiki…”
do a task you have to do anyway, talk aloud
be kind: break, drink coffee, bring chocolate
Cross-check against logfile and video. Follow up questions later via phone/mail.
Interviewer has no computer but sits there and plays nice with a paper and pen.
sauermann,heim: semantic desktop long-term evaluation

19. please read the paper and ask questions Create classes when:
2008 Results
please read the paper and ask questions
Create classes when:
semantics not useful any more: “first all was project, then I split to my projects VS other people’s projects”
too many entries in one class & a sub-group was viable (3 Organizations turned to ResearchInstitute)
People and Documents were annotated with Topics (created 59, 201 topics)
sauermann,heim: semantic desktop long-term evaluation

20. Semantics and use of properties
User interface does not support creating properties well. mostly Predefined properties.
observation during interview: relations used as “bread crumbs”, trails A B
part of
566 63
related
122
193
has topic
401 78
sauermann,heim: semantic desktop long-term evaluation

21. DropBox and Wiki DropBox was used Wiki was used
386 and 149 times, allowed multi-tagging and fast filing
Wiki was used
288 instances with 92 pages
959 instances with 148 pages
meeting notes, brainstorming, intermediate texts, link collections, literature reviews
hacks: hidden features, repurposing functionality “using embedding as task-list replacement”
sauermann,heim: semantic desktop long-term evaluation

22. Reviewer: Orienteering Behavior
Besides being shown differently, hasPart and isRelated have no programmatic difference.
For orienteering,
“click somewhere to the right” is enough
Teevan, Alvarado, Ackerman, Karger
“users go somewhere, look around, know where they are, go on”
Filtering
keeping orientation
Long-term users stopped using web-search interface
spatial
Teevan, J.; Alvarado, C.; Ackerman, M.S. & Karger, D.R. The perfect search engine is not enough: a study of orienteering behavior in directed search CHI 2004, pp
sauermann,heim: semantic desktop long-term evaluation

23. Mental Model VS PIMO Structures
Mental Model and PIMO differ, in memory, they think they have more relations than in store
“Donald X, when I was looking at his homepage, I added topics to him” – no, to the documents
Orienteering: once they see it, they remember
“Entry Points” or “Anchors”: i,e one important person from which to start navigating
sauermann,heim: semantic desktop long-term evaluation

24. Interpretation A little semantics goes a long way
The Semantic Desktop helps in PIM’s typical activities: filing, finding, thinking
Limited RDF: No computation, no social exchange
Classes were only needed when the pain was too big
Typed relations were not needed: “related” would be enough for personal orienteering
Personal Semantic Desktop Wiki was used as a general knowledge tool
PIM research was confirmed:
orienteering
first navigating, then searching
CID
CID slides
sauermann,heim: semantic desktop long-term evaluation

25. We could have done better – borderline paper
A scientific question, ours was: give them Semantic Desktop, see how they use it
Analyze
Investigate structures, text and contextualize – in which situation was what changed – are there correlations
Do a trend analysis over time
Do a proper usefulness HCI study
Ping the user once a day to ask “are you using SemDesk now, for what?” Experience Sampling Method (ESM)
Measuring improvements of specific activities: filing/finding/thinking
Its all tricky: we evaluate people, not algorithms privacy! I must not keep the data!
Chernov, Serdyukov, Chirita, Demartini, Nejdl, Building a Desktop Search Test-Bed, ECIR2007 pp (not cited in the paper)
sauermann,heim: semantic desktop long-term evaluation

26. Summary – Results and methodology
PIM evaluations are long-term, real tasks, on site (Kelly 2006). KM Eval also?
You cannot force/pay/motivate long-term studies without usefulness. So it has to be simple software that works. 80% of my users dropped out.
I start a company to go on here
We need more end-user studies on long-term RDF use.
sauermann,heim: semantic desktop long-term evaluation

27. questions? </SemanticDesktop> </rdf:RDF>
not cited:
Chernov, Serdyukov, Chirita, Demartini, Nejdl Building a Desktop Search Test-Bed ECIR2007 pp
Teevan, Alvarado, Ackerman, Karger The perfect search engine is not enough: a study of orienteering behavior in directed search CHI 2004 pp d
asdf
asfd
sadf
sda fd
dfs df da
sfd f
fad
dsfsfd
fds
little semantics!
questions? </SemanticDesktop> </rdf:RDF>
Leo Sauermann
Dominik Heim
Thanks to Jesus, Ingrid Brunner-Sauermann, Heiko Maus, Sven Schwarz, Ansgar Bernardi, Andreas Dengel, Frank Osterfeld, Dominik Heim, Man Luo, Jeen Broekstra, Giovanni Tummarello, Michael Zeltner, Stephan Baumann, Gunnar A Grimnes, Ludger Van Elst, Harald Holz, Stefan Decker, Malte Kiesel, timbl and the bunch from #swig