1 © disclosure or duplication without consent prohibited COMET K2 Competence Center - Initiated by the Federal Ministry of Transport, Innovation & Technology (BMVIT) and the Federal Ministry of Economics & Labour (BMWFI). Funded by FFG, Land Steiermark and Steirische Wirtschaftsf ö rderung (SFG) Is Enterprise Search useful at all? Lessons Learned from Studying Usage behavior. I-KNOW 2014 Graz, Austria Dr. Alexander Stocker

2 © disclosure or duplication without consent prohibited  Intro  User Satisfaction  User Evaluation Enterprise Search Agenda

3 © disclosure or duplication without consent prohibited Enterprise Search Business Case Findability Survey

4 © disclosure or duplication without consent prohibited  Huge demand from industry  Enterprise Search is becoming more and more popular  EU-Market for EU Enterprise Saerch vendors is between 100 and 200 Mio. €  Lack of application oriented research  Lack of academic case studies  Lack of academic best practices  Lack of academic user evaluations  Is the topic Enterprise Search in the hand of practicioners, only? Enterprise Search Introduction

5 © disclosure or duplication without consent prohibited  Enterprise search includes any organization with text content in electronic form,  search in an organization’s external website,  search in the organization’s internal websites (i.e. its intranet), and  search in other electronic text held by the organizations in the form of , database records, documents on fileshares, etc. („Challenges in Enterprise Search“, Hawking, 2004) Enterprise Search Definition

6 © disclosure or duplication without consent prohibited Enterprise Search Satisfaction Mindmetre: Report „Minding the Search Gap“

7 © disclosure or duplication without consent prohibited Enterprise Search Satisfaction Findability Survey

8 © disclosure or duplication without consent prohibited  Web Seach  Interlinked Web pages. Standard formats  Relevance of a web page is (initially) calculated by assessing, how many (good) links point to it  Information on the Web wants to be found. Hence content is optimized for search.  Web search has indexed open content, access rights are not relevant  Searcher is satisfied with a good answer (out of many possible good answers). Search queries are of a more common nature.  … Search on the Web vs. Search in the Enterprise  Enterprise Search  Heterogeneous content (structured and unstructured)  No links between the documents. Different ways of relevance assessment needed  Users are not motivated to optimize their documents for search  Complex group & roles structures. Access rights first, information access second  In most cases, there is only one document relevant for a user. This has to be „found“ as quick as possible. ..

9 © disclosure or duplication without consent prohibited  Case Study  Research organization in the automotive domain, employing about 200 people  Enterprise Search Pilot to find information in project-relevant documents on a fileshare (Microsoft Sharepoint Foundation 2013)  Pilot study with 10 participants to identify user-centric aspects of Enterprise Search implementation  Evaluation  Short structured interviews (tasks, documents and tools, information demand, search-scope and time)  Definition of possible & realistic search tasks  Participants performed these search tasks afterwards  Participants had to think aloud what they did and why they did it  Transcription and analysis of interviews  Identification of commonalities and differences Enterprise Search User Evaluation

10 © disclosure or duplication without consent prohibited Enterprise Search (Microsoft Sharepoint Foundation 2013)

11 © disclosure or duplication without consent prohibited  Finding the right keywords is crucial, but very challenging  Better keywords lead to more relevant search results  Thinking process for finding the best keywords is very challenging  Explication of a personal information demand for a search engine is a challenge Evaluation results Keyword selection and query formulation

12 © disclosure or duplication without consent prohibited  The benefit of metadata for search gets lost, if metadata is not administered in organizational practice  Metadata is used in search to filter search results („facets“)  Missing or/and false metadata reduces search quality (e.g. the shown author is not the correct author of a document, but the creator of its template)  Users lack motivation to provide correct & useful metadata, as they do not perceive any benefit Evaluation results Usage of metadata

13 © disclosure or duplication without consent prohibited  Consistent document content (formulations) would increase perceived usefulness of search  The availability of essential terms in documents increases search success (e.g. authors, location, date, consistent project titles, MOM vs. meeting minutes,..)  Knowledge about document content and writing practices is crucial (terms, syntax, semantics, …)  Common writing practices of essential project-information in documents have to be found, first, otherwise searchers may always have to guess (“..what terms could be used in the document containing the information I seek..”) Evaluation results Document Content

14 © disclosure or duplication without consent prohibited  The relevance assessment of a search engine does not necessarily meet the user needs  The search engine calculates relevance displayed in the hit list based on the appearance of query terms within documents  The ranking is crucial for perceived usefulness of search. Employees requested other mechanisms, e.g. ranking in term of up-to-dateness Evaluation results Relevance

15 © disclosure or duplication without consent prohibited  Enterprise Search has to compete against existing individual information seeking practices (search alternatives).  So far, anything had to be „found“ without enterprise search. Employees (may) have created individual search strategies.  Face-2-Face conversations between colleagues are the number one tool for information acquisition (e.g. the location of a document is gained by asking a colleague who knows it)  Users know the structure of project-folders, they browse instead of search  Users take advantage of links to projects, project documents and others via their desktop or via messenger/skype etc. Evaluation results Search Alternatives

16 © disclosure or duplication without consent prohibited  Enterprise Search has to immediately provide an added value from a user‘s perspective for daily work.  Enterprise Search can require more time to find information compared to browsing through (known) folder structures  Finding the right office-document was challenging, because of the high number of similar documents in different versions (e.g. V21_meeting_minute_xy)  User satisfaction was especially very low, when keywords were not available in documents Evaluation results Overall usefulness and satisfaction

17 © disclosure or duplication without consent prohibited 1.Enterprise Search Introduction 2.Enterprise Search Satisfation 3.Enterprise Search Potentials & Pitfalls  Pioneering work to motivate other researches studying user-aspects of enterprise search (adoption)  Call for Research Cases: Enterprises having a successful enterprise search implementation with a very high user satisfaction are warmly welcome Enterprise Search Conclusion

18 © disclosure or duplication without consent prohibited Kontakt Dr. Alexander Stocker VIRTUAL VEHICLE