Knowledge Architecture in the Enterprise 2.0 Tom Reamy Chief Knowledge Architect KAPS Group Knowledge Architecture Professional Services

2 Agenda  Introduction  Web 2.0: Folksonomies  Search, Content Management, Text Analytics  Conclusion

3 Folksonomies – Wikipedia Definition  Folksonomy (also known as collaborative tagging, social classification, social indexing, and social tagging) is the practice and method of collaboratively creating and managing tags to annotate and categorize content. Folksonomy describes the bottom-up classification systems that emerge from social tagging. [1] In contrast to traditional subject indexing, metadata is generated not only by experts but also by creators and consumers of the content. Usually, freely chosen keywords are used instead of a controlled vocabulary. [2] Folksonomy (from folk + taxonomy) is a user generated taxonomy.collaborativelytagscategorize content [1]subject indexingkeywordscontrolled vocabulary [2] taxonomyuser generated

4 Web 2.0 – No need for Taxonomies etc.?  “ Tags are great because you throw caution to the wind, forget about whittling down everything into a distinct set of categories and instead let folks loose categorizing their own stuff on their own terms." - Matt Haughey - MetaFilter  Tyranny of the majority - worst type of central authority  More Madness of Crowds than Wisdom of Crowds  “Things fall apart; the center cannot hold; Mere anarchy is loosed upon the world,… The best lack all conviction, while the worst Are full of passionate conviction.” - The Second Coming – W.B. Yeats

5 Advantages of Folksonomies  Simple (no complex structure to learn) – No need to learn difficult formal classification system  Lower cost of categorization – Distributes cost of tagging over large population  Open ended – can respond quickly to changes  Relevance – User’s own terms  Support serendipitous form of browsing  Easy to tag any object – photo, document, bookmark  Better than no tags at all  Getting people excited about metadata!

6 Folksonomies – Problems and Limits  Folksonomies don’t compare with taxonomies or ontologies  Serendipity browsing is small part of search  Limited areas of success – popular sites are popular  Quality Content – finance, science, etc – not good candidates  No mechanism for improving folksonomies  Scale – Too Big (million hits) – Too Little (200 items) – Amazon and LibraryThing  Need intrinsic value of tagging – not tagging for better tags  Bad Tags - idiosyncratic or too broad, errors, limited reach – Most people can’t tag very well – learned skill

7 Del.icio.us Tags  Design blog software music tools reference art video programming webdesign web2.0 mac howto linux tutorial web free news photography shopping blogs css imported education travel javascript food games  Development inspiration politics flash apple tips java google osx business windows iphone science productivity books toread helath funny internet wordpress ajax ruby research humor fun technology search opensource  Photoshop media recipes cool work article marketing security mobile jobs rails lifehacks tutorials resources php social download diy ubuntu freeware portfolio photo movies writing graphics youtube audio online

8 Del.icio.us - Folksonomy Findability  Too many hits (where have we heard that before?) – Design – 1 Mil, software – 931,259, sex – 129,468  No plurals, stemming (singular preferred) – Folksonomy – 14,073, folksonomies – 3,843, both – 1,891 – Blog-1.7M, blogs – 516,340, Weblog- 155,917, weblogs – 36,434, blogging – 157,922, bloging – 697 – Taxonomy – 9.683, taxonomies – 1,574  Personal tags – cool, fun, funny, etc – Good for social research, not finding documents or sites – How good for personal use? Funny is time dependent

9 Library Thing  Book people aren’t much better at tagging  High level concepts – psychology (55,000), religion (120,000), science (101,000)  Issue – variety of terms – cognitive science – need at least 40 other tags to cover the actual field of cognitive science  Strange tags – book (19,000) – it’s a book site?  Combination of facets and topics – Facets – Date (16 th century, 1950’s, 2007) // Function (owned, not read) // Type (graphic novel, novel) // Genre (horror, mystery) – Topics – majority like Del.icio.us

10 Library Thing – Book on Neuroscience  1) (Location: dining room)(1) biological(1) biology(8) box74(1) Brain(1) brain research(1) brains(1) cognitive neuroscience(1) cognitive science(1) consciousness(1) currently reading(1) HelixHealth(1) kognitionswissenschaft(1) medical(1) medicine(1) neuroscience(19) non-fiction(5) partread(1) Psychology(4) Science(10) textbook(10) theory(1)(Location: dining room)biologicalbiology box74Brainbrain researchbrainscognitive neurosciencecognitive scienceconsciousness currently readingHelixHealth kognitionswissenschaftmedicalmedicine neurosciencenon-fictionpartreadPsychology Sciencetextbooktheory  Too General: Science, Psychology, biology, textbook  Too specific: Location: dining room, box74  Facets: currently reading, partread

11 Better Folksonomies:  Will social networking make tags better?  Not so far – example of Del.icio.us – same tags  Quality and Popularity are very different things  Most people don’t tag, don’t re-tag  Study – folksonomies follow NISO guidelines – nouns, etc – but do they actually work – see analysis  Most tags deal with computers and are created by people that love to do this stuff – not regular users and infrequent users – Beware true believers!

12 Knowledge Architecture : Social Networking  KM 2.0 (or 3.0?) – KM always concerned with social aspects of knowledge  New relationship of center and users – more sophisticated support, more freedom, more suggestions, more user input – - New roles – for users (taggers, part of variety of communities – both distributed and central) – New roles for central – create feedback system, tweak the evolution of the system, Develop initial candidates  Communities of Practice – apply to tagging, ranking – Community Maps – formal and informal – Map tags to communities – more useful suggestions – Use tags to uncover communities (see tech SNA)

13 Knowledge Architecture: Technology for Web 2.0  Enterprise Content Management – Place to add metadata – of all kinds, not just keywords – Policy support – important, part of job performance – Add tag clouds to input page – More sophisticated displays Tag clouds mapped to community map Tag clusters, taxonomy location  Semantic Software – Inxight, Teragram etc. – Suggest terms based on text, on tag clouds  Social Networking – add semantics – SNA – apply to people and tags  Wiki – more powerful than blogs, more work to set up & maintain

14 Knowledge Architecture: Putting it all together Complexity Theory and Folksonomies: Feedback  Ranking Methods – Explicit – people rank directly Categories, tags, taggers Good tags, best bets for terms or categories? – Implicit – software evaluation, reverse relevance  Ranking Roles – Taggers – everyone (rewards, make it easy and fun) – Meta-taggers – everyone (but levels of meta-taggers) – Editors – tagging system, integration with taxonomy, resolve disputes, Wikipedia model

15 Knowledge Architecture: Content Structures – Best of Both Worlds  Start and end with a formal taxonomy / Ontology – Findability vastly superior – Communication with others – share tags – Take advantage of conceptual relationships  Tagging experience – folksonomies plus – Users can type any word – system looks it up – plurals, synonyms, preferred terms, spelling variations – Software suggestions – based on content of bookmark, document and on popular user tags – natural level not top down – New terms flagged and routed to central team  Facets – for both things and documents (faceted taxonomy) – Software suggests facet values, user override – Cognitively simpler task than own value, complex hierarchy

16 Knowledge Architecture in the Enterprise Technology  Text Analytics  Content Management  Search – Browse – Faceted Navigation

17 Varieties of Taxonomy/ Text Analytics Software  Taxonomy Management  Text Analytics – Auto-Categorization, Entity Extraction – Sentiment Analysis  Software Platforms – Content Management, Search  Application Specific – Business Intelligence

Vendors of Taxonomy/ Text Analytics Software  Attensity  Business Objects – Inxight  Clarabridge  ClearForest  Data Harmony / Access Innovations  Lexalytics  Multi-Tes  Nstein  SchemaLogic  Synaptica  Teragram  Wikionomy  Wordmap  Lots More 18

19 Why Taxonomy Software?  If you have to ask, you can’t afford it  Spreadsheets – Good for calculations, days of taxonomy development over – (almost)  Ease of use – more productive – Increase speed of taxonomy development – Better Quality – synonyms, related terms, etc.  Distributed development – lower cost, user input (good and bad)

20 Text Analytics Software – Features  Entity Extraction – Multiple types, custom classes  Auto-categorization – Taxonomy Structure – Training sets – Bayesian, Vector space – Terms – literal strings, stemming, dictionary of related terms – Rules – simple – position in text (Title, body, url) – Boolean– Full search syntax – AND, OR, NOT – Advanced – NEAR (#), PARAGRAPH, SENTENCE  Advanced Features – Facts / ontologies /Semantic Web – RDF + – Sentiment Analysis

21 Current State of Content Management  Content Management – Strong on management, weak on content – Content is a black box – simply moved around  What is missing is the meaning dimension – In-depth and articulated understanding of content  Perceived Solution – Delphi Survey – Taxonomy – 90% plan on taxonomy strategy in 24 months – 76% taxonomy is important  Text Analytics integrated into CM - essential

22 Taxonomic Content Management: Work Flow with Meaning  Preliminary Foundation Work – Design the ontology – Develop taxonomies – Design metadata standards – Collaborative development of controlled vocabularies  Authors, SME’s – check document in: – Have a summary either written by human or software – List of metadata suggestions, entities – people, places, etc. – Provisional categorization – Decision: publish or submit for review, central team or community of experts. – Request for additional keywords or categorization issues

23 Taxonomic Content Management: Work Flow with Meaning  Central Team – Review documents – easier, faster – Use summaries, metadata, entities to provide context – Review infrastructure requests – new keywords, categories  Integrated Work Flow – Strengths of local and central – Variety of roles, flexible (few dedicated roles needed). – Collaborative categorization and keywords by SME, software, and central team SME’s can function as central team

24 Semantics and Search: An Integrated Approach: Elements  Multiple Knowledge Structures – Facet – orthogonal dimension of metadata – Taxonomy - Subject matter / aboutness – Ontology – Relationships / Facts – Subject – Verb - Object  Software - Text analytics, auto-categorization, entity extraction  People – tagging, evaluating tags, fine tune rules and taxonomy  People – Users, social tagging, suggestions

25 Faceted Navigation: Strengths and Weaknesses  Strengths: – More intuitive – easy to guess what is behind each door 20 questions – we know and use – Dynamic selection of categories Allow multiple perspectives – Trick Users into “using” Advanced Search wine where color = red, price = x-y, etc..  Weaknesses: – Difficulty of expressing complex relationships Simplicity of internal organization – Loss of Browse Context Difficult to grasp scope and relationships – Limited Domain Applicability – type and size Entities not concepts, documents, web sites

26

27

28

29

30

31 Integrated Design – Facets & Semantics Design Issues - General  What is the right combination of elements? – Faceted navigation, metadata, browse, search, categorized search results, file plan  What is the right balance of elements? – Dominant dimension or equal facets  Full Facets – Multiple intersecting filters – 1 or 2 filters (source / type) – No  When to combine search, topics, and facets? – Search first and then filter by topics / facet – Browse/facet front end with a search box

32 Integrated Design – Facets & Semantics Design Issues - General  Good Information Architecture – Space wars – summary or full facet display – Simplicity vs. research power – Source and Type are basics – Standard Facets – People, Companies, Place, Industry – Interactive interface – sliders, date ranges  Semantics still hardest – summaries, related, rank  Taxonomy – just another facet? – Keywords vs. simple taxonomy  Tag Clouds / Clusters – how useful?  Feedback – numbers of stories vs. top stories

33 Conclusion  Web 2.0 is not the answer.  The answer is:  A knowledge architecture that integrates web 2.0 with: – Knowledge structures like taxonomies, facets, and basic level categories – Text analytics – categorization, entity extraction – Content Management with meaning – KA team that works with distributed groups in a variety of ways – A smart search that builds on a Knowledge Architecture that includes taxonomies, faceted metadata, best bets and search logs

34 Resources  Information Today – Enterprise Search Sourcebook – 2008  Knowledge Architecture Approach to Search – Tom Reamy – Enterprise Search Sourcebook – 2009  CMS Watch – Content Management –  Text Analytics Software – Information Today –  Web Sites – Taxonomy Community of Practice: