1. 2002.08.27 - SLIDE 1IS 202 - Fall 2002 Course Introduction Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday 10:30 am - 12:00 am Fall 2002 SIMS 202: Information Organization and Retrieval Credits to Marti Hearst for some of the slides in this lecture

2. 2002.08.27 - SLIDE 2IS 202 - Fall 2002 Today Introductions Course overview Administrivia

3. 2002.08.27 - SLIDE 3IS 202 - Fall 2002 Goals of the Course Learn about –Design, development and use of information storage and retrieval systems –Practical and theoretical foundations of information organization and analysis –Evaluation of information access systems –Cognitive and user-centric considerations –Hands-on experience with information systems

4. 2002.08.27 - SLIDE 4IS 202 - Fall 2002 Two Main Themes Information Organization and Design Information Retrieval and the Search Process

5. 2002.08.27 - SLIDE 5IS 202 - Fall 2002 Information Organization and Retrieval To organize is to (1) furnish with organs, make organic, make into living tissue, become organic; (2) form into an organic whole; give orderly structure to; frame and put into working order; make arrangements for. Knowledge is knowing, familiarity gained by experience; person’s range of information; a theoretical or practical understanding of; the sum of what is known. To retrieve is to (1) recover by investigation or effort of memory, restore to knowledge or recall to mind; regain possession of; (2) rescue from a bad state, revive, repair, set right. Information is (1) informing, telling; thing told, knowledge, items of knowledge, news. The Oxford English Dictionary, cf. Rowley

6. 2002.08.27 - SLIDE 6IS 202 - Fall 2002 (Approximate) Course Schedule Organization –Overview –Categorization –Metadata and markup –Metadata for multimedia Photo project –Controlled vocabularies, classification, thesauri –Information design Thesaurus design Database design

7. 2002.08.27 - SLIDE 7IS 202 - Fall 2002 Information Properties Information can be communicated electronically –Broadcasting –Networking Information can be easily duplicated and shared –Problems of ownership –Problems of control Adapted from ‘Silicon Dreams’ by Robert W. Lucky

8. 2002.08.27 - SLIDE 8IS 202 - Fall 2002 Information Hierarchy Wisdom Knowledge Information Data

9. 2002.08.27 - SLIDE 9IS 202 - Fall 2002 Information Hierarchy Data –The raw material of information Information –Data organized and presented by someone Knowledge –Information read, heard or seen and understood Wisdom –Distilled and integrated knowledge and understanding

10. 2002.08.27 - SLIDE 10IS 202 - Fall 2002 Information Where is the Life we have lost in living? Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information? -- T.S. Eliot, “The Rock” Where is the information we have lost in data?

11. 2002.08.27 - SLIDE 11IS 202 - Fall 2002 Information Life Cycle Creation UtilizationSearching Active Inactive Semi-Active Retention/ Mining Disposition Discard Using Creating Authoring Modifying Organizing Indexing Storing Retrieval Distribution Networking Accessing Filtering

12. 2002.08.27 - SLIDE 12IS 202 - Fall 2002 Authoring/Modifying Converting data+information+knowledge to new information Creating information from observation, thought Editing and publication Gatekeeping

13. 2002.08.27 - SLIDE 13IS 202 - Fall 2002 Organizing/Indexing Collecting and integrating information Affects data, information and metadata “Metadata” describes data and information –More on this later Organizing information –Types of organization? Indexing

14. 2002.08.27 - SLIDE 14IS 202 - Fall 2002 Storing/Retrieving Information storage –How and where is information stored? Retrieving information –How is information recovered from storage –How to find needed information –Linked with accessing/filtering stage

15. 2002.08.27 - SLIDE 15IS 202 - Fall 2002 Distribution/Networking Transmission of information –How is information transmitted? Networks vs. broadcast

16. 2002.08.27 - SLIDE 16IS 202 - Fall 2002 Accessing/Filtering Using the organization created in the O/I stage to: –Select desired (or relevant) information –Locate that information –Retrieve the information from its storage location (often via a network)

17. 2002.08.27 - SLIDE 17IS 202 - Fall 2002 Using/Creating Using information Transformation of information to knowledge Knowledge to new data and new information

18. 2002.08.27 - SLIDE 18IS 202 - Fall 2002 Key Issues in This Course How to describe information resources or information-bearing objects in ways so that they may be effectively used by those who need to use them –Organizing How to find the appropriate information resources or information-bearing objects for someone’s (or your own) needs –Retrieving

19. 2002.08.27 - SLIDE 19IS 202 - Fall 2002 Key Issues Creation UtilizationSearching Active Inactive Semi-Active Retention/ Mining Disposition Discard Using Creating Authoring Modifying Organizing Indexing Storing Retrieval Distribution Networking Accessing Filtering

20. 2002.08.27 - SLIDE 20IS 202 - Fall 2002 (Approximate) Course Schedule Organization –Overview –Categorization –Metadata and markup –Metadata for multimedia Photo Project –Controlled vocabularies, classification, thesauri –Information design Thesaurus design Database design Retrieval –The search process –Content analysis Tokenization, Zipf’s law, lexical associations –IR implementation –Term weighting and document ranking Vector space model –User interfaces Overviews, query specification, providing context

21. 2002.08.27 - SLIDE 21IS 202 - Fall 2002 Web Search Questions What do people search for? How do people use search engines? –How often do people find what they are looking for? –How difficult is it for people to find what they are looking for? How can search engines be improved?

22. 2002.08.27 - SLIDE 22IS 202 - Fall 2002 What Do People Search for on the Web? Study by Spink et al., Oct 98 –www.shef.ac.uk/~is/publications/infres/paper53.html –Survey on Excite, 13 questions –Data for 316 surveys

23. 2002.08.27 - SLIDE 23IS 202 - Fall 2002 What Do People Search for on the Web? Topics Genealogy/Public Figure:12% Computer related:12% Business:12% Entertainment: 8% Medical: 8% Politics & Government 7% News 7% Hobbies 6% General info/surfing 6% Science 6% Travel 5% Arts/education/shopping/images 14% Something is missing…

24. 2002.08.27 - SLIDE 24IS 202 - Fall 2002 What Do People Search for on the Web? 4660 sex 3129 yahoo 2191 internal site admin check from kho 1520 chat 1498 porn 1315 horoscopes 1284 pokemon 1283 SiteScope test 1223 hotmail 1163 games 1151 mp3 1140 weather 1127 www.yahoo.com 1110 maps 1036 yahoo.com 983 ebay 980 recipes 50,000 queries from excite 1997 Most frequent terms:

25. 2002.08.27 - SLIDE 25IS 202 - Fall 2002 Why Do These Differ? Self-reporting survey The nature of language –Only a few ways to say certain things –Many different ways to express most concepts UFO, Flying Saucer, Space Ship, Satellite How many ways are there to talk about history?

26. 2002.08.27 - SLIDE 26IS 202 - Fall 2002 65002930 the 62789720 a 60857930 to 57248022 of 54078359 and 52928506 in 50686940 s 49986064 for 45999001 on 42205245 this 41203451 is 39779377 by 35439894 with 35284151 or 34446866 at 33528897 all 31583607 are 30998255 from 30755410 e 30080013 you 29669506 be 29417504 that 28542378 not 28162417 an 28110383 as 28076530 home 27650474 it 27572533 i 24548796 have 24420453 if 24376758 new 24171603 t 23951805 your 23875218 page 22292805 about 22265579 com 22107392 information Source: http://elib.cs.berkeley.edu/docfreq/index.html What is on the Web?

27. 2002.08.27 - SLIDE 27IS 202 - Fall 2002 Intranet Queries (Aug 2000) 3351 bearfacts 3349 telebears 1909 extension 1874 schedule+of+classes 1780 bearlink 1737 bear+facts 1468 decal 1443 infobears 1227 calendar 989 career+center 974 campus+map 920 academic+calendar 840 map 773 bookstore 741 class+pass 738 housing 721 tele-bears 716 directory 667 schedule 627 recipes 602 transcripts 582 tuition 577 seti 563 registrar 550 info+bears 543 class+schedule 470 financial+aid

28. 2002.08.27 - SLIDE 28IS 202 - Fall 2002 Intranet Queries Summary of sample data from 3 weeks of UCB queries –13.2% Telebears/BearFacts/InfoBears/BearLink (12297) –6.7% Schedule of classes or final exams (6222) –5.4% Summer Session (5041) –3.2% Extension (2932) –3.1% Academic Calendar (2846) –2.4% Directories (2202) –1.7% Career Center (1588) –1.7% Housing (1583) –1.5% Map (1393) Average query length over last 4 months: 1.8 words This suggests what is difficult to find from the home page

29. 2002.08.27 - SLIDE 29IS 202 - Fall 2002 An Example Search System: Cha-Cha A system for searching complex intranets Places retrieval results in context Important design goals: –Users at any level of computer expertise –Browsers at any version level –Computers of any speed

30. 2002.08.27 - SLIDE 30IS 202 - Fall 2002 30 An Example Search System: Cha-Cha

31. 2002.08.27 - SLIDE 31IS 202 - Fall 2002 Search: Where to Start? Guess words? –Search engine plunges you into the middle of a site/collection –Too many or too few results –No context Use a directory? –If large, may be difficult/frustrating to navigate –Several ways to organize the information –May not reflect users’ needs Solution: Integrate browsing and search –How do you organize the information to optimize searching?

32. 2002.08.27 - SLIDE 32IS 202 - Fall 2002 32 An Example Search System: Cha-Cha

33. 2002.08.27 - SLIDE 33IS 202 - Fall 2002 33 An Example Search System: Cha-Cha

34. 2002.08.27 - SLIDE 34IS 202 - Fall 2002 How Cha-Cha Works Crawl entire intranet Compute the shortest hyperlink path from a certain root page to every web page Index and compute metadata for the pages –Using Cheshire II –Run a user query –Gather all the hits –Create a “directory” based on combining the shortest paths –Special graph algorithm removes redundant links and internal nodes

35. 2002.08.27 - SLIDE 35IS 202 - Fall 2002 35 Cha-Cha System Architecture crawl the web store the documents

36. 2002.08.27 - SLIDE 36IS 202 - Fall 2002 36 Cha-Cha System Architecture crawl the web store the documents create files of metadata Cheshire II

37. 2002.08.27 - SLIDE 37IS 202 - Fall 2002 37 Cha-Cha System Architecture crawl the web create a keyword index store the documents create files of metadata Cheshire II

38. 2002.08.27 - SLIDE 38IS 202 - Fall 2002 38 Cha-Cha System Architecture Cheshire II user query Searching

39. 2002.08.27 - SLIDE 39IS 202 - Fall 2002 39 Cha-Cha System Architecture Cheshire II server accesses the databases Searching

40. 2002.08.27 - SLIDE 40IS 202 - Fall 2002 40 Cha-Cha System Architecture Cheshire II results shown to user Searching

41. 2002.08.27 - SLIDE 41IS 202 - Fall 2002 41 Cha-Cha System Architecture Cheshire II results shown to user server accesses the databases user query Searching

42. 2002.08.27 - SLIDE 42IS 202 - Fall 2002 What Hasn’t Been Explained Here? What metadata is collected How the indexes are created How queries are formed How documents are ranked How shortest paths are computed How the system is built –… among other things! –This is just an introduction! Much more on these issues in the second half of the course

43. 2002.08.27 - SLIDE 43IS 202 - Fall 2002 (Approximate) Course Schedule Organization –Overview –Categorization –Metadata and markup –Metadata for multimedia Photo Project –Controlled vocabularies, classification, thesauri –Information design Thesaurus design Database design Retrieval –The search process –Content analysis Tokenization, Zipf’s law, lexical associations –IR implementation –Term weighting and document ranking Vector space model –User interfaces Overviews, query specification, providing context

44. 2002.08.27 - SLIDE 44IS 202 - Fall 2002 Assignments and Exams Approximately 10 assignments (due within one week to ten days) –Sometimes “checked”, sometimes graded Final exam (during Finals week) Grading: –Assignments: 60% Not evenly weighted –Final: 25% –Class Participation: 15%

45. 2002.08.27 - SLIDE 45IS 202 - Fall 2002 Readings Course reader –Will be available in about a week (will announce) –Textbooks Modern Information Retrieval, Baeza-Yates and Ribiero-Neto (Eds.), Addison Wesley, 1999 The Organization of Information, Arlene G. Taylor, Libraries Unlimited, 1999,

46. 2002.08.27 - SLIDE 46IS 202 - Fall 2002 Homework (!) Read the handouts –Borges, Dennett, and Reddy Write one or two paragraphs on –What is information, according to your background or area of expertise? Due in class this Thursday, Aug 30.

47. 2002.08.27 - SLIDE 47IS 202 - Fall 2002 What is Information? There is no “correct” definition Can involve philosophy, psychology, signal processing, physics Cookie Monster’s definition: – “news or facts about something” Oxford English Dictionary –information: informing, telling; thing told, knowledge, items of knowledge, news –knowledge: knowing familiarity gained by experience; person’s range of information; a theoretical or practical understanding of; the sum of what is known

48. 2002.08.27 - SLIDE 48IS 202 - Fall 2002 Next Time Introduction to the Photo Project More on what is information? And how much of it is out there?