1. LIS618 lecture 0 Thomas Krichel 2003-09-14

2. today's lecture I will not talk about the strike. A look at the course home page http://wotan.liu.edu/home/krichel/lis618n03a administrative stuff historical matters about the course about me business of database searching indexes the Boolean information retrieval model practice example on Dialog

3. Organization homepage http://wotan.liu.edu/home/krichel/lis618n03a Contents to be discussed today. Send mail to krichel@openlib.orgkrichel@openlib.org –Your name –Your secret word for grades delivery Interrupt me with as many questions as possible! Ask for breaks!

4. Proposed Organization Normal lecture Quiz at the beginning of every lecture –Factually oriented, around 15 minutes –Remove worst performance –Average to form 50% Search exercise 50% Formal syllabus to be made early next week!

5. Search exercise find victim of an information need best to take someone you know in a professional capacity conduct interview about an information need experienced by the victim, write down expectations search in formal database and on web discuss results with the victim write essay, no longer than 7 pages.

6. about the course This course is new wine in an old bottle Officially a merger of –lis566 information resources on the Internet mailing lists usenet news web searching –lis618 database searching access and use of commercial databases

7. mix of theory and practice I am not a database search practitioner. Each database is different, practical skills are not easily transferable. Thus my emphasis in the course is more on theory. In the past, I theory first, then practice. This year I will try to mix. Some theory and some practice in every session.

8. What databases? Dialog has been the traditional database covered. –They were the market leaders in online databases in the past. –Nowadays the field is much more open In addition I have done Nexis, FirstSearch (OCLC) in the past. But I am open to suggestions.

9. About me Born 1965, in Völklingen (Germany) Studied economics and social sciences at the Universities of Toulouse, Paris, Exeter and Leiceister. PhD in theoretical macroeconomics Lecturer in Economics at the University of Surrey 1993 and 2001 Since 2001 assistant professor at the Palmer School

10. Why? During research assistantship period, (1990 to 1993) I was constantly frustrated with difficult access to scientific literature. At the same time, I discovered easy access to freely downloadable software over the Internet. I decided to work towards downloadable scientific documents. This lead to my library career (eventually).

11. Steps taken I 1993 founded the NetEc project at http://netec.mcc.ac.uk, later available at http://netec.ier.hit-u.ac.jp as well as at http://netec.wustl.edu. http://netec.mcc.ac.uk http://netec.ier.hit-u.ac.jp http://netec.wustl.edu These are networking projects targeted to the economics community. The bulk is –Information about working papers –Downloadable working papers –Journal articles were added later

12. Steps taken II Set up RePEc, a digital library for economics research. Catalogs –Research documents –Collections of research documents –Researchers themselves –Organizations that are important to the research process Decentralized collection, model for the open archives initiative

13. Steps taken III Co-founder of Open Archives Initiative Work on the Academic Metadata Format Co-founded rclis, a RePEc clone for (Research in Computing, Library and Information Science)

14. Interest in databases From my point of view I have two interests in database searching –As a provider, I must understand how people search in order to provide some data that they can use and will use. –As an economist, I have a strong interest in information as a commodity. The database market is an important market place. Main emphasis of course is still on databases.

15. Database searching (DS) subset of the subject of information retrieval (IR) DS mainly thought as applicable to the set of large structured databases as opposed to do web searching for those, a general knowledge of what databases are seems useful Concentrate on textual databases

16. traditional social model user goes to a library describes problem to the librarian librarian does the search –without the user present –with the user present hands over the result to the user user fetches full-text or asks a librarian to fetch the full text.

17. economic rational for traditional model In olden days the cost of telecommunication was high. database use costs –cost of communication –cost of access time to the database the traditional model controls an upper bound on costs

18. disintermediation with access cost time gone, the traditional model is under threat there is disintermediation where the librarian looses her role but that may not be good news for information retrieval results –user knows subject matter best –librarian knows searching best

19. Web searching IR has received a lot of impetus through the web, which poses unprecedented search challenges. with more and more data appearing on the web DS may be a subject in decline –it is primarily concerned with non-web databases –There is more and more web-based methods of searching

20. Public access vs quality Now the public at large is able to do online searching. At the same time need for quality answers has grown. Quality-filtered services will become more important. In the current databases, there is as lot that would already be available for free mixed with quality-controlled stuff. Publishers have direct offerings and intermediated vending is in decline.

21. Main theory part Literature: "Modern Information Retrieval" by Ricardo Baeza-Yates and Berthier Ribiero-Neto Don't buy it. It is a not a good book.

22. before the IR process provider –define data that is available documents that can be used document operations document structure –index user –user need –IR system familiarity

23. the IR process query expresses user need in a query language processing of query yields retrieved documents calculation of relevance ranking examination of retrieved documents possible relevance cycle

24. main problem user is not an expert at the formulation of a query garbage in garbage out, the retrieval yields poor result ways out –design very intuitive interface for the query –give expert guidance

25. taxonomy of classic IR models Boolean, or set-theoretic –fuzzy set models –extended Boolean vector, or algebraic –generalized vector model –latent semantic indexing –neural network model probabilistic –inference network –belief network

26. summary There are three basic types of models in classic information retrieval. Extensions of these types are a matter of research concern and require good mathematical skills. All classic models treat document as individual pieces.

27. key aid: index an index is a list of terms, with a list of locations where the term is to be found. The way to express locations usually depends on the form that the indexed data takes. –for a book, it is usually the page number, e.g. "shmoo 34, 75" –for computer files it is usually the name of the file plus the number of the byte where the indexed term starts, e.g. "krichel index.html 34, cv.html 890 1209" there is usually more than one location of the term.

28. key aid: index terms index term is a part of the document that has a meaning on its own. it is usually a noun word. retrieval based on index term raises questions –semantics in query or document is lost –matching done in imprecise space of index terms predicting relevance is a central problem the IR model determines the process of relevance ranking

29. basic concept: weight of index term given all nouns, not all appear to have the same relevance to the text sometimes, we can have a simple measure of the importance of a term, example? more generally, for each indexing term and each document we can associate a weight with the term and the document. usually, if the document does not contain the term, its weight is zero

30. Boolean model in the Boolean model, the index weight of all index term for any document is 1 if the term appears in the document. It is 0 otherwise. This allows to combine query terms with Boolean operator AND, OR, and NOT thus powerful queries can be written

31. Classic implementation: dialog http://training.dialog.com/sem_info/courses/ pdf_sem/dlg1.pdf http://training.dialog.com/sem_info/courses/ pdf_sem/dlg2.pdf http://training.dialog.com/sem_info/courses/ pdf_sem/dlg3.pdf http://training.dialog.com/sem_info/courses/ pdf_sem/dlg4.pdf

32. Dialog is a databank over 500 databases these are also known as files and cover – references and abstracts for published literature, – business information and financial data; –complete text of articles and news stories; –statistical tables –Directories DIALOG uses the Boolean model

33. DIALOG interface is still rooted in "traditional" database systems dismissed as "dial-a-dog" is uses a command-driven interface it is very complicated to learn fully it is not suitable for the end-user it therefore offers a valuable skill to the information professional it is a challenge for a professor to teach

34. Accessing DIALOG On the web, go to http://www.dialogweb.com/ Enter username and password Forget about subaccount then click on logon On the next screen go to command search "continue" at the next screen

35. two steps in DIALOG step one: select databases (aka files) to look at step two: perform searches on the selected databases You may wonder why one does not have one single step like in a search engine. Discuss.

36. sample search We want to know something about "current awareness in digital libraries" From dialogweb command search: –databases –social sciences and humanities –library and information science leads you to http://www.dialogweb.com/cgi/logoff?mode= guided&url=/cgi/dwframe?href=search.html

37. This is database selection… At that screen you see a number of "files" with their number. You can select those that you want to search then you click "begin datasbase" and you get back to the command search "b numbers" it will say. That is the command to begin working with files.

38. Boolean seach Do a number of searches –s current(N)awarness –s digital(N)library –s digital(N)libraries Each search retrieves a set of documents The sets can be combined –s s1 and (s2 or s3)

39. What is the deal? There are two stages. At stage two we make Boolean queries. Each query splits the the records into matching and non-matching records. The set of matching records is return. It can be further searched or combined with other sets using Boolean operators. Try this at home.

40. http://openlib.org/home/krichel Thank you for your attention!