Information Retrieval Liam Quin, Barefoot Computing, Toronto

Agenda Overview of Information Retrieval What people want, and how to give it to them Things people dont know they want, and how to do them

Chapter One: The Problem gooseberry

Gooseberry Picking Hurts Gooseberries have thorns. Gooseberry pickers in Botswana might not wear shirts (or shoes). When you pick one gooseberry, others fall to the ground. The harvest would be improved if we could retrieve the fallen fruit safely. There are texts on this on the Internet.

Searching for an Answer Search for information on texts about gooseberry retrieval on the web.

The Result Pages on text retrieval on information retrieval......and, at the top of the list... Cycling in Cape Gooseberry, Labrador

Lessons Indexes on words alone arent enough Word order can be important Relevance Ranking is often bogus Sometimes you have to wear shirt and shoes.

How can we Improve? Better textual analysis Word order Context Metadata External categorisation (RDF, Topic Maps) Grow thornless gooseberries

Better Textual Analysis Part of speech information during indexing Stemming (boy/boys, foot/feet, run/running, me/mine) Record more in index (caps, separation) Co-location analysis (mine next to gold) Ask the User what she means in the query (mine as in of me, or as in quarry?) Thesaurus Expansion of queries

Word Order Give added weight to word order: information retrieval vs. retrieval of information Times Square vs. square times include all words (What If Inc., The Times)

Context Co-location of words helps disambiguate The xml containing element Feedback from nearby documents (e.g. on the same website, or in the same chapter or publication) Domain-specific information at index- time

Metadata Add information to documents Dublin Core (e.g. Warwick Framework) The html meta element The html rel/rev attributes in links

External Categorisation Use xml schemas to add context information Document or site-wide information Resource Description Framework Topic Maps (iso 13250) Categorise the result set [see picture]

Grow thornless gooseberries Sometimes its easier to change the problem than to solve it as stated. Sometimes people dont describe the problem that they need solved. Sometimes its easier to solve a more general problem (thornless fruit? Or padded shirts)

What Most People Want Find this string or phrase in this element. Thats all most people ask for. Its all they want. But its hardly ever all they need.

The real needs Needs of other staff Executives who understand the problem Indirect needs –internal use by software –other departments –private uses by sneaky employees –enabling technologies change perspectives

I didnt know I could... Quality control –check for known errors –find unusual words or phrases –phrases not marked up Analysis –look for unusual markup –co-location (phrase summary)

Oooh, can you really? Automatic linking –Glossary –Glossary Index page –Dictionary Samples Add markup automatically –based on phrases in context

Summery Summary You may need more than you thought… …but it might do more than you expected… …but...

There is no gooseberry pie for lunch.

Liam Quin Barefoot Computing Toronto