Web-based Information Architectures MSEC Mini II Location:GSIA Simon Auditorium Time:1:30-3:20pm, Tues. & Thurs. Instructor:Prof. Jaime Carbonell Office: NSH Tel: [Augmented with expert guest lectures] Teaching assistant:Jian Zhang Office: NSH Tel: Offices Hours: TBD Administrative assistant: TBD Office: NSH Tel:

Administrative Issues Prerequisites Basic programming skills (preferably JAVA) Familiarity with the web (HTML, browsing, etc.) Fundamentals of Web Programming (20-753). Grading 30% homeworks (2 programming assignments) 30% miniproject (student teams will propose) 15% midterm (5 pages notes, calculator OK, no laptops) 25% final (10 pages notes, calculator OK, no laptops) Bulletin Board Schedule/syllabus Lecture notes (in powerpoint) Homework Announcements & discussions

Textbook and Reference Materials (1) Required: Class notes (slides on web site) and handouts (to be provided) Required: "Understanding Search Engines: Mathematical Modeling and Text Retrieval" by Michael W. Berry, Murray Browne Available at (tel: ) Optional: Background reading material provided

Textbook and Reference Materials (2) Optional: "Advances in Information Retrieval" Edited by Croft, Kluwer Academic Pub., 2000 [more detailed state-of-the-art IR book] Optional: "Machine Learning" by Tom M. Mitchell, WCB McGraw-Hill [Tools for text categorization and data mining.]

Information Retrieval: The Challenge (1) Text DB includes: (1) Rainfall measurements in the Sahara continue to show a steady decline starting from the first measurements in In 1996 only 12mm of rain were recorded in upper Sudan, and 1mm in Southern Algiers... (2) Dan Marino states that professional football risks loosing the number one position in heart of fans across this land. Declines in TV audience ratings are cited... (3) Alarming reductions in precipitation in desert regions are blamed for desert encroachment of previously fertile farmland in Northern Africa. Scientists measured both yearly precipitation and groundwater levels...

Information Retrieval: The Challenge (2) User query states: "Decline in rainfall and impact on farms near Sahara" Challenges How to retrieve (1) and (3) and not (2)? How to rank (3) as best? How to cope with no shared words?

Information Retrieval in eCommerce (1) Bringing in Customers How do Web-search engines work? How to maximize hits on my eCommerce pages? How to maximize preselection of customers who will transact?

Information Retrieval in eCommerce (2) Analyzing the Competition How do we find the competition? How will customers find the competition? Can we do preemptive information strikes? Text Mining How to learn what customers want most? How to find out what they missed, but wanted? How to discover customer search/browsing patterns?

Information Retrieval Assumption (1) Basic IR task There exists a document collection {D j } Users enters at hoc query Q Q correctly states user’s interest User wants {D i } < {D j } most relevant to Q

"Shared Bag of Words" assumption Every query = {w i } Every document = {w k }...where w i & w k in same Σ All syntax is irrelevant (e.g. word order) All document structure is irrelevant All meta-information is irrelevant (e.g. author, source, genre) => Words suffice for relevance assessment Information Retrieval Assumption (2)

Information Retrieval Assumption (3) Retrieval by shared words If Q and D j share some w i, then Relevant(Q, D j ) If Q and D j share all w i, then Relevant(Q, D j ) If Q and D j share over K% of w i, then Relevant(Q, D j )

Boolean Queries (1) Industrial use of Silver Q: silver R: "The Count’s silver anniversary..." "Even the crash of ’87 had a silver lining..." "The Lone Ranger lived on in syndication..." "Sliver dropped to a new low in London..."... Q: silver AND photography R: "Posters of Tonto and the Lone Ranger..." "The Queen’s Silver Anniversary photos..."...

Boolean Queries (2) Q: (silver AND (NOT anniversary) AND (NOT lining) AND emulsion) OR (AgI AND crystal AND photography)) R: "Silver Iodide Crystals in Photography..." "The emulsion was worth its weight in silver..."...

Boolean Queries (3) Boolean queries are: a) easy to implement b) confusing to compose c) seldom used (except by librarians) d) prone to low recall e) all of the above

Beyond the Boolean Boondoggle (1) Desiderata (1) Query must be natural for all users Sentence, phrase, or word(s) No AND’s, OR’s, NOT’s,... No parentheses (no structure) System focus on important words Q: I want laser printers now

Beyond the Boolean Boondoggle (2) Desiderata (2) Find what I mean, not just what I say Q: cheap car insurance (pAND (pOR "cheap" [1.0] "inexpensive" [0.9] "discount" [0.5)] (pOR "car" [1.0] "auto" [0.8] "automobile" [0.9] "vehicle" [0.5]) (pOR "insurance" [1.0] "policy" [0.3]))

Beyond the Boolean Boondoggle (3) Desiderata (3) Speech-recognized queries Coming soon, to a system near you longer queries more fluff words to filter acoustic recognition errors

INFORMATION RETRIEVAL The Web Library, etc. Spider Inverted Index User Search Engine

INFORMATION RETRIEVAL: APPLICATIONS Searching Document Archives –Libraries (title, subject, full-text) –Data bases of patents and applications –DBs of legal cases (e.g. Lexis, Westlaw) Searching the Web –Pure search engines (Google, Inktomi, …) –Browsing + Search (Yahoo, Terra-Lycos, …) –Meta-search (Metacrawler, Vivisimo, …) Corporate or Government Intranets Non-traditional (e.g. Software DBs, News)

INFORMATION RETRIEVAL (IR) EVOLUTION IR in the 1980s: –Single collection with < 10 6 documents (archive) –Boolean queries with unordered-set answer IR circa 2000: –Single collection with > 10 9 documents (web) –Free-form queries with ranked-list answer IR circa 2010: –Multiple collections > docs (invisible web) –“Find what I mean” queries with clustering, summarization and customization.

Content for Rest of the Course (1) [See the course BB for the latest updates to the course schedule.] Under the Hood The vector space model for retrieval Building an inverted index Term weighting and selection Web spidering Automated text categorization

Content for Rest of the Course (2) IR Uses in eCommerce How to make search engine work for you How to build optimal search-attractive web sites The business(es) of web-based information Beyond Web Search Engines Speech processing primer Information extraction from web pages Data mining primer Multi-media applications Business models

Optional Quick Review of Linear Algebra If you know n-dimensional vectors, matrices, computing inner products, etc.., Then you do not need this review. You may take a break. If you learned this material, but do not remember it, please stay and listen to refresh your knowledge. If you never learned linear algebra, stay, listen and (optionally) read either: G. Hadley. Linear Algebra. Addison-Wesley, Ch 3. Or, Stephen W. Goode. An Introduction to Differential Equations and Linear Algebra. Prentice Hall, Ch.3).