WIRED Week 2 Syllabus Update Readings Overview

Why IR? IR originally mostly for systems, not people IR in the last 25 years: classification and categorization systems and languages user interfaces and visualization A small world of concern The Web changed everything Huge amount of accessible information Varied information sources Relatively easy to look for information Improving IR means improving learning Digital technology changes everything (again) We cut out the middle man and pass the savings on to you!

WIRED Focus Information Retrieval: representation, storage, organization of, and access to information items Focus is on the user information need User information need: Find all docs containing information on Austin which: Are hosted by utexas.edu Discuss restaurants Emphasis is on the retrieval of information (not data, not just a keyword match)

The Search Who is John Battelle? Magazine Editor: WIRED, The Industry Standard Web 2.0 conference organizer Business 2.0 magazine columnist Federated Media Publishing Boingboing.net “manager”

Database of Intentions What do you think the database of intentions is? Is it more than Google’s Zeitgeist? What we’re thinking about and interested in. Everything we want to know and when we want to know it. “the aggregate results of every search ever entered, every result list ever tendered, and every path taken as a result” (Battelle, p 6) “a real time history of post-Web culture” (p 6) What other databases like this are there? How is this possible?

Searchiness? The “tasking” of search? Everything could be a search task? Every task has an ad associated with it? Our expectations are met and made with search. How would the Web work without search? Yahoo and email links, LOTS of email links You are your clickstream? Products & services based on it “marketing, media, technology, pop culture, international law, and civil liberties” (p 13)

Elements of Search Crawl Index Runtime system (query processor) Segments the data Analyzes the Crawl Optimizes everything Interface Query Reults Users

Search before Google Traditional systems: SMART (Salton) Strongly typed information, (traditional databases) Not always interactive or easy to use Library Catalogs online Controlled vocabulary & limited records Internet: Archie & Veronica Titles only (mostly) over text Web: WWW Wanderer, Web Crawler Full text, HTML & links

AltaVista gets serious Web now large enough to be a challenge Now enough content that you’d want to search it Costs of hardware & bandwidth falling Parallel crawlers Significant CPU resources 1995 = 16 million documents Why didn’t people get it ?

The Web goes Pro Lycos Yahoo AOL Excite Anchor text & content location context Yahoo Directory & clean interface for browsing links Adversiting & user (logs) analysis AOL Gateway to the internet for many Excite Consumer-driven, word relationships Acquisitions of Magellan, WebCrawler ++ MyExcite - the Portal @Home (compete with AOL)

Google is Born Larry Page & Sergey Brin Links are the key (Bibliometrics) Impact factor (“link it if you like it”) Patterns of citation (links) expand the text Defending & setting the context of your work by associating it with others Backrub Crawl pages, store links, analyze them, publish Large computing challenges PageRank Link counts with a recipe for deriving (relative) value Value is who & and their rank too

Google goes Pro More resources for more data Help with (significant) analysis design Lack of commercial approach may have been a strength Not ads, but just good search Simple (non-existent) design of interface had an impact More people getting online Broadband adoption & stabilizing browsers Growing content (to say the least)

Assignments Read weekly Primary Readings & Participate in class discussions 10% Re-design Search Results interface 10% Web (log) analytics 25% “Google 2010” (5 page paper) 10% Class Topic Presentation 15% Main Project 30%

Projects and/or Papers Overview How can (Web) IR be better? Better IR models Better User Interfaces More to find vs. easier to find Scriptable applications New interfaces for applications New datasets for applications