Evolving Internet Technologies: Web Search Engines Danny Sullivan Editor, SearchEngineWatch.com

Overview  Key “technology” in 2001 was survival  Crawlers replacing humans  New & old players to watch  11 September & Mindreading  Other Things

RIP 2001 Go.com (Infoseek) You were one of the first web-wide spiders and later added your own human directory of sites NBCi (Snap) You provided your own human-compiled guide to the web Excite You were another of the oldest web spiders to finally cease crawling

Search Economics  Economics is boring but important!  Makes search engines viable; may impact results  Banner ads no longer sell  Listing services new way to make money  Allow much needed “conversation” between search engines and site owners…  But more interactive with results than banners, so searchers and site owners have new fears  What’s offered & should you worry?

Paid Placement  Buy your way to the top  All sell it, even GoogleGoogle  Overture (GoTo) sells for AOL, AltaVista, Ask, HotBot, Lycos, Yahoo and others OvertureHotBotYahoo  In Europe, Espotting sells for Yahoo, Lycos, others

Paid Placement Concerns  Users don’t really seem to mind -- yet  Similarity to “editorial” may cause distrust  Main reason behind FTC complaint last July  Ask Jeeves, Lycos recently improved labels  Why deny users top sites, if they don’t pay???  Heavy “ad break” might drive users away…

Meta Search or Meta Ads?  Meta Ads  Dogpile, Dogpile  Search.com, Search.com  Mamma, Metacrawler like above  Meta Search  Vivisimo,  IxQuick,  qbSearch,  SurfWax,

Paid Submission  Pay to get your site reviewed quickly  No guaranteed ranking – no guarantee to even be included!  Yahoo and LookSmart both offer  Mandatory for business categories  Annual charge at Yahoo: Yellow Pages

Paid Submission Concerns  Is it fair to miss some businesses?  How many florists do you want? 100, 1000?  What about non-profits, hobbyists?  Non-commercial categories exempt at Yahoo  LookSmart’s use of Zeal.com feeds its non-commercial listings, give good balance

Paid Inclusion  Get deeper representation in listings and with crawlers, faster revisits  Usually doesn’t guarantee rankings, but…  Like having more tickets in the lottery – more chances to win  Every major crawler but Google sells this, as does LookSmart

Paid Inclusion: Example  Inktomi: $39 gets first URL listed in 2 days, revisited each week  Want more, $12-15 each, or CPC pricing  No pay? Still might get included, anyway  Program has provisions for non-profits  No rank boost

Paid Inclusion Concerns  Will we see important sites / pages dropped just because they don’t pay?  That works against users and site owners  Fair those who pay better represented?  The “real” world works this way  Northern Light worked this way for years Northern Light  May depend on a case-by-case basis

Humans Were Supreme  From start of popular use of the web, human- powered Yahoo has been top search site  Why? It helped you refine. Search for “travel” gave 10 categories rather than 10 million resultstravel  Yahoo “seemed” to find things when it actually gave you less but forced you to be more specific Others followed Yahoo’s lead…

Rise & Fall Of Humans  By 2000, 5 major human directories “powered” 6 of top 10 search engines  But now, 3 directories power 4 of top 10  Yahoo, MSN, Netscape, LookSmart  AOL, Lycos, Ask Jeeves abandoned humans in 2001/2002 Why the change?

Human Weaknesses  Editors cost money  Go and NBCi ran out of this in 2001  Ask also scaled back on human answers  Machines can now do some of what humans originally did…  “Related Searches” refine queries in the way categories didRelated Searches  “Autocategorization” also refines…

Auto-Categorization  Group pages into categories, on the fly  One reason why Teoma, Wisenut and Vivisimo get good reviewsTeoma  Not news to Northern Light!Northern Light  Google says not necessary, but we’ll see  They find human categorization better (directory tab)

But Humans Still Involved  Crawlers better at being “human” because they leverage human work more than in the past  Human-made links used to determine importance  Links used to determine context of pages  Links used to autocategorize into “communities”  Crawlers also dependent on directories, giving them great weight in considering how to rank  So what happens now that Yahoo & LookSmart are more commercial?  What happens if the Open Directory fails?

Who’s New: Crawlers  Teoma.com  Potential there, but will Ask have the funds and know how (this time) to make it happen  Coverage is set to grow; of course, paid inclusion was first “improvement” shown  WiseNut.com  LookSmart set to buy it; will this solve the freshness issue?

Who’s Still Hot  Google (and Google Toolbar)  Everything they do is magic  Good: finally, a tool you can learn and depend on  AllTheWeb.com  Big improvements recently; take another look  Don’t forget to visit Yahoo’s categories or surprise yourself with a search at MSN

Lessons Of 11 September  People hit Google & others for news  Terms included: cnn, news, world trade center, bbc, reuters, msnbc, sky news, new york times, pentagon, bin laden, american airlines, united airlines  What did they get?…

Google results, 4 hours after attack

FAST/AllTheWeb.com results, 4 hours after attack

Ask Jeeves results, 4 hours after attack

But at AltaVista, less than 2 hours after the attack…

“Blended” results mixed in news content, even if news option on home page had been ignored Why Did AltaVista Succeed? We know historically that home page options DO get ignored, but we learn this again on Sept. 11, by watching Google and others

Read My Mind  Sept. 11 dramatically illustrates the main search challenge – the need to somehow automatically hit the correct dataset  Images for search on “pictures of spain”  MP3 files for search on “madonna”  Movie info for search on Harry Potter this month  How NOT to do it, then good examples…

Examples Of Mind Reading  Smart query analysis, then suggestions or insertions of non-web material  Products at AltaVistaAltaVista  Sidebar results at AllTheWebAllTheWeb  News, dictionary, stock & more at GoogleGoogle  Encyclopedia at MSNMSN  Careful not to take away all control  Power search for few who want to drive

Specialty Search / Vortals  To mind read, you need specialty datasets  Among the majors, Google & AllTheWeb pushing here, & I think they’ll keep going  Also think (and hope) we’ll still see more “vortals” or “vertical portals”  Moreover,  MessageKing,  xrefer,  LawCrawler,

Other Issues & Trends  Freshness  Size  “Off the page” ranking criteria

Feeling For Freshness  AllTheWeb pushed end of last year to be 9-12 days old at most – now more likely to be a month, like others  Google & others aiming to be less than a month old or fresher for key documents  Just show dates when pages were visited!dates

Still Growing, But Still Missing  The leaders?  Google, 1.1 to 1.6 billion documents  AllTheWeb, 625 million  Large index probably more comprehensive  We do want more index growth!  However, don’t judge a search engine only based on its index size…

Does Size Matter?  To professionals, yes. Coverage helps them find unusual or obscure material  What good is half a haystack?  To average users, not really. They desperately need better relevancy.  How about I dump a haystack on your head?  100 million extra pages makes no difference to best matches for “horoscopes” or “britney spears”

“Off The Page” Ranking  Looking beyond content of the page, since webmasters can’t easily control this  Link analysis still going strong  But can produce oddities, like infamous Bush result  Under new pressure from link spammers  Clickthrough measurements not as hot  Personalization might get revived with Google  Past fears would limit results, rather than help

Some Closing Thoughts  Yet we’ve had them less than 10 years!  Answers to everything weren’t on web before, aren’t now and never will be, so… Search engines are the top resource for Americans seeking answers, used 32% of the time --Consumer Daily Question Study, Fall 2000

Just One Of Many Tools  Don’t expect miracles from search engines  They’re great “Swiss Army Knives,” but you’ll still want an entire toolbox  My hot search tools? Telephone & !  Use them to avoid “search rage”  Stop searching after 10 minutes and try other means. Also…

Be Non-Traditional  Forget Boolean, please  Don’t cast your net wide  You don’t need every synonym in your query…  Instead, explore what’s in the first catch!  Unlike traditional tools, web documents LINK  A few good pages usually lead you to more good pages – your answer may be a few clicks away  You’ll also find links bring you to documents that contain the synonyms you would have tried

This Presentation - Search Engine Watch - Web Searching Tips – Search Engine Listings Free Search Engine Newsletters (SearchDay – Search Engine Report) Become A Member – You Support Me & Chris Sherman! (and get some extra benefits)