1. Web Evaluation Websites and the Invisible Web HIST 221/INFO 221 February 25, 2004 Presented By: Teresa Ferguson tfergus@u.washington.edu

2. What we will cover Evaluating Websites URLs Domains Quality Author Content Linking Introduction to the Invisible Web Define the invisible web Why search engines can’t see some web sites Four types of invisibility Distinguish between visible and invisible How to find and use the invisible web How to develop your own favorite invisible web site toolkit

3. Web Evaluation - URLs Is it a personal page?personal Read the URL carefully Is there a tilde (~) or percent sign (%) followed by a personal name? Do you see the words “users” “members” or “people”? Is the site on a commercial ISP like AOL? Personal pages are not intrinsically bad. But remember, anyone can publish on the internet and no one is editing the content.

4. Web Evaluation - Domains What is the domain? Government -.gov,.mil,.us, or country code.gov Educational -.edu Non-profit -.org.org Commercial -.com,.net, etc. Does the domain match the content?

5. What about quality? Is information supported with footnotes or links? Is the information reproduced, altered or forged? Does the author provide links to useful resources on the topic? Has the author provided references?

6. More quality issues Can you find information about the author on the site?author Is the information verified on other sites? Is the information current? When was the webpage last updated?current Are there a lot of broken links? Is there an obvious bias in the material? Are there advertisements or pop-ups?

7. Maybe opinions of others will help you decide Is this page linked on someone else’s page? Try a link search in Google (link:www.densho.org)Google Is it included in a quality directory?directory Is the author’s name cited on any other sites?

8. This brings us to the Invisible Web The invisible web are web sites that search engines cannot crawl for several reasons: The web site is actually a PDF document and not HTML The web site is a picture that has no links or explanation The web site has an application, usually a database, that the crawlers can’t look into Directories will have invisible web sites if a human has found it and listed it.

9. First type of Invisibility Opaque web sites are web sites that can be indexed by search engines but are not because of the following reasons They are too deep They are new links on a web site There are so many results returned that any site listed after #50 is not going to be looked at URLs that are not submitted for indexing, or are not linked to other web sites (Sherman and Price, 70)

10. Second type of Invisibility Private web sites are web sites accessed over the public web but only for use by specific groups or person. MyUW is an example of a private web (Sherman and Price, 73)

11. Third type of Invisibility Proprietary Web – Web sites that can only be viewed by those that agree to specific requirements: money, agreements, or registrations (Sherman and Price, 73) America, History and Life

12. Fourth Type of Invisibility Truly Invisible - web sites that cannot be indexed by search engines File type not recognized by search engines File type, or content is not searched because the creator of the search engine told it not to, usually not searched to save time and money Dynamically created web pages, like this: click hereclick here Data is stored in a relational database that search engines cannot crawl (Sherman and Price, 74-75)a relational database

13. Why should we use the Invisible Web? Specialized content Specialized search interfaces, usually means it supports Boolean searches Increased precision and recall, which means you will get more accurate results Highest level of authority on the web Answer may only be available in an invisible web site (Sherman and Price, 92-93)

14. The Invisible Web Will continue to grow as more associations or groups index their own publications Those indexes can contain scholarly items There are no search tools that index databases, they are usually found by humans who then make note of the site

15. Invisible Web Directories Invisible Web Directory www.invisible-web.net www.invisible-web.net Complete Planet http://www.completeplan et.com/ http://www.completeplan et.com/ Infomine http://infomine.ucr.edu/ http://infomine.ucr.edu/ Librarian’s Index to the Internet http://www.lii.org/ http://www.lii.org/ Finding Associations Unlimited http://galenet.galegroup.co m/servlet/AU;jsessionid=7C A478FA1265DC7F28E856570 07EB76F?locID=wash_main http://galenet.galegroup.co m/servlet/AU;jsessionid=7C A478FA1265DC7F28E856570 07EB76F?locID=wash_main Scholarly Societies Project http://www.scholarly- societies.org/ http://www.scholarly- societies.org/ Firstgov http://firstgov.gov/ http://firstgov.gov/

16. A Simple Example Let’s go to Google, and find the book, Japanese American Internment During World War II: A History and Reference Guide, by Wendy Ng, which I know is in a UW Library Why is Google not showing the library’s copy?Google

17. Remember to Evaluate ALL scholarly information needs to be evaluated for: Reliability - dependable Validity - logical Accuracy - correctness Authority - source of reliable information Timeliness - occurring at a good time Bias – preference

18. Developing a Toolkit Can be as simple as adding web sites to Favorites in your Browser Keeping a list of websites in Word or HTML Can use more advanced Bookmark Software like, Cogitum Co-citer (http://www.cogitum.com/co-tracker-text/)http://www.cogitum.com/co-tracker-text/ Found through Search Engine Watch (www.searchenginewatch.com) located on the Search Utilities pagewww.searchenginewatch.com Sign up for current awareness services Scout Report (http://scout.cs.wisc.edu/)http://scout.cs.wisc.edu/ Librarian’s Index to the Internet (http://lii.org/)http://lii.org/ Research Buzz (http://www.researchbuzz.com/)http://www.researchbuzz.com/ Search Day (http://searchenginewatch.com/searchday/)http://searchenginewatch.com/searchday/

19. References Many of these slides were modified from Information Research Strategies in Sociology, Winter 2003 lecture notes presented by Laura Lillard, University of Washington Education Librarian Sherman, Chris & Gary Price. The invisible Web: uncovering information sources search engines can't see. Medford, N.J.: CyberAge Books, Information Today, 2001. (ZA4450.S54 2001)