Data Mining for Web Intelligence Presentation by Julia Erdman

Data Mining the Web Searching, comprehending, and using the semi-structured data on the web poses a significant challenge over data mining in a commercial database system The data from the web is more sophisticated and dynamic Data mining helps search engine find high- quality web pages

Why Data Mining? Challenges of data mining the web Web page complexity far exceeds the complexity of any traditional text document collection The Web constitutes a highly dynamic information source The Web serves a broad spectrum of user communities Only a small portion of the Web’s pages contain truly relevant or useful information

Why Data Mining? Approaches to accessing information on the web Keyword-based search or topic-directory browsing i.e. Google, Yahoo Querying deep Web sources i.e. Amazon.com, Realtor.com Random surfing

Design Challenges Traditional schemes for accessing data on the web are based on text-oriented, keyword- based web pages The current access schemes must be replaced with more sophisticated schemes in order to exploit the Web completely

Access Limitations Lack of high-quality keyword-based searches A search can return many answers i.e. searching popular categories, like sports or politics Overloading keyword semantics can return many low-quality answers i.e. a search for jaguar could be for an animal, car, sports team A search can miss many highly related pages that do not contain the posed keywords

Access Limitations Lack of effective deep-Web access There are at least 100,000 searchable databases on the Web with high-quality, well-maintained information, but are not effectively accessible There is an extremely large collection of autonomous and heterogeneous databases, each supporting specific query interfaces with different schema and query constraints

Access Limitations Lack of automatically constructed directories A topic or type-oriented Web information directory creates an organized picture of a web sector Developers must organizes these directories manually Costly Provides only limited coverage Not easily scalable or adaptable

Access Limitations Lack of semantics-based query primitives Most keyword-based searches only allow of small set of search options

Access Limitations Lack of feedback on human activities Web links may not be updated frequently, regularly, or at all Changes in access frequency do not automatically adjust search results

Access Limitations Lack of multidimensional analysis and data mining support Cannot drill deeply into sites in order to find the data we are looking for

Mining Web search-engine data Current keyword-based search engines have several deficiencies A widely covered topic can contain hundreds of thousands of documents Highly relevant documents may not contain the keywords used in the search

Analyzing the Web’s link structure When one web page contains a link to another, this can be considered an endorsement of the linked page Collected endorsements of the same page from many different web authors leads to an authoritative web page A hub is a single web page that contains a collection of links to authoritative web pages

Classifying Web documents automatically Generally, human readers classify Web documents, but an automatic classification is highly desirable Hyperlinks contain high-quality semantic clues to a page’s topic, which can help achieve accurate classifications However, links to unrelated sites can cloud the classification i.e. many sites have a link to weather.com, but generally are not weather sites Automatic classification can determine what classification a web page belongs to, but not to which classification it does not belong to

Mining Web page semantics structures and page contents Fully automatic extraction of Web page structures and semantic contents can be difficult due to the limitations on automated natural-languages parsing Semiautomatic methods can recognize a portion of such structures Then further analysis can see how the contents fit into these structures

Mining Web page semantics structures and page contents To identify the structures to extract, either an expert manually specifies the structures, or techniques must be developed to automatically produce the structures Or developers can use Web page classes for automatic extraction Semantic page structure and content recognition will provide for more in-depth analysis of Web pages

Mining Web dynamics Contents, structures, and access patterns change on the Web Storing historical data about Web pages assists in finding changes in content and links But due to phenomenal breadth of the Web, it is impossible to store images and updates Mining web logs records can provide quality results This data needs to be analyzed and transformed into useful, significant information

Building a multilayered, multidimensional Web Systematically analyze a set of Web pages Group closely related local Web pages or an individual page into a cluster, called a semantic page The analysis provides a descriptor for the cluster Then create a semantics-based, evolving, multidimensional, multilayered Web information directory

Questions? Comments?

Jiawei H. & Chang, K.C.-C. "Data mining for Web intelligence" IEEE Computer, Volume 35, Issue 11, Nov pp