1. WebMiningResearch ASurvey Web Mining Research: A Survey By Raymond Kosala & Hendrik Blockeel, Katholieke Universitat Leuven, July 2000 Presented 4/18/2002 by Caitlin C Coughlin

2. 4/18/2002Caitlin C Coughlin, University of Vermont2 Overview n Introduction n Web Mining n Web Content Mining n Web Structure Mining n Web Usage Mining n Conclusions

3. 4/18/2002Caitlin C Coughlin, University of Vermont3 Introduction n The Web is huge, dynamic & diverse, and thus raises the scalability, multimedia data and temporal issues respectively. n Thus we are drowning in information and facing information overload. Information users can encounter problems when interacting with the Web

4. 4/18/2002Caitlin C Coughlin, University of Vermont4 More Introduction PROBLEMS: l Finding Relevant information l Creating new knowledge out of the information available on the web l Personalization of the information l Learning about consumers or individual users

5. 4/18/2002Caitlin C Coughlin, University of Vermont5 More Introduction Web mining techniques could be directly or indirectly used to solve the information overload problems described before. directly - application of web mining techniques directly addresses the problem indirectly- web mining approach techniques are used as part of a bigger application that addresses the aforementioned problems. Web mining NOT only useful tool: other useful techniques include nDB database nIR Information Retrieval nNLP Natural Language Processing nWeb document community

6. 4/18/2002Caitlin C Coughlin, University of Vermont6 Web Mining: Outline n Overview of Web Mining n Describe some confusion in use of the term “Web Mining” n Provide a Classification n Relate Classification to the agent paradigm

7. 4/18/2002Caitlin C Coughlin, University of Vermont7 Web Mining: Overview Web mining is the use of data mining techniques to automatically discover and extract information from web documents and services. We suggest decomposing Web mining into these subtasks: Resource finding l 1 Resource finding: the task of retrieving intended web documents Information selection and pre-processing l 2 Information selection and pre-processing: automatically selecting and pre-processing specific information from retrieved Web resources Generalization l 3 Generalization: automatically selecting and preprocessing specific information from retrieved Web resources Analysis l 4 Analysis: validation and/or interpretation of the mined patterns. We’ll call this pattern 1-2-3-4, as we’ll later see, sometimes 1-3-4 is also used.

8. 4/18/2002Caitlin C Coughlin, University of Vermont8 Web Mining: Confusion Information Retrieval Information Extraction l Web mining is often associated with Information Retrieval or Information Extraction, but it is different from both. l IR l IR is the automatic retrieval of all relevant documents while at the same time retrieving as few non-relevant ones as possible. [views documents as bag-of-words] l IE l IE has the goal of transforming a collection of documents, usually with the help of an IR system, into information that is more readily digested and analyzed. [interested in the structure or representation of a document] l We argue that Web mining intersects with the application of machine learning on the web.

9. 4/18/2002Caitlin C Coughlin, University of Vermont9 Web Mining: Classification u Web content mining u Web content mining: describes the discovery of useful information from Web contents/data/documents. [IR and DB views] u Web structure mining u Web structure mining: tries to discover the model underlying the link structures of the Web. u Web usage mining: u Web usage mining: tries to make sense of the data generated by the Web surfer’s sessions or behaviors

10. 4/18/2002Caitlin C Coughlin, University of Vermont10

11. 4/18/2002Caitlin C Coughlin, University of Vermont11 Web Mining & the Agent Paradigm Web mining is often viewed from or implemented within an agent paradigm. Thus, web mining has a close relationship with software agents or intelligent agents. Two relevant types of software agents: ¨ User interface agents : information retrieval agents, information filtering agents, & personal assistant agents ¨ Distributed agents : distributed agents for knowledge discovery or data mining [content-based or collaborative]

12. 4/18/2002Caitlin C Coughlin, University of Vermont12 Web Mining & the Agent Paradigm

13. 4/18/2002Caitlin C Coughlin, University of Vermont13 Web Content Mining: IR view Information retrieval view for unstructured documents: çmost of the research uses “bag of words” to represent unstructured documents. çTakes single words as features. Features could be boolean or frequency based. çSee the table that follows

14. 4/18/2002Caitlin C Coughlin, University of Vermont14 1998 1999 1995 1998 1995 1998 1999 1997 1999 1997 2000 1999 1996 1999 1995 1999

15. 4/18/2002Caitlin C Coughlin, University of Vermont15 Web Content Mining: IR view

16. 4/18/2002Caitlin C Coughlin, University of Vermont16 Web Content Mining: DB View © The database techniques on the web are related to the problem of managing and querying the information on the web. © Three classes of tasks: modeling and querying the web, information extraction and integration, and web site construction and restructuring. © Tries to model the data on the web and to integrate them so that more more sophisticated queries other than the keywords based search can be performed. © Research in this area mainly deals with semi-structured data

17. 4/18/2002Caitlin C Coughlin, University of Vermont17 Web Content Mining: DB view

18. 4/18/2002Caitlin C Coughlin, University of Vermont18 Web Structure Mining « In Web structure mining we are interested in the structure of the hyperlinks within the Web itself. (inter-document structure) « This line of research inspired by the study of social networks and citation analysis. « A few different algorithms have been proposed to do this such as HITS, PageRank, improved HITS using content info & outlier filtering [example coming up]

19. 4/18/2002Caitlin C Coughlin, University of Vermont19 Successful example of Web Structure Mining t The heart of Google software is PageRank™, a system for ranking web pages developed by our founders Larry Page and Sergey Brin at Stanford University t PageRank uses the web’s link structure as an indicator of an individual page's value. Google interprets a link from page A to page B as a vote, by page A, for page B. Google also analyzes the page that casts the vote. Votes cast by pages that are themselves "important" weigh more heavily and help to make other pages "important.” t Google combines PageRank with sophisticated text-matching techniques to find pages that are both important and relevant to your search. [they don’t specify] t Google does not sell placement within the results themselves (i.e., no one can buy a higher PageRank).

20. 4/18/2002Caitlin C Coughlin, University of Vermont20 Web Usage Mining * W eb usage mining focuses on techniques that could predict user behavior while the user interacts with the web. * T wo commonly used approaches: 1) mapping the usage data of the web server into relational tables before an adapted data mining technique is performed, 2) uses the log data directly by using special preprocessing techniques. * A pplications of web usage mining fall into two main categories: learning a user profile/user modeling in adaptive interfaces [personalized] and learning user navigation patterns [impersonalized]

21. 4/18/2002Caitlin C Coughlin, University of Vermont21 Conclusions W We surveyed research in Web Mining, W clarified some confusion in the use of the term Web mining, W explored the connection between Web mining categories and the agent paradigm, W & suggested three Web mining categories and situated some current research with respect to these categories. W The Web presents new challenges to the traditional data mining algorithms that work on flat data. We have seen that some of the traditional data mining algorithms have been extended or new algorithms have been used to work on the Web data.

22. 4/18/2002Caitlin C Coughlin, University of Vermont22 Questions?