Discovering Web Access Patterns and Trends by Applying OLAP and Data Mining Technology on Web logs Data Engineering Lab 성 유 진

Abstract Web server log files analysis problem and difficulty server performance improvement system performance improvement customer targeting in electronic commerce problem and difficulty large raw log data processing is not easy data reduce size and time

WebLogMiner current weglogminer only frequency count  not enough slow, inflexible, difficult to maintain only frequency count  not enough WebLogMiner Virtual University/data mining WeblogMiner OLAP and data mining technique multi-dimensional data cube scalability, interactivity, variety, flexibility

Design of a Web log Miner Web log server log file information domain name of the request / user name / date and time of the request / the method of the request(GET, POST) / the name of the file requested / the result of the request(success, failure, error, etc) / size of the data sent back / the URL of the referring page / identification of the client agent Example 210.114.3.64 - - [01/Jul/1998:17:34:05 0900] "GET/~yjsung/sign.htmlHTTP/1.1" 200 740 210.114.3.64 -- [01/Jul/1998:17:38:44-0900] "POST/cgi-bin/yjsung/signHTTP/1.1" 200 352  POST : 브라우저가 채워진 양식을 서버에 전달 할 때 GET : 서버로부터의 데이터 요청 시

Sequence of requests can predict next request  improve traffic Cache information frequent backtracking and reload : deficient design client site log Access count not always the measure of interestingness 특정 document를 access하기 위해 반드시 거쳐야하는 사이트 Time and Date evaluate user interest by time spent Domain name Sequence of requests can predict next request  improve traffic

WebLogMiner 4 Stages .Filtering the data, creating relational DB 2. Data cube construction 3. OLAP is used 4. Data mining technique are used

1.DATABASE CONSTRUCTION FROM SERVER LOG FILES Data Cleansing and Transformation filter out page graphics(sound and video) but 보존 two types without knowledge about site (time day, month, year등으로의 transformation은 서버 정보 없이 가능) with knowledge about site : associating server request to intended action needs site structure relation database cleaned data and new implicit data is added

2.MULTI-DIMENSIONAL WEB LOG DATA CUBE CONSTRUCTION AND MANIPULATION group by operator in SQL is used to compute aggregates on a set of attributes sum of sales by P, C: for each product, give a breakdown on how much of it was sold to each customer CUBE is the n-dimensional generalization of group-by gives remarkable flexibility to manipulate and view the data allow OLAP operation such as drill-down, roll-up, slice and dice

Attributes - URL - domain name - size of resource, - time .

3.DATA MINING ON WEB LOG DATA CUBE AND WEB LOG DATABASE Data Characterization find rule that summarize user defined data set ☞ the traffic on a web server for a given type of media in a particular time of day Class comparison discover discriminant rules ☞ compare requests from two different web browsers Association discover the patterns that access to different resources consistently occurring together Prediction ☞ access to a new resource on a given day can be prediected based on accesses to similar old resources on similar days

Time-series analysis - Classification can be used to develop a better understanding of each class in the web log database, and perhaps restructure a web sit or customize answers to requests based on classes of requests Time-series analysis - to analyze data along time sequences to discover time-related interesting patterns … ☞ disclose the patterns and trends of the improvement of services of the web server Focus will be on time-series analysis because web log records are highly time-related

Experiments with the web log miner Virtual-U:six different major component: Goal - understand the usage and user behavior patterns Data Cleaning and transformations all entries were mapped one on one into relational database field site, user action are added. Problem extraneous information => define those entries and eliminate them multiple server requests by same user action same server request by multiple user actions local activities are not recorded

Multi-dimensional data cube construction manipulation summarization(group-bys on different dimensions) request/domain /event/session/bandwidth/ error/referring organization /browser summary Examples Figure2) OLAP analysis of Web log

Fig3) Typical event sequence and user behavior pattern analysis Fig4) Web traffic analysis of Web log

Fig6) Event trees of month one to four

Discussion and Conclusion WebLogMiner OLAP and data mining technique multi-dimensional data cube major strength scalability, interactivity, variety, flexibility Current log file의 문제점 web server should collect more information new structure is needed ==> would simplify pre-processing