Comparing between machine learning methods for a remote monitoring system. Ronit Zrahia Final Project Tel-Aviv University

Overview zThe remote monitoring system zThe project database zMachine learning methods: yDecision of Association Rules yInductive Logic Programming yDecision Tree zApplying the methods for project database and comparing the results

Remote Monitoring System - Description zSupport Center has ongoing information on customer’s equipment zSupport Center can, in some situations, know that customer is going to be in trouble zSupport Center initiates a call to the customer zSpecialist connects to site from remote and tries to eliminate problem before it has influence

Remote Monitoring System - Description Gateway Product AIX/NT Customer TCP/IP [FTP] TCP/IP [Mail/FTP] Support Server AIX/NT/95 Modem

Remote Monitoring System - Technique zOne of the machines on site, the Gateway, is able to initiate a PPP connection to the support server or to ISP zAll the Products on site have a TCP/IP connection to the Gateway zBackground tasks on each Product collect relevant information zThe data collected from all Products is transferred to the Gateway via ftp zThe Gateway automatically dials to the support server or ISP, and sends the data to the subsidiary zThe received data is then imported to database

Project Database z12 columns, 300 records zEach record includes failure information of one product at a specific customer site zThe columns are: record no., date, IP address, operating system, customer ID, product, release, product ID, category of application, application, severity, type of service contract

Project Goals zDiscover valuable information from database zImprove the products marketing and the customer support of the company zLearn different learning methods, and use them for the project database zCompare the different methods, based on the results

The Learning Methods zDiscovery of Association Rules zInductive Logic Programming zDecision Tree

Discovery of Association Rules - Goals zFinding relations between products which are bought by the customers yImpacts on product marketing zFinding relations between failures in a specific product yImpacts on customer support (failures can be predicted and handled before influences)

Discovery of Association Rules - Definition zA technique developed specifically for data mining yGiven xA dataset of customer transactions xA transaction is a collection of items yFind xCorrelations between items as rules zExample ySupermarket baskets

Determining Interesting Association Rules zRules have confidence and support yIF x and y THEN z with confidence c xIf x and y are in the basket, then so is z in c% of cases yIF x and y THEN z with support s xThe rule holds in s% of all transactions

Discovery of Association Rules - Example zInput Parameters: confidence=50%; support=50% zIf A then C: c=66.6% s=50% zIf C then A: c=100% s=50% TransactionItems 12345A B C 12346A C 12347A D 12348B E F

Itemsets are Basis of Algorithm zRule A => C zs=s(A, C) = 50% zc=s(A, C)/s(A) = 66.6% TransactionItems 12345A B C 12346A C 12347A D 12348B E F ItemsetSupport A75% B50% C A, C50%

Algorithm Outline zFind all large itemsets ySets of items with at least minimum support yApriori algorithm zGenerate rules from large itemsets yFor ABCD and AB in large itemset the rule AB=>CD holds if ratio s(ABCD)/s(AB) is large enough yThis ratio is the confidence of the rule

Pseudo Algorithm

Relations Between Products

Relations Between Failures Association RulesConfidence ( CF )Item Set ( L ) 4  6 14 / 16 =  4 14 / 15 =  / 18 =  5 15 / 15 = 1

Inductive Logic Programming - Goals zFinding the preferred customers, based on: yThe number of products bought by the customer yThe failures types (i.e severity level) occurred in the products

Inductive Logic Programming - Definition zInductive construction of first-order clausal theories from examples and background knowledge zThe aim is to discover, from a given set of pre- classified examples, a set of classification rules with high predictive power zExamples: yIF Outlook=Sunny AND Humidity=High THEN PlayTennis=No

Horn clause induction Given: P: ground facts to be entailed (positive examples); N: ground facts not to be entailed (negative examples); B: a set of predicate definitions (background theory); L: the hypothesis language; Find a predicate definition (hypothesis) such that 1.for every (completeness) 2.for every (consistency)

Inductive Logic Programming - Example zLearning about the relationships between people in a family circle

Algorithm Outline zA space of candidate solutions and an acceptance criterion characterizing solutions to an ILP problem zThe search space is typically structured by means of the dual notions of generalization (induction) and specialization (deduction) yA deductive inference rule maps a conjunction of clauses G onto a conjunction of clauses S such that G is more general than S yAn inductive inference rule maps a conjunction of clauses S onto a conjunction of clauses G such that G is more general than S. zPruning Principle: yWhen B and H don’t include positive example, then specializations of H can be pruned from the search yWhen B and H include negative example, then generalizations of H can be pruned from the search

Pseudo Algorithm

The preferred customers If ( Total_Products_Types( Customer ) > 5 ) and ( All_Severity(Customer) < 3 ) then Preferred_Customer

Decision Trees - Goals zFinding the preferred customers zFinding relations between products which are bought by the customers zFinding relations between failures in a specific product zCompare the Decision Tree results to the previous algorithms results.

Decision Trees - Definition zDecision tree representation: yEach internal node tests an attribute yEach branch corresponds to attribute value yEach leaf node assigns a classification zOccam’s razor: prefer the shortest hypothesis that fits the data zExamples: yEquipment or medical diagnosis yCredit risk analysis

Algorithm outline zA  the “best” decision attribute for next node zAssign A as decision attribute for node zFor each value of A, create new descendant of node zSort training examples to leaf nodes zIf training examples perfectly classified, Then STOP, Else iterate over new leaf nodes

Pseudo algorithm

Information Measure zEntropy measures the impurity of the sample of training examples S : y is the probability of making a particular decision yThere are c possible decisions zThe entropy is the amount of information needed to identify class of an object in S yMaximized when all are equal yMinimized (0) when all but one is 0 (the remaining is 1)

Information Measure zEstimate the gain in information from a particular partitioning of the dataset zGain(S, A) = expected reduction in entropy due to sorting on A zThe information that is gained by partitioning S is then: zThe gain criterion can then be used to select the partition which maximizes information gain

Decision Tree - Example DayOutlookTemperatureHumidityWindPlayTennis D1sunnyhothighweakNo D2sunnyhothighstrongNo D3overcasthothighweakYes D4rainmildhighweakYes D5raincoolnormalweakYes D6raincoolnormalstrongNo D7overcastcoolnormalstrongYes D8sunnymildhighweakNo D9sunnycoolnormalweakYes D10rainmildnormalweakYes D11sunnymildnormalstrongYes D12overcastmildhighstrongYes D13overcasthotnormalweakYes D14rainmildhighstrongNo

Decision Tree - Example (Continue) humiditywind highweaknormalstrong NP S: [9+,5-] E=0.940 S: [9+,5-] E=0.940 [6+,2-] E=0.811 [3+,3-] E=1.00 Gain (S, Wind) = (8/14) (6/14)1.0 =.048 [3+,4-] E=0.985 [6+,1-] E=0.592 Gain (S, Humidity) = (7/14) (7/14).592 =.151 Which attribute is the best classifier? Gain(S, Outlook) = Gain(S, Temperature) = 0.029

Decision Tree Example – (Continue) outlook ? sunnyovercastrain Yes {D1, D2, …, D14} [9+,5-] {D4,D5,D6,D10,D14} [3+,2-] {D1,D2,D8,D9,D11} [2+,3-] {D3,D7,D12,D13} [4+,0-] ? S sunny = {D1,D2,D8,D9,D11} Gain(S sunny, Humidity) =.970 – (3/5)0.0 – (2/5)0.0 =.970 Gain(S sunny, Temperature) =.970 – (2/5)0.0 – (2/5)1.0 – (1/5)0.0 =.570 Gain(S sunny, Wind) =.970 – (2/5)1.0 – (3/5).918 =.019

Decision Tree Example – (Continue) outlook humiditywind sunnyovercastrain Yes highstrongnormalweak NoYesNoYes

Overfitting zThe tree may not be generally applicable called overfitting zHow can we avoid overfitting? yStop growing when data split not statistically significant yGrow full tree, then post-prun zThe post-pruning approach is more common zHow to select “best” tree: yMeasure performance over training data yMeasure performance over separate validation data set

Reduced-Error Pruning zSplit data into training and validation set zDo until further pruning is harmful: 1.Evaluate impact on validation set of pruning each possible node (plus those below it) 2.Greedily remove the one that most improves validation set accuracy Produces smallest version of most accurate sub-tree

The Preferred Customer NO: 7 YES: 0 NO: 0 YES: 3 NoOfProducts < 2.5>= 2.5 MaxSev < 4.5 >= 4.5 NO: 3 YES: 8 Target attribute is TypeOfServiceContract

Relations Between Products NO: 0 YES: 1 NO: 4 YES: 0 Product2 Product Product6 0 1 NO: 0 YES: 15 NO: 0 YES: 1 Target attribute is Product3

Relations Between Failures NO: 5 YES: 1 NO: 1 YES: 0 Application8 Application Application NO: 0 YES: 11 NO: 2 YES: 2 Target attribute is Application5