3. 의사결정나무 Decision Tree (Rule Induction)

Poll: Which data mining technique..?

Classification Process with 10 records Step 1: Model Construction with 6 records Algorithms Training Data Classifier (Model) IF rank = ‘professor’ OR years > 6 THEN tenured = ‘yes’

Step 2: Test model with 6 records & Use the Model in Prediction Classifier Testing Data Unseen Data (Jeff, Professor, 4) Tenured?

Who buys notebook computer? Training Dataset is given below: This follows an example from Quinlan’s ID3

Tree Output: A Decision Tree for Credit Approval age? <=30 overcast 30..40 >40 student? yes credit rating? no yes excellent fair no yes yes no

Extracting Classification Rules from Trees Represent the knowledge in the form of IF-THEN rules One rule is created for each path from the root to a leaf Each attribute-value pair along a path forms a conjunction The leaf node holds the class prediction Rules are easier for humans to understand Example IF age = “<=30” AND student = “no” THEN buys_computer = “no” IF age = “<=30” AND student = “yes” THEN buys_computer = “yes” IF age = “31…40” THEN buys_computer = “yes” IF age = “>40” AND credit_rating = “excellent” THEN buys_computer = “yes” IF age = “>40” AND credit_rating = “fair” THEN buys_computer = “no”

An Example of ‘Car Buyers’ – Who buys Lexton? no Job M/F Area Age Y/N 1 NJ M N 35 2 F 51 3 OW 31 Y 4 EM 38 5 S 33 6 54 7 49 8 32 9 10 11 12 50 13 36 14 Job (14,5,9) Emplyee (5,2,3) Owner (4,0,4) No Job (5,3,2) Age Below 43 (3,0,3) Above 43 (2,2,0) Y Res. Area N South (2,0,2) North (3,3,0) * (a,b,c) means a: total # of records, b: ‘N’ counts, c: ‘Y’ counts

Lab on Decision Tree(1) SPSS Clementine, SAS Enterprise Miner See5/C5.0Download See5/C5.0 2.02 Evaluation from http://www.rulequest.com

Lab on Decision Tree(2) From below initial screen, choose File – Locate Data

Lab on Decision Tree(3) Select housing.data from Samples folder and click open.

Lab on Decision Tree(3(4) This data set is on deciding house price in Boston area. It has 350 cases and 13 variables.

Lab on Decision Tree (5) Input variables crime rate proportion large lots: residential space proportion industrial: ratio of commercial area CHAS: dummy variable nitric oxides ppm: polution rate in ppm av rooms per dwelling: # of room for dwelling proportion pre-1940 distance to employment centers: distance to the center of city accessibility to radial highways: accessibility to high way property tax rate per $10\,000 pupil-teacher ratio: teachers’ rate B: racial statistics percentage low income earners: ratio of low income people Decision variable Top 20%, Bottom 80%

Lab on Decision Tree(6) For the analysis, click Construct Classifier or click Construct Classifier from File menu

Lab on Decision Tree(7) Click on Global pruning to (V ). Then, click OK

Lab on Decision Tree(8) Decision Tree Evaluation with Training data Evaluation with Test data

Lab on Decision Tree(9) Understanding picture We can see that (av rooms per dwelling) is the most important variable in deciding house price.

Lab on Decision Tree(11) 의사결정나무 그림으로는 규칙을 알아보기 어렵다. To view the rules, close current screen and click Construct Classifier again or click Construct Classifier from File menu.

Lab on Decision Tree(12) Choose/click Rulesets. Then click OK.

Lab on Decision Tree(13)

How decision tree is derived from a data set : A case of predicting Play/Not Play with weather information

A sample problem Predict Play or Not Play (ex: Playing Golf) with independent variables such as outlook temperature humidity windy

Output Variables(decision variables) .Play (golf) .Not Play(golf)

Data set

But, it still needs to be refined! Sort data with outlook But, it still needs to be refined!

Final Decision Tree Induced from Data

4. 인공신경망 (Neural Networks) 125

Table of Contents I. Introduction of Neural Networks II. Application of Neural Networks III. Theory of Neural Networks IV. A Neural Network Demo

What is neural networks ? http://www.youtube.com/watch?v=DG5- UyRBQD4&feature=rellist&playnext=1&list=PL4FA5D71B0 BA92C1C

I. Introduction of Neural Networks It is simulation of human brain It is the most well known artificial intelligence techniques We are using them: voice recognition system, reading hand writes, door rocks et al. It is a called black box

It is a simulator for human brain Neural Networks simulate human brain Learning in Human Brain Neurons Connection Between Neurons Neural Networks As Simulator For Human Brain Processing Elements or Nodes Weights

II. Applications of Neural Networks Prediction of Outcomes Patterns Detection in Data Classification

Business ANN Applications -1 Accounting Identify tax fraud Enhance auditing by finding irregularities Finance Signatures and bank note verifications Foreign exchange rate forecasting Bankruptcy prediction Customer credit scoring Credit card approval and fraud detection* Stock and commodity selection and trading Forecasting economic turning points Pricing initial public offerings* Loan approvals

Business ANN Applications -2 Human Resources Predicting employees’ performance and behavior Determining personnel resource requirements Management Corporate merger prediction Country risk rating Marketing Consumer spending pattern classification Sales forecasts Targeted marketing, … Operations Vehicle routing Production/job scheduling, …

III. Theory of Neural Networks Neural Computing is a problem solving methodology that attempts to mimic how human brain functions Artificial Neural Networks (ANN) Machine Learning/Artificial Intelligence

The Biological Analogy Neurons: brain cells Nucleus (at the center) Dendrites provide inputs Axons send outputs Synapses increase or decrease connection strength and cause excitation or inhibition of subsequent neurons

Artificial Neural Networks (ANN) Biological Artificial Soma <-> Node Dendrites <-> Input Axon <-> Output Synapse <-> Weight Three Interconnected Artificial Neurons

Basic structure of Neural Networks Network Structure : Layers, Nodes and Weights Input Layer Hidden Layer Output Layer

ANN Fundamentals

ANN Fundamentals: how informatio is processed in ANN Processing Information by the Network Inputs Outputs Weights Summation Function Figure 15.5

Learning in NN(Neural Network) is finding the best numeric values (X), representing input (4) and output(8) relationship ( ex: 4 * X = 8 ) *Try with x= 1, x= 2, x=3, …… When x=4, it solve the problem. Compute outputs Compare outputs with desired targets Adjust the weights and repeat the process

Neural Network Architecture There are several ANN architectures :feed forward, recurrent, Hopfield et al.

Neural Network Architecture Feed forward Neural Network : Multi Layer Perceptron, - Two, Three, sometimes Four or Five Layers, But normally 3 layers are common structure.

How a Network Learns Step function evaluates the summation of input values Calculating outputs Measure the error (delta) between outputs and desired values Update weights, reinforcing correct results At any step in the process for a neuron, j, we get Delta(Error) = Zj - Yj where Z and Y are the desired and actual outputs, respectively

Backpropagation Drawbacks: Initialize the weights Read the input vector Generate the output Compute the error Error = Output – Desired output Change the weights Drawbacks: A large network can take a very long time to train May not converge

Training A Neural Networks Neural Networks learn from data Learning is finding the best weights values which represent the input and output relationship in Neural Networks (ex: 4*X= 8)-> finding the value for X

training data set and test data set Collect data and separate it into Training set (50%), Testing set (50%) Training set (60%), Testing set (40%) Training set (70%), Testing set (30%) Training set (80%), Testing set (20%) Training set (90%), Testing set (10%) Use training data set to build model Use test data set to validate the trained network

Prediction with New Data If the Neural Network's performance in test is good , it can be used to predict outcome of new unseen data If the performance with test is not good, you should collect more data, add more input variables

How does Neural Network work for prediction? Terms in Neural Networks

Demo – How does Neural Network work for prediction?

ANN Development Tools E-Miner Clementine Trajan Neural Network Simulator NeuroSolutions NeuroShell Easy Statistica Neural Network Toolkit SPSS Neural Connector Braincel (Excel Add-in) NeuroWare NeuralWorks Brainmaker PathFinder

Why use Neural Networks in Prediction Why use Neural Networks in Prediction? - major benefits of Neural Networks

Benefits of ANN Advantages: Non-linear model leads to better performance It works generally good when data size is small It works generally good when there are noises in data It works generally good when there are missing in data (incomplete data set) Fast decision making Diverse Applications: Pattern recognition Character, speech and visual recognition

Limitations of ANN Black box that is hardly understood by human Lack of explanation capabilities Training time can be excessive and tedious

IV. A Neural Networks Demo How do neural networks learn? : trials and errors http://www.youtube.com/watch?v=0Str0Rdkxxo

5. 사례기반추론 (Case-Based Reasoning) 154

Case-Based Reasoning (CBR) A methodology in which knowledge and/or infe rences are derived from historical cases Definition and concepts of cases in CBR Stories Cases with rich information and episodes. Lessons may be derived from this kind of cases in a case base

Case-based reasoning Case-based reasoning (CBR), broadly construed, is the process of solving new problems based on the solutions of similar past problems. An auto mechanic who fixes an engine by recalling another car that exhibited similar symptoms is using case-based reasoning. A lawyer who advocates a particular outcome in a trial based on legal precedents or a judge who creates case law is using case-based reasoning.

It has been argued that case-based reasoning is not only a powerful method for computer reasoning, but also a pervasive behavior in everyday human problem solving; or, more radically, that all reasoning is based on past cases personally experienced. This view is related to prototype theory, which is most deeply explored in cognitive science.

Case-Based Reasoning (CBR)

Case-Based Reasoning (CBR) Benefits and usability of CBR CBR makes learning much easier and the recommen dation more sensible

Case-Based Reasoning (CBR) Advantages of using CBR Knowledge acquisition is improved. System development time is faster Existing data and knowledge are leveraged Complete formalized domain knowledge is not requi red Experts feel better discussing concrete cases Explanation becomes easier Acquisition of new cases is easy Learning can occur from both successes and failures

Case-Based Reasoning (CBR)

CBR solves problems using the already stored knowledge, and captures new knowledge, making it immediately available for solving the next problem. Therefore, case-based reasoning can be seen as a method for problem solving, and also as a method to capture new experience and make it immediately available for problem solving.

It can be seen as a learning and knowledge-discovery approach, since it can capture from new experience some general knowledge, such as case classes, prototypes and some higher-level concept. The idea of case-based reasoning originally came from the cognitive science community which discovered that people are rather reasoning on formerly successfully solved cases than on general rules.

The case-based reasoning community aims to develop computer models that follow this cognitive process. For many application areas computer models have been successfully developed, which were based on CBR, such as signal/image processing and interpretation tasks, help-desk applications, medical applications and E-commerce product-selling systems.

In the tutorial we will explain the case-based reasoning process scheme. We will show what kind of methods are necessary to provide all the functions for such a computer model. We will develop the bridge between CBR and other disciplines. Examples will be given based on signal-interpreting applications and information management.

Case-based reasoning is a problem solving paradigm that in many respects is fundamentally different from other major AI approaches. Instead of relying solely on general knowledge of a problem domain, or making associations along generalized relationships between problem descriptors and conclusions, CBR is able to utilize the specific knowledge of previously experienced, concrete problem situations (cases).

A new problem is solved by finding a similar past case, and reusing it in the new problem situation. A second important difference is that CBR also is an approach to incremental, sustained learning, since a new experience is retained each time a problem has been solved, making it immediately available for future problems. The CBR field has grown rapidly over the last few years, as seen by its increased share of papers at major conferences, available commercial tools, and successful applications in daily use.

4 step processes in CBR 1. Retrieve: Given a target problem, retrieve from memory cases relevant to solving it. A case consists of a problem, its solution, and, typically, annotations about how the solution was derived. For example, suppose Fred wants to prepare blueberry pancakes. Being a novice cook, the most relevant experience he can recall is one in which he successfully made plain pancakes. The procedure he followed for making the plain pancakes, together with justifications for decisions made along the way, constitutes Fred's retrieved case.

2. Reuse: Map the solution from the previous case to the target problem. This may involve adapting the solution as needed to fit the new situation. In the pancake example, Fred must adapt his retrieved solution to include the addition of blueberries. 3. Revise: Having mapped the previous solution to the target situation, test the new solution in the real world (or a simulation) and, if necessary, revise. Suppose Fred adapted his pancake solution by adding blueberries to the batter. After mixing, he discovers that the batter has turned blue – an undesired effect. This suggests the following revision: delay the addition of blueberries until after the batter has been ladled into the pan.

4. Retain: After the solution has been successfully adapted to the target problem, store the resulting experience as a new case in memory. Fred, accordingly, records his new-found procedure for making blueberry pancakes, thereby enriching his set of stored experiences, and better preparing him for future pancake-making demands.

Comparison to other methods At first glance, CBR may seem similar to the rule induction algorithms of machine learning. Like a rule-induction algorithm, CBR starts with a set of cases or training examples; it forms generalizations of these examples, albeit implicit ones, by identifying commonalities between a retrieved case and the target problem.

Prominent CBR systems SMART: Support management automated reasoning technology for Compaq customer service CoolAir: HVAC specification and pricing system Vidur - A CBR based intelligent advisory system, by C-DAC Mumbai, for farmers of North-East India. jCOLIBRI - A CBR framework that can be used to build other custom user-defined CBR systems. CAKE - Collaborative Agile Knowledge Engine. Edge Platform - Applies CBR to the healthcare, oil & gas and financial services sectors.