Data Mining and Neural Networks Danny Leung CS157B, Spring 2006 Professor Sin-Min Lee

2 Artificial Intelligence for Data Mining Neural networks are useful for data mining and decision-support applications. Neural networks are useful for data mining and decision-support applications. People are good at generalizing from experience. People are good at generalizing from experience. Computers excel at following explicit instructions over and over. Computers excel at following explicit instructions over and over. Neural networks bridge this gap by modeling, on a computer, the neural behavior of human brains. Neural networks bridge this gap by modeling, on a computer, the neural behavior of human brains.

3 Neural Network Characteristics Neural networks are useful for pattern recognition or data classification, through a learning process. Neural networks are useful for pattern recognition or data classification, through a learning process. Neural networks simulate biological systems, where learning involves adjustments to the synaptic connections between neurons Neural networks simulate biological systems, where learning involves adjustments to the synaptic connections between neurons

4 Anatomy of a Neural Network Neural Networks map a set of input-nodes to a set of output-nodes Neural Networks map a set of input-nodes to a set of output-nodes Number of inputs/outputs is variable Number of inputs/outputs is variable The Network itself is composed of an arbitrary number of nodes with an arbitrary topology The Network itself is composed of an arbitrary number of nodes with an arbitrary topology

5 Biological Background A neuron: many-inputs / one-output unit A neuron: many-inputs / one-output unit Output can be excited or not excited Output can be excited or not excited Incoming signals from other neurons determine if the neuron shall excite ("fire") Incoming signals from other neurons determine if the neuron shall excite ("fire") Output subject to attenuation in the synapses, which are junction parts of the neuron Output subject to attenuation in the synapses, which are junction parts of the neuron

6 Basics of a Node  A node is an element which performs a function y = f H (∑(w i x i ) + W b ) Connection Node

7 A Simple Preceptron Binary logic application Binary logic application f H (x) [linear threshold] f H (x) [linear threshold] Wi = random(-1,1) Wi = random(-1,1) Y = u(W0X0 + W1X1 + Wb) Y = u(W0X0 + W1X1 + Wb)

8 Preceptron Training It’s a single-unit network It’s a single-unit network Adjust weights based on a how well the current weights match an objective Adjust weights based on a how well the current weights match an objective Perceptron Learning Rule Δ W i = η * (D-Y).I i –η = Learning Rate –D = Desired Output

9 Neural Network Learning From experience: examples / training data From experience: examples / training data Strength of connection between the neurons is stored as a weight-value for the specific connection Strength of connection between the neurons is stored as a weight-value for the specific connection Learning the solution to a problem = changing the connection weights Learning the solution to a problem = changing the connection weights

10 Neural Network Learning Continuous Learning Process Continuous Learning Process Evaluate output Evaluate output Adapt weights Adapt weights Take new inputs Take new inputs Learning causes stable state of the weights Learning causes stable state of the weights

11 Learning Performance Supervised Supervised –Need to be trained ahead of time with lots of data Unsupervised networks adapt to the input Unsupervised networks adapt to the input –Applications in Clustering and reducing dimensionality –Learning may be very slow –No help from the outside –No training data, no information available on the desired output –Learning by doing –Used to pick out structure in the input: –Clustering –Compression

12 Topologies – Back- Propogated Networks Inputs are put through a ‘Hidden Layer’ before the output layer Inputs are put through a ‘Hidden Layer’ before the output layer All nodes connected between layers All nodes connected between layers

13 BP Network – Supervised Training Desired output of the training examples Desired output of the training examples Error = difference between actual & desired output Error = difference between actual & desired output Change weight relative to error size Change weight relative to error size Calculate output layer error, then propagate back to previous layer Calculate output layer error, then propagate back to previous layer Hidden weights updated Hidden weights updated Improved performance Improved performance

14 Neural Network Topology Characteristics Set of inputs Set of inputs Set of hidden nodes Set of hidden nodes Set of outputs Set of outputs Increasing nodes makes network more difficult to train Increasing nodes makes network more difficult to train

15 Applications of Neural Networks Prediction – weather, stocks, disease Prediction – weather, stocks, disease Classification – financial risk assessment, image processing Classification – financial risk assessment, image processing Data association – Text Recognition (OCR) Data association – Text Recognition (OCR) Data conceptualization – Customer purchasing habits Data conceptualization – Customer purchasing habits Filtering – Normalizing telephone signals (static) Filtering – Normalizing telephone signals (static)

16 Overview Advantages Advantages –Adapt to unknown situations –Robustness: fault tolerance due to network redundancy –Autonomous learning and generalization Disadvantages Disadvantages –Not exact –Large complexity of the network structure

17 Referenced Work Intro to Neural Networks - Computer Vision Applications and Training Techniques. Doug Gray. GroupMeeting/NN_Doug_2004_12_1.ppt Intro to Neural Networks - Computer Vision Applications and Training Techniques. Doug Gray. GroupMeeting/NN_Doug_2004_12_1.ppt Introduction to Artificial Neural Networks. Nicolas Galoppo von Borries. comp290-58/nn-presentation/ann-intro.ppt Introduction to Artificial Neural Networks. Nicolas Galoppo von Borries. comp290-58/nn-presentation/ann-intro.ppt