Computer Science Department FMIPA IPB 2003 Neural Computing Yeni Herdiyeni Computer Science Dept. FMIPA IPB

Computer Science Department FMIPA IPB 2003 Neural Computing : The Basic Artificial Neural Networks (ANN) Artificial Neural Networks (ANN) Mimics How Our Brain Works Mimics How Our Brain Works Machine Learning Machine Learning Neural Computing = Artificial Neural Networks (ANNs)

Computer Science Department FMIPA IPB 2003 Machine Learning : Overview ANN to automate complex decision making ANN to automate complex decision making Neural networks learn from past experience and improve their performance levels Neural networks learn from past experience and improve their performance levels Machine learning: methods that teach machines to solve problems or to support problem solving, by applying historical cases Machine learning: methods that teach machines to solve problems or to support problem solving, by applying historical cases

Computer Science Department FMIPA IPB 2003 Neural Network and Expert System Different technologies complement each other Expert systems: logical, symbolic approach Expert systems: logical, symbolic approach Neural networks: model-based, numeric and associative processing Neural networks: model-based, numeric and associative processing

Computer Science Department FMIPA IPB 2003 Expert System Good for closed-system applications (literal and precise inputs, logical outputs) Good for closed-system applications (literal and precise inputs, logical outputs) Reason with established facts and pre- established rules Reason with established facts and pre- established rules

Computer Science Department FMIPA IPB 2003 Major Limitation ES Experts do not always think in terms of rules Experts do not always think in terms of rules Experts may not be able to explain their line of reasoning Experts may not be able to explain their line of reasoning Experts may explain incorrectly Experts may explain incorrectly Sometimes difficult or impossible to build knowledge base Sometimes difficult or impossible to build knowledge base

Computer Science Department FMIPA IPB 2003 Neural Computing Use : Neural Networks in Knowledge Acquisition Fast identification of implicit knowledge by automatically analyzing cases of historical data Fast identification of implicit knowledge by automatically analyzing cases of historical data ANN identifies patterns and relationships that may lead to rules for expert systems ANN identifies patterns and relationships that may lead to rules for expert systems A trained neural network can rapidly process information to produce associated facts and consequences A trained neural network can rapidly process information to produce associated facts and consequences

Computer Science Department FMIPA IPB 2003 Benefit NN Pattern recognition, learning, classification, generalization and abstraction, and interpretation of incomplete and noisy inputs Pattern recognition, learning, classification, generalization and abstraction, and interpretation of incomplete and noisy inputs Character, speech and visual recognition Character, speech and visual recognition Can provide some human problem-solving characteristics Can provide some human problem-solving characteristics Can tackle new kinds of problems Can tackle new kinds of problems Robust Robust Fast Fast Flexible and easy to maintain Flexible and easy to maintain Powerful hybrid systems Powerful hybrid systems

Computer Science Department FMIPA IPB 2003 Biology Analogy : Biological Neural Network Neurons: brain cells Neurons: brain cells –Nucleus (at the center) –Dendrites provide inputs –Axons send outputs Synapses increase or decrease connection strength and cause excitation or inhibition of subsequent neurons Synapses increase or decrease connection strength and cause excitation or inhibition of subsequent neurons

Computer Science Department FMIPA IPB 2003 Biology Analogy : Biological Neural Network

Computer Science Department FMIPA IPB 2003 Neural Network ? Neural Network is a networks of many simple processors, each possibly having a small amount of local memory. The processors are connected with communication channels (synapses).

Computer Science Department FMIPA IPB 2003 Neural Network (Haykin*) Neural Network is a massively parallel- distributed processor that has a natural prosperity for storing experiential knowledge and making it available for use. Simon Haykin, Neural Networks: A Comprehensive Foundation, Prentice Hall Inc., New Jersey, 1999.

Computer Science Department FMIPA IPB 2003 Neural Net = Brain ? Knowledge is acquired by the network through a learning process Inter-neuron connection strengths known as synaptic weights are used to store the knowledge.

Computer Science Department FMIPA IPB 2003 Neural Network Fundamentals Components and Structure Components and Structure –Processing Elements –Network –Structure of the Network Processing Information by the Network Processing Information by the Network –Inputs –Outputs –Weights –Summation Function

Computer Science Department FMIPA IPB 2003 Processing Information in an Artificial Neuron x1x1 w 1j x2x2 xixi YjYj w ij w 2j Neuron j  w ij x i Weights Output Inputs SummationsTransfer function 

Computer Science Department FMIPA IPB 2003 Learning : 3 Tasks 1. Compute Outputs 2. Compare Outputs with Desired Targets 3. Adjust Weights and Repeat the Process

Computer Science Department FMIPA IPB 2003 Training The Network Present the training data set to the network Present the training data set to the network Adjust weights to produce the desired output for each of the inputs Adjust weights to produce the desired output for each of the inputs –Several iterations of the complete training set to get a consistent set of weights that works for all the training data

Computer Science Department FMIPA IPB 2003 Testing Test the network after training Test the network after training Examine network performance: measure the network’s classification ability Examine network performance: measure the network’s classification ability Black box testing Black box testing Do the inputs produce the appropriate outputs? Do the inputs produce the appropriate outputs? Not necessarily 100% accurate Not necessarily 100% accurate But may be better than human decision makers But may be better than human decision makers Test plan should include Test plan should include –Routine cases –Potentially problematic situations May have to retrain May have to retrain

Computer Science Department FMIPA IPB 2003 ANN Application Development Process 1. Collect Data 2. Separate into Training and Test Sets 3. Define a Network Structure 4. Select a Learning Algorithm 5. Set Parameters, Values, Initialize Weights 6. Transform Data to Network Inputs 7. Start Training, and Determine and Revise Weights 8. Stop and Test 9. Implementation: Use the Network with New Cases

Computer Science Department FMIPA IPB 2003 Data Collection and Preparation Collect data and separate into a training set and a test set Collect data and separate into a training set and a test set Use training cases to adjust the weights Use training cases to adjust the weights Use test cases for network validation Use test cases for network validation

Computer Science Department FMIPA IPB 2003 Single Layer Perceptron

Computer Science Department FMIPA IPB 2003 Each pass through all of the training input and target vector is called an epoch.

Computer Science Department FMIPA IPB 2003 Example :

Computer Science Department FMIPA IPB 2003

Disadvantage Perceptron Perceptron networks can only solve linearly separable problems see:Marvin Minsky and Seymour Papert’s book Perceptron [10]. [10] M.L. Minsky, S.A. Papert, Perceptrons: An Introduction To Computational Geometry, MIT Press, See XOR problem

Computer Science Department FMIPA IPB 2003 Multilayer Perceptrons (MLP)

Computer Science Department FMIPA IPB 2003 MLP MLP has ability to learn complex decision boundaries MLPs are used in many practical computer vision applications involving classification (or supervised segmentation).

Computer Science Department FMIPA IPB 2003 Backpropagation

X = -1 : 0.1 : 1; Y = [ ]; Normalisasi : pr = [-1 1]; m1 = 5; m2 = 1; net_ff = newff (pr, [m1 m2], {'logsig' 'purelin'}); net_ff = init (net_ff); %Default Nguyen-Widrow initialization %Training: net_ff.trainParam.goal = 0.02; net_ff.trainParam.epochs = 350; net_ff = train (net_ff, X, Y); %Simulation: X_sim = -1 : 0.01 : 1; Y_nn = sim (net_ff, X_sim);

Computer Science Department FMIPA IPB 2003 Backpropagation Backpropagation (back-error propagation) Backpropagation (back-error propagation) Most widely used learning Most widely used learning Relatively easy to implement Relatively easy to implement Requires training data for conditioning the network before using it for processing other data Requires training data for conditioning the network before using it for processing other data Network includes one or more hidden layers Network includes one or more hidden layers Network is considered a feedforward approach Network is considered a feedforward approach

Computer Science Department FMIPA IPB 2003 Externally provided correct patterns are compared with the neural network output during training (supervised training) Externally provided correct patterns are compared with the neural network output during training (supervised training) Feedback adjusts the weights until all training patterns are correctly categorized Feedback adjusts the weights until all training patterns are correctly categorized

Computer Science Department FMIPA IPB 2003 Error is backpropogated through network layers Error is backpropogated through network layers Some error is attributed to each layer Some error is attributed to each layer Weights are adjusted Weights are adjusted A large network can take a very long time to train A large network can take a very long time to train May not converge May not converge

Computer Science Department FMIPA IPB 2003 Next Time ….. ANFIS Neural Network By Ir. Agus Buono, M.Si, M.Komp