1. CHAPTER 15 Neural Computing: The Basics

2. n Artificial Neural Networks (ANN) n Mimics How Our Brain Works n Machine Learning

3. Machine Learning: An Overview n ANN to automate complex decision making n Neural networks learn from past experience and improve their performance levels n Machine learning: methods that teach machines to solve problems or to support problem solving, by applying historical cases

4. Complications n Many models of learning n Match the learning model with problem type

5. What is Learning? n Through analogy, discovery, and special procedures; by observing; or by analyzing examples n Can improve the performance of AI methods n Is a support area of AI

6. Learning as Related to AI n Learning systems demonstrate interesting learning behaviors n No claims about learning as well as humans or in the same way n Learning systems are not defined very formally; implications are not well understood n Learning in AI involves the manipulation of symbols (not numeric information)

7. Machine Learning Method Examples n Neural Computing n Inductive Learning n Case-based Reasoning and Analogical Reasoning n Genetic Algorithms n Statistical Methods n Explanation-based Learning

8. Neural Computing n Computers that mimic certain processing capabilities of the human brain n Knowledge representations based on –Massive parallel processing –Fast retrieval of large amounts of information –The ability to recognize patterns based on historical cases Neural Computing = Artificial Neural Networks (ANNs)

9. The Biology Analogy Biological Neural Networks (Figure 15.1) n Neurons: brain cells –Nucleus (at the center) –Dendrites provide inputs –Axons send outputs n Synapses increase or decrease connection strength and cause excitation or inhibition of subsequent neurons

10. Artificial Neural Networks (ANN) n A model that emulates a biological neural network n Software simulations of the massively parallel processes that involve processing elements interconnected in a network architecture n Originally proposed as a model of the human brain’s activities n The human brain is much more complex

12. Neural Network Fundamentals n Components and Structure –Processing Elements –Network –Structure of the Network n Processing Information by the Network –Inputs –Outputs –Weights –Summation Function

13. jth Neuron (Figure 15.3) n Transformation (Transfer) Function n Sigmoid Function (Logical Activation Function) –where Y T is the transformed (normalized) value of Y

14. Processing Information in an Artificial Neuron (Figure 15.3) x1x1 w 1j x2x2 xixi YjYj w ij w 2j Neuron j  w ij x i Weights Output Inputs SummationsTransfer function 

15. Learning: Three Tasks 1. Compute Outputs 2. Compare Outputs with Desired Targets 3. Adjust Weights and Repeat the Process

16. n Set the weights by either rules or randomly n Set Delta = Error = actual output minus desired output for a given set of inputs n Objective is to Minimize the Delta (Error) n Change the weights to reduce the Delta n Information processing: pattern recognition n Different learning algorithms

17. Neural Network Application Development n Preliminary steps of system development are done n ANN Application Development Process 1. Collect Data 2. Separate into Training and Test Sets 3. Define a Network Structure 4. Select a Learning Algorithm 5. Set Parameters, Values, Initialize Weights 6. Transform Data to Network Inputs 7. Start Training, and Determine and Revise Weights 8. Stop and Test 9. Implementation: Use the Network with New Cases

18. Data Collection and Preparation n Collect data and separate into a training set and a test set n Use training cases to adjust the weights n Use test cases for network validation

19. Neural Network Architecture Representative Architectures n Associative Memory Systems –Associative memory - ability to recall complete situations from partial information –Systems correlate input data with stored information –Hidden Layer –Three, Sometimes Four or Five Layers

20. Recurrent Structure n Recurrent network (double layer) - each activity goes through the network more than once before the output response is produced n Uses a feedforward and feedbackward approach to adjust parameters to establish arbitrary numbers of categories n Example: Hopfield

21. Neural Network Preparation (Non-numerical Input Data (text, pictures): preparation may involve simplification or decomposition) n Choose the learning algorithm n Determine several parameters –Learning rate (high or low) –Threshold value for the form of the output –Initial weight values –Other parameters n Choose the network's structure (nodes and layers) n Select initial conditions n Transform training and test data to the required format

22. Training the Network n Present the training data set to the network n Adjust weights to produce the desired output for each of the inputs –Several iterations of the complete training set to get a consistent set of weights that works for all the training data

23. Learning Algorithms n Two Major Categories Based On Input Format –Binary-valued (0s and 1s) –Continuous-valued n Two Basic Learning Categories –Supervised Learning –Unsupervised Learning

24. Supervised Learning n For a set of inputs with known (desired) outputs n Examples –Backpropagation –Hopfield network

25. Unsupervised Learning n Only input stimuli shown to the network n Network is self-organizing n Number of categories into which the network classifies the inputs can be controlled by varying certain parameters n Examples –Adaptive Resonance Theory (ART) –Kohonen Self-organizing Feature Maps

26. How a Network Learns Single neuron - learning the inclusive OR operation Two input elements, X 1 and X 2 Inputs CaseX 1 X 2 Desired Results 1 000 2011 (positive) 3101 (positive) 4111 (positive)

27. –Step function evaluates the summation of input values n Calculating outputs –Measure the error (delta) between outputs and desired values –Update weights, reinforcing correct results At any step in the process for a neuron, j, we get Delta = Z j - Y j where Z and Y are the desired and actual outputs, respectively

28. Updated Weights are W i (final) = W i (initial) + alpha × delta × X 1 where alpha is the learning rate parameter n Weights are initially random n The learning rate parameter, alpha, is set low n Delta is used to derive the final weights, which then become the initial weights in the next iteration (row) n Threshold value parameter: sets Y to 1 in the next row if the weighted sum of inputs is greater than 0.5; otherwise, to 0

29. Backpropagation n Backpropagation (back-error propagation) n Most widely used learning n Relatively easy to implement n Requires training data for conditioning the network before using it for processing other data n Network includes one or more hidden layers n Network is considered a feedforward approach Continue

30. n Externally provided correct patterns are compared with the neural network output during training (supervised training) n Feedback adjusts the weights until all training patterns are correctly categorized

31. n Error is backpropogated through network layers n Some error is attributed to each layer n Weights are adjusted n A large network can take a very long time to train n May not converge

32. Testing n Test the network after training n Examine network performance: measure the network’s classification ability n Black box testing n Do the inputs produce the appropriate outputs? n Not necessarily 100% accurate n But may be better than human decision makers n Test plan should include –Routine cases –Potentially problematic situations n May have to retrain

33. Implementation n Frequently requires –Interfaces with other CBIS –User training n Gain confidence of the users and management early

34. Neural Computing Paradigms Decisions the builder must make n Size of training and test data n Learning algorithms n Topology: number of processing elements and their configurations n Transformation (transfer) function n Learning rate for each layer n Diagnostic and validation tools Results in the Network's Paradigm

35. Neural Network Software n Program in: –Programming language –Neural network package or NN programming tool –Both n Tools (shells) incorporate: –Training algorithms –Transfer and summation functions n May still need to: –Program the layout of the database –Partition the data (test data, training data) –Transfer the data to files suitable for input to an ANN tool

36. NN Development Tools n Braincel (Excel Add-in) n NeuralWorks n Brainmaker n PathFinder n Trajan Neural Network Simulator n NeuroShell Easy n SPSS Neural Connector n NeuroWare

37. Neural Network Hardware n Massive parallel processing greatly enhances performance

38. Possible Hardware Systems for ANN Training n Faster general purpose computers n General purpose parallel processors n Neural chips n Acceleration boards n Parallel graphics processors

39. Neural Network Development Examples n Electricity Demand –BrainCel –Screens: Figures 15.12a, b, c n GPA Predictor –BrainMaker –Screens: Figures 15.13a, b

40. The Self-Organizing Map: An Alternative NN Architecture n Kohonen Self-Organizing Map (SOM) n Unsupervised learning n Weights self-adjust to input pattern n Topology Figure 15.14

41. Benefits of Neural Networks n Pattern recognition, learning, classification, generalization and abstraction, and interpretation of incomplete and noisy inputs n Character, speech and visual recognition n Can provide some human problem-solving characteristics n Can tackle new kinds of problems n Robust n Fast n Flexible and easy to maintain n Powerful hybrid systems

42. Limitations of Neural Networks n Do not do well at tasks that are not done well by people n Lack explanation capabilities n Limitations and expense of hardware technology restrict most applications to software simulations n Training time can be excessive and tedious n Usually requires large amounts of training and test data

43. Neural Networks and Expert Systems Different technologies complement each other n Expert systems: logical, symbolic approach n Neural networks: model-based, numeric and associative processing n Main features of each (Table 15.2)

44. Expert Systems n Good for closed-system applications (literal and precise inputs, logical outputs) n Reason with established facts and pre-established rules

45. Major Limitations n Experts do not always think in terms of rules n Experts may not be able to explain their line of reasoning n Experts may explain incorrectly n Sometimes difficult or impossible to build knowledge base

46. Neural Computing Use Neural Networks in Knowledge Acquisition n Fast identification of implicit knowledge by automatically analyzing cases of historical data n ANN identifies patterns and relationships that may lead to rules for expert systems n A trained neural network can rapidly process information to produce associated facts and consequences

47. Neural Networks For Decision Support n Inductive means for gathering, storing, and using experiential knowledge n Neural network-based DSS to appraise real estate in New York (90% accurate) n Forecasting n ANN in decision support: Easy sensitivity analysis and partial analysis of input factors n The relationship between a combined expert system, ANN and a DSS (Figure 15.15) n ANN can expand the boundaries of DSS