IE 585 Introduction to Neural Networks

2 Modeling Continuum Unarticulated Wisdom Articulated Qualitative Models Theoretic (First Principles) Models Empirical Categorical Models Empirical Continuous Models Low Cost / High Error High Cost / Low Error

3 Rise of Empirical Models Sensoring - lots of data Fast computing Computing available on site More complicated systems - do not adhere to simple models Easy to use software

4 Typical Empirical Models linear regression splines nearest neighbor clustering neural networks

5 What is a Neural Net? An NN is a network of many simple processors (“units, neurons”), each possibly having a small amount of local memory. The units are connected by communication channels (“connections”) which usually carry numeric data, encoded by any of various means. The units operate only on their local data and on the inputs they receive via the connections. Usenet newsgroup comp.ai.neural-nets

6 What is a Neural Net? An NN is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: 1. Knowledge is acquired by the network through a learning process. 2. Interneuron connection strengths known as synaptic weights are used to store the knowledge. Haykin (1994)

7 Objectives of Neural Nets High Computing Speed Large Memory Capacity Adaptive Learning Fault Tolerance

8 Neural Network Predictive Models - Advantages Can accommodate non-linear relationships with interactions among variables Generalize well even for noisy and imprecise data No assumption of analytical function or theoretic relation needed User friendly software available Computationally very fast, once built

9 Neural Network Predictive Models - Disadvantages Strongly data dependent No statistical interpretation of significance or confidence Difficult to build and validate properly - too many choices, too little general guidance, misleading validation results

10 How Do Neural Networks Work? Inspired by the biological brain Consist of small, but numerous, parallel elements - weighted connections (synapses) and summing nodes (neurons) “Learn” relationships through repeated calculations called “training” Remain fixed after training to be applied to new data

11 Biological Neuron

12 How are Signals Transmitted?

13 Elements of Neural Networks

14 Typical Neural Network Hidden Layer Output Layer Input Layer Error Feedback Weighted Synapses During Training Neural Network Output INPUTS INPUTS

15 Terminology Neurons / nodes / units / cells / processing elements (PEs) Transfer / activation function Connections / links / synapses Weights / bias (fixed input of 1) Feedforward / feedback Input / output vectors / patterns Self organizing (unsupervised) / supervised Training / testing data sets

16 Biological vs Artificial Neural Networks Biological neurons are all excitatory (positive) or inhibitory (negative) - ANN neurons can be mixed Biological neurons operate asynchronously - ANN neurons usually synchronize by layer Biological neurons transmit signals at varying rates but ANN use a single rate

17 Biological vs Artificial Neural Networks There are many specialized biological neurons - ANN neurons tend to be generic Biological neurons work through chemical / electrical transmission (“wet ware”) Biological neurons are much slower but there are many, many more of them (~ neurons with 10 4 synapses per neuron!)

18 Types of Neural Nets Supervised Unsupervised Associate Optimization

19 Common Neural Net Applications Pattern classification / recall –medical –defense –manufacturing quality –machine vision / postal –speech recognition –security detection –noise removal

20 Common Neural Net Applications Clustering / compression –data mining –signal processing –space exploration applications –speech recognition

21 Common Neural Net Applications Prediction / simulation –financial / stock market –music composition –utility usage –fault / degradation detection –sunspots

22 Common Neural Net Applications Control - real time / on line –robots –vehicles –manufacturing Control - off line –batch manufacturing –process optimization

23 Common Neural Net Applications Optimization –traveling salesman –routing –scheduling –facility location

24 Cool Neural Net Web Sites Detailed class notes and some matlab code. C source code for lots of neural nets.