1 Neural Networks - Basics Artificial Neural Networks - Basics Uwe Lämmel Business School Institute of Business Informatics

2 Neural Networks - Basics Neural Networks Idea Artificial Neuron & Network Supervised Learning Unsupervised Learning Data Mining – other Techniques

3 Neural Networks - Basics Artificial Neuron & Network Idea An artificial Neuron Neural Network Example Learning Application

4 Neural Networks - Basics Idea A human being learns by example “learning by doing” –seeing(Perception), Walking, Speaking,… Can a machine do the same? A human being uses his brain. A brain consists of millions of single cells. A cell is connected with ten thousands of other cell. Is it possible to simulate a similar structure on a computer?

5 Neural Networks - Basics Idea Artificial Neural Network –Information processing similar to processes in a mammal brain –heavy parallel systems, –able to learn –great number of simple cells ? Is it useful to copy nature ? –wheel, aeroplane,...

6 Neural Networks - Basics Idea we need: –software neurons –software connections between neurons –software learning algorithms An artificial neural network functions in a similar way a natural neural network does.

7 Neural Networks - Basics A biological Neuron Dendrits:(Input) Getting other activations Axon:(Output ) forward the activation (from 1mm up to 1m long) Synapse:transfer of activation: –to other cells, e.g. Dendrits of other neurons –a cell has about to connections to other cells Cell Nucleus:(processing) evaluation of activation cell and nucleus Axon (Neurit) Dendrits Synapsis

8 Neural Networks - Basics Natural vs. Artificial Neuron cell and nucleus Axon (Neurit) Dendrits Synapsis w 1i w 2i w ji... oioi

9 Neural Networks - Basics Abstraction Dendrits:weighted (real number) connections Axon:output: real number Synapse:--- (identity: output is directly forwarded) Cell nucleus: unit contains simple functions input = (many) real numbers processing = activation function output = real number (~activation)

10 Neural Networks - Basics An artificial Neuron net: input from the network w: weight of a connection act: activation f act : activation function  : bias/threshold f out : output function (mostly ID) o: output w 1i w 2i w ji... oioi

11 Neural Networks - Basics Exercise: AND/OR -LTU Built a „network“ that works like an AND-function Built an OR-network Try to built an XOR-network LTU – Linear Threshold Unit

12 Neural Networks - Basics A simple switch Neuron = AND-function Find parameters: –Input neurons 1,2 : a 1,a 2 input pattern, –weights of edges: w 1, w 2 –bias  Now evaluate output o ! a 1 =__a 2 =__ o=__ net = o 1  w 1 +o 2  w 2 a= 1, if net >  = 0, otherwise o = a w 1 =__ w 2 =__

13 Neural Networks - Basics Mathematics in a Cell Propagation function (neuron input) net i (t) =  o j  w j = w 1i  o 1 + w 2i  o Activation a i (t) – Activation at time t Activation function f act : a i (t+1) = f act (a i (t), net i (t),  i )  i – bias Output function f out : o i = f out (a i )

14 Neural Networks - Basics Activation

15 Neural Networks - Basics Bias function -1,0 -0,5 0,0 0,5 1,0 -4,0-2,00,02,04,0 Identity -4,0 -2,0 0,0 2,0 4,0 -4,0-2,5-1,00,52,03,5 Activation Functions activation functions are sigmoid functions

16 Neural Networks - Basics y = tanh(c·x) -1,0 -0,5 0,5 1,0 -0,60,61,0 c=1 c=2 c=3 -1,0 Activation Functions y = 1/(1+exp(-c·x)) 0,5 1,0 -1,0 0,0 1,0 c=1 c=3 c=10 Logistic function: activation functions are sigmoid functions

17 Neural Networks - Basics Structure of a network layers –input layer– input neurons –output layer – output neurons –hidden layer – hidden neurons An n-layer network has: –n layer of connections which can be trained 

18 Neural Networks - Basics Definition: A Neural Network … … is characterized by – connections of many (a lot of) simple units (neurons) and – units exchanging signals via these connections … is a – coherent, directed graph which has – weighted edges and – each node (neuron, unit ) contains a value (activation).

19 Neural Networks - Basics XOR-network weights are set by hand f act = 1, net >  = 0, otherwise  = 1.0 input output

20 Neural Networks - Basics XOR-Example standard propagation function: net i (t) =  o j (t)  w ji activation function = bias function – a i = 1, if net i (t)>  i e.g.  =0.5 0, otherwise –output function = Identity: o j = a j use EXCEL and built the XOR-example network!

21 Neural Networks - Basics Learning Network changing network parameters evaluation network error learning examples

22 Neural Networks - Basics Learning - can be done by: 1.Modification of the weight of a connection –most frequently used 2.Deleting connections –can be done by (1): w=0  w  0 3.Modification of the bias of a neuron –Can be done by (1) using an extra neuron 4.changing functions (activation, propagation, output function) 5.Building new cells (GNG) 6.Building new connections 7.Deleting cells

23 Neural Networks - Basics Learning supervised learning –We know the results for certain input pattern: teaching input  teaching output –Network error is used to adapt weights –Fast, but not natural reinforced learning –We know whether output is right or wrong –Information is used to adapt weights –Slower than supervised; natural unsupervised learning –Network has to learn by itself; –Slow, natural

24 Neural Networks - Basics Applications  Pattern recognition (text, numbers, faces)  Checking the quality of a surface  Control of autonomous vehicles  Monitoring of credit card accounts  Data Mining

25 Neural Networks - Basics Applications  Speech recognition  Control of artificial limbs  classification of galaxies  Product orders (Supermarket)  Forecast of energy consumption  Stock value forecast

26 Neural Networks - Basics Application - Summary  Classification  Clustering  Forecast  Pattern recognition  Learning by examples, generalization  Recognition of not known structures