Presentation is loading. Please wait.

Presentation is loading. Please wait.

Artificial Neural Networks

Published byJúlio Cordeiro Carmona Modified over 6 years ago

Similar presentations

Presentation on theme: "Artificial Neural Networks"— Presentation transcript:

1 Artificial Neural Networks
Unsupervised ANNs

2 Inteligência Artificial
Contents Unsupervised ANNs Kohonen Self-Organising Map (SOM) Structure Processing units Learning Applications Further Topics: Spiking ANNs Application Adaptive Resonance Theory (ART) Further Topics: ARTMAP 13/01/2019 Inteligência Artificial

3 Inteligência Artificial
Unsupervised ANNs Usually 2-layer ANN Only input data are given ANN must self-organise output Two main models: Kohonen’s SOM and Grossberg’s ART Clustering applications Output layer Feature layer 13/01/2019 Inteligência Artificial

4 Inteligência Artificial
Learning Rules Instar Learning rule: incoming weights of neuron converge to input pattern (previous layer) Convergence speed is determined by learning rate Step size proportional to node output value Neuron learns association between input vectors and their outputs Outstar Learning rule: outgoing weights of neuron converge to output pattern (next layer) Learning is proportional to neuron activation Step size proportional to node input value Neuron learns to recall pattern when stimulated 13/01/2019 Inteligência Artificial

5 Self-Organising Map (SOM)
T. Kohonen (1984) 2D map of output neurons Input layer and output layer fully connected Lateral inhibitory synapses Model of biological topographic maps, e.g. primary auditory cortex in animal brains (cats and monkeys) Hebbian learning Akin to K-means Data clustering applications Output layer Feature layer 13/01/2019 Inteligência Artificial

6 Inteligência Artificial
SOM Clustering Neuron = prototype for a cluster Weights = reference vector (protoype features) Euclidean distance between reference vector and input pattern Competitive layer (winner take all) In biological systems winner take all via inhibitory synapses Neuron with reference vector closest to input wins yi x1 x2 x5 x3 x4 wi1 wi3 wi2 wi4 wi5 Neuron i 13/01/2019 Inteligência Artificial

7 SOM Learning Algorithm
Only weights of winning neuron and its neighbours are updated Weights of winning neuron brought closer to input pattern (instar rule) Reference vector is usually normalised Neighbourhood function in biological systems via short range excitatory synapses Decreasing width of neighbourhood ensures increasingly finer differences are encoded Global convergence is not guaranteed. Gradual lowering of learning rate ensures stability (otherwise vectors may oscillate between clusters) At end neurons are “tagged”, similar ones become sub-clusters of larger cluster N(t) = Neighbourhood function E(t0) E(t1) E(t2) E(t3) 13/01/2019 Inteligência Artificial

8 Inteligência Artificial
SOM Mapping Adaptive Vector Quantisation Reference vectors iteratively moved towards centres of (sub)clusters Best performing on gaussian distributions (distance is radial) 13/01/2019 Inteligência Artificial

9 Inteligência Artificial
SOM Topology Surface of map reflects frequency distribution of input set, i.e. the probability of input class occurring. More common vector ``types’’ occupy proportionally more of output map. The more frequent the pattern type, the finer grained the mapping. Biological correspondence in brain cortex Map allows dimension reduction and visualisation of input data 13/01/2019 Inteligência Artificial

10 Inteligência Artificial
Some Issues about SOM SOM can be used on-line (adaptation) Neurons need to be labelled Manually Automatic algorithm Sometimes may not converge Precision not optimal Some neurons may be difficult to label Results sensitive to choice of input features Results sensitive to order of presentation of data Epoch learning 13/01/2019 Inteligência Artificial

11 Inteligência Artificial
SOM Applications Natural language processing Document clustering Document retrieval Automatic query Image segmentation Data mining Fuzzy partitioning Condition-action association 13/01/2019 Inteligência Artificial

12 Further Topics – Spiking ANNs
Image segmentation task SOM of spiking units Lateral connections Short range excitatory Long range inhibitory Train using Hebbian Learning Train showing one pattern at a time 13/01/2019 Inteligência Artificial

13 Inteligência Artificial
Spiking SOM Training Hebbian Learning Different learning coefficients afferent weights la lateral inhibitory weights li lateral excitatory weights le Initially learn long-term correlations for self-organisation Then learn activity modulation for segmentation N = normalisation factor la li, le t 13/01/2019 Inteligência Artificial

14 Spiking Neuron Dynamics
y(t) urest+(t-tf) 13/01/2019 Inteligência Artificial

15 Inteligência Artificial
Spiking SOM Recall Show different shapes together Bursts of neuron activity Each cluster alternatively fires 13/01/2019 Inteligência Artificial

16 Adaptive Resonance Theory (ART)
Carpenter and Grossberg (1976) Inspired by studies on biological feature detectors On-line clustering algorithm Leader-follower algorithm Recurrent ANN Competitive output layer Data clustering applications Stability-plasticity dilemma Output layer Feature layer 13/01/2019 Inteligência Artificial

17 Inteligência Artificial
ART Types ART1 binary patterns ART2 binary or analog patterns ART3 hierarchical ART structure ARTMAP supervised ART 13/01/2019 Inteligência Artificial

18 Stability-Plasticity Dilemma
Plasticity: System adapts its behaviour according to significant events Stability: system behaviour doesn’t change after irrelevant events Dilemma: how to achieve stability without rigidity and plasticity without chaos? Ongoing learning capability Preservation of learned knowledge 13/01/2019 Inteligência Artificial

19 Inteligência Artificial
ART Architecture Bottom-up weights wij Normalised copy of vij Top-down weights vij Store class template Input nodes Vigilance test Input normalisation Output nodes Forward matching Long-term memory ANN weights Short-term memory ANN activation pattern top down bottom up (normalised) 13/01/2019 Inteligência Artificial

20 Inteligência Artificial
ART Algorithm Adapt winner node Initialise uncommitted node new pattern categorisation known unknown recognition comparison Incoming pattern matched with stored cluster templates If close enough to stored template joins best matching cluster, weights adapted according to outstar rule If not, a new cluster is initialised with pattern as template 13/01/2019 Inteligência Artificial

21 Inteligência Artificial
Recognition Phase Forward transmission via bottom-up weights Input pattern matched with bottom-up weights (normalised template) of output nodes Inner product x•wi Hypothesis formulation: best matching node fires (winner-take-all layer) Similar to Kohonen’s SOM algorithm, pattern associated to closest matching template ART1: fraction of bits of template also in input pattern Innner product x=input pattern wi=bottom-up weight of neuron I N=input features x q wi 13/01/2019 Inteligência Artificial

22 Inteligência Artificial
Comparison Phase Backward transmission via top-down weights Vigilance test: class template matched with input pattern Hypothesis validation: if pattern close enough to template, categorisation was successful and “resonance” achieved If not close enough reset winner neuron and try next best matching Repeat until Either vigilance test passed Or hypotheses (committed neurons) exhausted ART1: fraction of bits of input pattern also in template x=input pattern vi=top-down weight of neuron I r=vigilance threshold 13/01/2019 Inteligência Artificial

23 Inteligência Artificial
Vigilance Threshold Small r, imprecise Vigilance threshold sets granularity of clustering It defines basin of attraction of each prototype Low threshold Large mismatch accepted Few large clusters Misclassifications more likely High threshold Small mismatch accepted Many small clusters Higher precision Large r, fragmented 13/01/2019 Inteligência Artificial

24 Inteligência Artificial
Adaptation Only weights of winner node are updated ART1: only features common to all members of cluster are kept ART1: prototype is intersection set of members ART2: prototype brought closer to last example ART2: b determines amount of modification ART1 ART2 13/01/2019 Inteligência Artificial

25 Inteligência Artificial
Additional Modules Categorisation result Output layer Gain control Reset module Input layer Input pattern 13/01/2019 Inteligência Artificial

26 Inteligência Artificial
Reset Module Fixed connection weights Implements the vigilance test Excitatory connection from input lines Inhibitory connection from input layer Output of reset module inhibitory to output layer Disables firing output node if match with pattern is not close enough Duration of reset signal lasts until pattern is present New pattern p is presented Reset module receives excitatory signal E from input lines All active nodes are reset Input layer is activated Reset module receives inhibitory signal I from input layer I>E If p•v<r inhibition weakens and reset signal is sent 13/01/2019 Inteligência Artificial

27 Inteligência Artificial
Gain module Fixed connection weights Controls activation cycle of input layer Excitatory connection from input lines Inhibitory connection from output layer Output of gain module excitatory to input layer Shuts down system if noise produces oscillations 2/3 rule for input layer New pattern p is presented Gain module receives excitation signal E from input lines Input layer allowed to fire Input layer is activated Output layer is activated Gain module turned down Now is feedback from output layer that keeps input layer active If p•v<r output layer switched off and gain allows input to keep firing for another match 13/01/2019 Inteligência Artificial

28 2/3 Rule 2 inputs out of 3 are needed for input layer to be active
Input signal Gain module Output layer Input layer 1 2 3 4 5 6 7 New pattern p is presented Input layer is activated Output layer is activated Reset signal is sent New match Resonance Input off 13/01/2019 Inteligência Artificial

29 Inteligência Artificial
Issues about ART Learned knowledge can be retrieved Fast learning algorithm Difficult to tune vigilance threshold Noise tends to lead to category proliferation New noisy patterns tend to “erode” templates ART is sensitive to order of presentation of data Accuracy sometimes not optimal Assumes samples distribution to be Gaussian (see SOM) Only winner neuron is updated, more “point-to-point” mapping than SOM 13/01/2019 Inteligência Artificial

30 SOM Plasticity vs. ART Plasticity
SOM mapping ART mapping new pattern new pattern Given new pattern, SOM moves previously committed node and rearrange its neighbours, prior learning is partly “forgotten” 13/01/2019 Inteligência Artificial

31 Inteligência Artificial
ART Applications Natural language processing Document clustering Document retrieval Automatic query Image segmentation Character recognition Data mining Data set partitioning Detection of emerging clusters Fuzzy partitioning Condition-action association 13/01/2019 Inteligência Artificial

32 Further Topics - ARTMAP
Desired output Composed of 2 ART ANNs and a mapping field Online, supervised, self-organising ANN Mapping field: connects output nodes of ART 1 to output nodes of ART 2 Mapping field trained using hebbian learning ART 1 partitions input space ART 2 partitions output space Mapping field learns stimulus-response associations Input layer ART 2 Output layer Mapping field Output layer ART 1 Input layer Input pattern 13/01/2019 Inteligência Artificial

33 Inteligência Artificial
Conclusions - ANNs ANNs can learn where knowledge is not available ANNs can generalise from learned knowledge There are several different ANN models with different capabilities ANNs are robust, flexible and accurate systems Parallel distributed processing allows fast computations and fault tolerance ANNs require a set of parameters to be defined Architecture Learning rate Training is crucial to ANN performance Learned knowledge often not available (black box) 13/01/2019 Inteligência Artificial

34 Inteligência Artificial
Further Readings Mitchell, T. (1997), Machine Learning, McGraw Hill. Duda, R. O., Hart, P. E., and Stork, D. G. (2000), Pattern Classification, New York: Wiley. 2nd Edition. ANN Glossary 13/01/2019 Inteligência Artificial

Download ppt "Artificial Neural Networks"

Similar presentations

© Negnevitsky, Pearson Education, 2005 1 Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.

© Negnevitsky, Pearson Education, Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.
2806 Neural Computation Self-Organizing Maps Lecture 9 2005 Ari Visa.

2806 Neural Computation Self-Organizing Maps Lecture Ari Visa.
Neural Networks Dr. Peter Phillips. The Human Brain (Recap of week 1)

Neural Networks Dr. Peter Phillips. The Human Brain (Recap of week 1)
Marković Miljan 3139/2011

Marković Miljan 3139/2011
Instar and Outstar Learning Laws Adapted from lecture notes of the course CN510: Cognitive and Neural Modeling offered in the Department of Cognitive and.

Instar and Outstar Learning Laws Adapted from lecture notes of the course CN510: Cognitive and Neural Modeling offered in the Department of Cognitive and.
Data mining in wireless sensor networks based on artificial neural-networks algorithms Authors: Andrea Kulakov and Danco Davcev Presentation by: Niyati.

Data mining in wireless sensor networks based on artificial neural-networks algorithms Authors: Andrea Kulakov and Danco Davcev Presentation by: Niyati.
Neural Networks Chapter 9 Joost N. Kok Universiteit Leiden.

Neural Networks Chapter 9 Joost N. Kok Universiteit Leiden.
Unsupervised learning. Summary from last week We explained what local minima are, and described ways of escaping them. We investigated how the backpropagation.

Unsupervised learning. Summary from last week We explained what local minima are, and described ways of escaping them. We investigated how the backpropagation.
Self Organization: Competitive Learning

Self Organization: Competitive Learning
5/16/2015Intelligent Systems and Soft Computing1 Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.

5/16/2015Intelligent Systems and Soft Computing1 Introduction Introduction Hebbian learning Hebbian learning Generalised Hebbian learning algorithm Generalised.
Kohonen Self Organising Maps Michael J. Watts

Kohonen Self Organising Maps Michael J. Watts
Artificial neural networks:

Artificial neural networks:
Adaptive Resonance Theory (ART) networks perform completely unsupervised learning. Their competitive learning algorithm is similar to the first (unsupervised)

Adaptive Resonance Theory (ART) networks perform completely unsupervised learning. Their competitive learning algorithm is similar to the first (unsupervised)
Competitive learning College voor cursus Connectionistische modellen M Meeter.

Competitive learning College voor cursus Connectionistische modellen M Meeter.
Unsupervised Networks Closely related to clustering Do not require target outputs for each input vector in the training data Inputs are connected to a.

Unsupervised Networks Closely related to clustering Do not require target outputs for each input vector in the training data Inputs are connected to a.
Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.

Machine Learning: Connectionist McCulloch-Pitts Neuron Perceptrons Multilayer Networks Support Vector Machines Feedback Networks Hopfield Networks.
X0 xn w0 wn o Threshold units SOM.

X0 xn w0 wn o Threshold units SOM.
Machine Learning Neural Networks

Machine Learning Neural Networks
Slides are based on Negnevitsky, Pearson Education, 2005 1 Lecture 8 Artificial neural networks: Unsupervised learning n Introduction n Hebbian learning.

Slides are based on Negnevitsky, Pearson Education, Lecture 8 Artificial neural networks: Unsupervised learning n Introduction n Hebbian learning.
1 Chapter 11 Neural Networks. 2 Chapter 11 Contents (1) l Biological Neurons l Artificial Neurons l Perceptrons l Multilayer Neural Networks l Backpropagation.

1 Chapter 11 Neural Networks. 2 Chapter 11 Contents (1) l Biological Neurons l Artificial Neurons l Perceptrons l Multilayer Neural Networks l Backpropagation.

Similar presentations

Ads by Google