Introduction to Artificial Neural Networks 主講人: 虞台文

Content Fundamental Concepts of ANNs. Basic Models and Learning Rules Neuron Models ANN structures Learning Distributed Representations Conclusions

Introduction to Artificial Neural Networks Fundamental Concepts of ANNs

What is ANN? Why ANN? ANN  Artificial Neural Networks To simulate human brain behavior A new generation of information processing system.

Applications Pattern Matching Pattern Recognition Associate Memory (Content Addressable Memory) Function Approximation Learning Optimization Vector Quantization Data Clustering . . .

Traditional Computers are inefficient at these tasks although their computation speed is faster. Applications Pattern Matching Pattern Recognition Associate Memory (Content Addressable Memory) Function Approximation Learning Optimization Vector Quantization Data Clustering . . .

The Configuration of ANNs An ANN consists of a large number of interconnected processing elements called neurons. A human brain consists of ~1011 neurons of many different types. How ANN works? Collective behavior.

The Biologic Neuron

The Biologic Neuron 二神經原之神經絲接合部分 軸突 樹狀突

The Biologic Neuron Excitatory or Inhibitory

The Artificial Neuron x1 x2 xm wi1 wi2 wim f (.) a (.) i  yi

The Artificial Neuron x1 x2 xm wi1 wi2 wim f (.) a (.) i  yi

 The Artificial Neuron wij positive  excitatory negative  inhibitory zero  no connection The Artificial Neuron x1 x2 xm wi1 wi2 wim f (.) a (.) i  yi

 The Artificial Neuron Proposed by McCulloch and Pitts [1943] M-P neurons x1 x2 xm wi1 wi2 wim f (.) a (.) i  yi

What can be done by M-P neurons? A hard limiter. A binary threshold unit. Hyperspace separation. x1 x2   y w1 w2 x1 x2 w1 x1 + w2 x2 = 1

What ANNs will be? ANN  A neurally inspired mathematical model. Consists a large number of highly interconnected PEs. Its connections (weights) holds knowledge. The response of PE depends only on local information. Its collective behavior demonstrates the computation power. With learning, recalling and, generalization capability.

Three Basic Entities of ANN Models Models of Neurons or PEs. Models of synaptic interconnections and structures. Training or learning rules.

Introduction to Artificial Neural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

 Processing Elements f (.) a (.) Extensions of M-P neurons What integration functions we may have? What activation functions we may have?

Integration Functions M-P neuron f (.) a (.) i  Quadratic Function Spherical Function Polynomial Function

 Activation Functions f (.) a (.) M-P neuron: (Step function) a f i 1 a f

 Activation Functions f (.) a (.) Hard Limiter (Threshold function) a 1 a 1 f

Activation Functions Ramp function: f (.) a (.) i  1 a f

Activation Functions Unipolar sigmoid function: f (.) a (.) i 

Activation Functions Bipolar sigmoid function: f (.) a (.) i 

Example: Activation Surfaces y L1 L3 x y L1 L2 L3 L2

Example: Activation Surfaces y L1 L3 x1=0 1 1=1 1 3= 4 xy+4=0 1 2=1 x y L1 L3 L2 L2 y1=0

Example: Activation Surfaces 010 x y L1 L2 L3 Region Code 011 x y L1 L3 L2 110 111 001 101 100

Example: Activation Surfaces z x y L1 L2 L3 z=0 x y L1 L3 L2 z=1

Example: Activation Surfaces z x y L1 L2 L3 z=0 1 4=2.5 x y L1 L2 L3 z=1

Example: Activation Surfaces M-P neuron: (Step function) L4 z x y L1 L2 L3

Example: Activation Surfaces Unipolar sigmoid function: Example: Activation Surfaces =2 =3 L4 z x y L1 L3 L2 =5 =10

Introduction to Artificial Neural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

ANN Structure (Connections)

Single-Layer Feedforward Networks yn x1 x2 xm w11 w12 w1m w21 w22 w2m wn1 wnm wn2 . . .

Multilayer Feedforward Networks . . . x1 x2 xm y1 y2 yn Output Layer Hidden Layer Input Layer

Multilayer Feedforward Networks Pattern Recognition Multilayer Feedforward Networks Where the knowledge from? Classification Output Analysis Learning Input

Single Node with Feedback to Itself Loop

Single-Layer Recurrent Networks . . . x1 x2 xm y1 y2 yn

Multilayer Recurrent Networks x1 x2 x3 y1 y2 y3 . . .

Introduction to Artificial Neural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

Learning Consider an ANN with n neurons and each with m adaptive weights. Weight matrix:

How? Learning To “Learn” the weight matrix. Consider an ANN with n neurons and each with m adaptive weights. Weight matrix: To “Learn” the weight matrix.

Learning Rules Supervised learning Reinforcement learning Unsupervised learning

Supervised Learning Learning with a teacher Learning by examples  Training set

Supervised Learning y x ANN W d Error signal Generator

Reinforcement Learning Learning with a critic Learning by comments

Reinforcement Learning y x ANN W Reinforcement Signal Critic signal Generator

Unsupervised Learning Self-organizing Clustering Form proper clusters by discovering the similarities and dissimilarities among objects.

Unsupervised Learning y x ANN W

The General Weight Learning Rule . wi1 wi2 wij wi,m-1 x1 x2 xj xm-1 yi i Input: Output:

The General Weight Learning Rule We want to learn the weights & bias. The General Weight Learning Rule i . wi1 wi2 wij wi,m-1 x1 x2 xj xm-1 yi i Input: Output:

The General Weight Learning Rule We want to learn the weights & bias. The General Weight Learning Rule x1 wi1 Input: x2 wi2 . wij i xj Let xm = 1 and wim = i. . xm-1 wi,m-1 i

The General Weight Learning Rule x1 wi1 Input: x2 wi2 . wij i xj Let xm = 1 and wim = i. . xm-1 wi,m-1 wim=i xm= 1

The General Weight Learning Rule We want to learn wi=(wi1, wi2 ,…,wim)T The General Weight Learning Rule x1 wi1 Input: x2 wi2 . wij i xj yi . xm-1 wi,m-1 wi(t) = ? wim=i xm= 1

The General Weight Learning Rule wi x yi r di Learning Signal Generator

The General Weight Learning Rule wi x yi r di Learning Signal Generator

The General Weight Learning Rule wi x yi r di Learning Signal Generator

The General Weight Learning Rule wi x yi r di Learning Signal Generator  Learning Rate

The General Weight Learning Rule We want to learn wi=(wi1, wi2 ,…,wim)T The General Weight Learning Rule Discrete-Time Weight Modification Rule: Continuous-Time Weight Modification Rule:

Hebb’s Learning Law Hebb [1994] hypothesis that when an axonal input from A to B causes neuron B to immediately emit a pulse (fire) and this situation happens repeatedly or persistently. Then, the efficacy of that axonal input, in terms of ability to help neuron B to fire in future, is somehow increased. Hebb’s learning rule is a unsupervised learning rule.

Hebb’s Learning Law + +  

Introduction to Artificial Neural Networks Distributed Representations

Distributed Representations An entity is represented by a pattern of activity distributed over many PEs. Each Processing element is involved in representing many different entities. Local Representation: Each entity is represented by one PE.

Example + _ Dog Cat Bread + _ + _ P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10

Advantages What is this? + _ Dog Cat Bread + _ + _ + P0 P1 P2 P3 P4 P5 Act as a content addressable memory. Advantages P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 + _ Dog Cat Bread + _ + _ P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 + What is this?

Advantages Dog has 4 legs? How many for Fido? + _ Dog Cat Bread + _ + Act as a content addressable memory. Make induction easy. Advantages P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 + _ Dog Cat Bread + _ + _ P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 + _ Fido Dog has 4 legs? How many for Fido?

Advantages Add doughnut by changing weights. + _ Dog Cat Bread + _ + _ Act as a content addressable memory. Make induction easy. Make the creation of new entities or concept easy (without allocation of new hardware). Advantages P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 + _ Dog Cat Bread + _ + _ + _ Doughnut Add doughnut by changing weights.

Advantages Some PEs break down don’t cause problem. + _ Dog Cat Bread Act as a content addressable memory. Make induction easy. Make the creation of new entities or concept easy (without allocation of new hardware). Fault Tolerance. Advantages P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 + _ Dog Cat Bread + _ + _ Some PEs break down don’t cause problem.

Disadvantages Learning procedures are required. How to understand? How to modify? Learning procedures are required.