1. Neural Networks 2nd Edition Simon Haykin
柯博昌
Chap 1. Introduction

2. What is a Neural Network
A neural network is a massively parallel distributed processor made up of simple processing units, which has a natural propensity for storing experiential knowledge and making it available for use.
Knowledge is acquired by the network from its environment through a learning process. The procedure performing learning process is called a learning algorithm.
Interneuron connection strengths, known as synaptic weights, are used to store the acquired knowledge.

3. Benefits of Neural Networks
The computing power of neural networks
Massively parallel distributed structure
Ability to learn and therefore generalize.
Using neural networks offers following properties:
Nonlinearity
Input-Output Mapping
Adaptively
Evidential Response
Contextual Information
Fault Tolerance
VLSI Implementability
Uniformity of Analysis and Design
Neurobiological Analogy
Supervised Learning: Modifying the synaptic weights by applying a set of training samples, which constitute of input signals and corresponding desired responses.

4. Human Brain - Function Block
Block diagram representation of human nervous system
Forward
Receptors
Neural
Net.
Effectors
Stimulus
Response
(Brain)
Feedback
Receptors: Convert stimulus from the human body or the external environment into electrical impulses that convey information to brain.
Effectors: Convert electrical impulses generated by brain into discernible responses as system outputs.

5. Comparisons: Neural Net. Vs. Brain
Neuron: The structural constituents of the brain.
Neurons are five to six orders of magnitude slower than silicon logic gates. (e.g. Silicon chips: 10-9 s, Neural Event: 10-3 s)
10 billion neurons and 60 trillion synapses or connections are in the human cortex.
The energetic efficiency
Brain: joules per operation per second.
Best Computer today: 10-6 joules per operation per second.

6. Synapses
Synapses are elementary structural and functional units that mediate the interactions between neurons.
The most common kind of synapse is chemical synapse.
The operations of synapse:
A pre-synaptic process liberates a transmitter substance that diffuses across the synaptic junction between neurons.
Acts on a post-synaptic process.
Synapse converts a pre-synaptic electrical signal into a chemical signal and then back into a post-synaptic electrical signal. (Nonreciprocal two-port device)

7. Pyramidal Cell

8. Cytoarchitectural map of the cerebral cortex

9. Nonlinear model of a neuron
Let bk=wk0 and x0=+1
and

10. Nonlinear model of a neuron (Cont.)
Affine transformation produced by the presence of a bias
Another Nonlinear model of a neuron

11. Types of Activation Function
Threshold Function
Piecewise-Linear Function
Sigmoid Function

12. Types of Activation Function (Cont.)
The activation functions defined above range from 0 to +1.
Sometimes, the activation function ranges from -1 to +1. (How to do?)
Assume the activation function ranging from 0 to +1 is denoted as (), ranging from -1 to +1 is denoted as ’()
 ’()=()*2-1
Notes: if (v)=sigmoid function

13. Stochastic Model of a Neuron
The above model is deterministic in that its input-output behavior is precisely defined.
Some applications of neural network base the analysis on a stochastic neuronal model.
Let x denote the state of the neuron, and P(v) denote the probability of firing, where v is the induced local field of the neuron.
A standard choice for P(v) is the sigmoid-shaped function. T is a pseudo-temperature that is used to control the noise level and therefore the uncertainty in firing.

14. Neural Network  Directed Graphxj
yk=wkjxj
wkj
Synaptic Links xj
yk=(xj)
()
Activation Links
yk=yi+yj yi yj
Synaptic Convergence
(fan-in) xj
Synaptic Divergence
(fan-out)

15. Signal-flow Graph of a Neuron

16. yk(n)=A[xj’(n)] xj’(n)=xj(n)+B[yk(n)] where A and B act as operators
Feedback
Feedback plays a major role in recurrent network.
xj(n)
yk=(xj)
xj’(n) B A
yk(n)=A[xj’(n)] xj’(n)=xj(n)+B[yk(n)] where A and B act as operators
A/(1-AB) is referred as closed-loop operator, AB as open-loop operator.
In general, ABBA

17. Let A be a fixed weight, w; and B is a unit-delay operator, z-1
Feedback (Cont.)
Let A be a fixed weight, w; and B is a unit-delay operator, z-1
Use Taylor’s Expansion or Binomial Expansion to prove it.

18. Time Responses for different weight, w
Conclusions:
|w|<1, yk(n) is exponentially convergent. System is stable.
|w|1, yk(n) is divergent. System is unstable.
Think about:
What does the time response change, If -1<w<0?
What does the time response change, If w-1?

19. Network Architectures
MultiLayer Feedforward Networks
Single-Layer Feedforward Networks
Fully Connected: Every node in each layer is connected to every other node in the adjacent forward layer. Otherwise, it’s Partially Connected.

20. Network Architectures (Cont.)
Recurrent Networks with no self-feedback loops and no hidden neurons
Recurrent Networks with hidden neurons

21. Knowledge Representation
Primary characteristics of knowledge representation
What information is actually made explicit
How the information is physically encoded for subsequent use
Knowledge is goal directed.
A good solution depends on a good representation of knowledge.
A set of input-output pairs, with each pair consisting of an input signal and the corresponding desired response, is referred to as a set of training data or training sample.

22. Rules for Knowledge Representation
Rule 1: Similar inputs from similar classes should usually produce similar representations inside the network.
Similarity Measuring:
(1) Using Euclidian distance, d(xi, xj)
Let xi=[xi1, xi2, …, xim]T
(2) Using Inner Product, (xi, xj)
Let xi=[xi1, xi2, …, xim]T xi
||xi-xj||  xj
xiTxj
If ||xi||=1 and ||xj||=1 d2(xi, xj)=(xi-xj)T(xi-xj)=2-2(xiTxj)

23. Rules for Knowledge Representation (Cont.)
Rule 2: Items to be categorized as separate classes should be given widely different representations in the network.
(This is the exact opposite of Rule 1.)
Rule 3: If a particular feature is important, then there should be a large number of neurons involved in the representation of that item.
Rule 4: Prior information and invariance should be built into the design of a neural network, thereby simplifying the network design by not having to learn them.

24. How to Build Prior Information into Neural Network Design
Restricting the network architecture though the use of local connections knows as receptive fields.
Constraining the choice of synaptic weights through the use of weight-sharing.
Ex:
Convolution Sum
Convolution Network
x1, …, x6 constitute the receptive field for hidden neuron 1 and so on for the other hidden neurons.

25. Artificial Intelligence (AI)
Goal: Developing paradigms or algorithms that require machines to perform cognitive tasks.
AI system must be capable of doing:
Store knowledge
Apply knowledge stored to solve problems
Acquire new knowledge through experience
Key components
Representation
Reasoning
Learning
Reasoning
Representation
Learning