Neural Networks 2nd Edition Simon Haykin 柯博昌 Chap 1. Introduction

What is a Neural Network A neural network is a massively parallel distributed processor made up of simple processing units, which has a natural propensity for storing experiential knowledge and making it available for use. Knowledge is acquired by the network from its environment through a learning process. The procedure performing learning process is called a learning algorithm. Interneuron connection strengths, known as synaptic weights, are used to store the acquired knowledge.

Benefits of Neural Networks The computing power of neural networks Massively parallel distributed structure Ability to learn and therefore generalize. Using neural networks offers following properties: Nonlinearity Input-Output Mapping Adaptively Evidential Response Contextual Information Fault Tolerance VLSI Implementability Uniformity of Analysis and Design Neurobiological Analogy Supervised Learning: Modifying the synaptic weights by applying a set of training samples, which constitute of input signals and corresponding desired responses.

Human Brain - Function Block Block diagram representation of human nervous system Forward Receptors Neural Net. Effectors Stimulus Response (Brain) Feedback Receptors: Convert stimulus from the human body or the external environment into electrical impulses that convey information to brain. Effectors: Convert electrical impulses generated by brain into discernible responses as system outputs.

Comparisons: Neural Net. Vs. Brain Neuron: The structural constituents of the brain. Neurons are five to six orders of magnitude slower than silicon logic gates. (e.g. Silicon chips: 10-9 s, Neural Event: 10-3 s) 10 billion neurons and 60 trillion synapses or connections are in the human cortex. The energetic efficiency Brain: 10-16 joules per operation per second. Best Computer today: 10-6 joules per operation per second.

Synapses Synapses are elementary structural and functional units that mediate the interactions between neurons. The most common kind of synapse is chemical synapse. The operations of synapse: A pre-synaptic process liberates a transmitter substance that diffuses across the synaptic junction between neurons. Acts on a post-synaptic process. Synapse converts a pre-synaptic electrical signal into a chemical signal and then back into a post-synaptic electrical signal. (Nonreciprocal two-port device)

Pyramidal Cell

Cytoarchitectural map of the cerebral cortex

Nonlinear model of a neuron Let bk=wk0 and x0=+1 and

Nonlinear model of a neuron (Cont.) Affine transformation produced by the presence of a bias Another Nonlinear model of a neuron

Types of Activation Function Threshold Function Piecewise-Linear Function Sigmoid Function

Types of Activation Function (Cont.) The activation functions defined above range from 0 to +1. Sometimes, the activation function ranges from -1 to +1. (How to do?) Assume the activation function ranging from 0 to +1 is denoted as (), ranging from -1 to +1 is denoted as ’()  ’()=()*2-1 Notes: if (v)=sigmoid function

Stochastic Model of a Neuron The above model is deterministic in that its input-output behavior is precisely defined. Some applications of neural network base the analysis on a stochastic neuronal model. Let x denote the state of the neuron, and P(v) denote the probability of firing, where v is the induced local field of the neuron. A standard choice for P(v) is the sigmoid-shaped function. T is a pseudo-temperature that is used to control the noise level and therefore the uncertainty in firing.

Neural Network  Directed Graph xj yk=wkjxj wkj Synaptic Links xj yk=(xj) () Activation Links yk=yi+yj yi yj Synaptic Convergence (fan-in) xj Synaptic Divergence (fan-out)

Signal-flow Graph of a Neuron

yk(n)=A[xj’(n)] xj’(n)=xj(n)+B[yk(n)] where A and B act as operators Feedback Feedback plays a major role in recurrent network. xj(n) yk=(xj) xj’(n) B A yk(n)=A[xj’(n)] xj’(n)=xj(n)+B[yk(n)] where A and B act as operators A/(1-AB) is referred as closed-loop operator, AB as open-loop operator. In general, ABBA

Let A be a fixed weight, w; and B is a unit-delay operator, z-1 Feedback (Cont.) Let A be a fixed weight, w; and B is a unit-delay operator, z-1 Use Taylor’s Expansion or Binomial Expansion to prove it.

Time Responses for different weight, w Conclusions: |w|<1, yk(n) is exponentially convergent. System is stable. |w|1, yk(n) is divergent. System is unstable. Think about: What does the time response change, If -1<w<0? What does the time response change, If w-1?

Network Architectures MultiLayer Feedforward Networks Single-Layer Feedforward Networks Fully Connected: Every node in each layer is connected to every other node in the adjacent forward layer. Otherwise, it’s Partially Connected.

Network Architectures (Cont.) Recurrent Networks with no self-feedback loops and no hidden neurons Recurrent Networks with hidden neurons

Knowledge Representation Primary characteristics of knowledge representation What information is actually made explicit How the information is physically encoded for subsequent use Knowledge is goal directed. A good solution depends on a good representation of knowledge. A set of input-output pairs, with each pair consisting of an input signal and the corresponding desired response, is referred to as a set of training data or training sample.

Rules for Knowledge Representation Rule 1: Similar inputs from similar classes should usually produce similar representations inside the network. Similarity Measuring: (1) Using Euclidian distance, d(xi, xj) Let xi=[xi1, xi2, …, xim]T (2) Using Inner Product, (xi, xj) Let xi=[xi1, xi2, …, xim]T xi ||xi-xj||  xj xiTxj If ||xi||=1 and ||xj||=1 d2(xi, xj)=(xi-xj)T(xi-xj)=2-2(xiTxj)

Rules for Knowledge Representation (Cont.) Rule 2: Items to be categorized as separate classes should be given widely different representations in the network. (This is the exact opposite of Rule 1.) Rule 3: If a particular feature is important, then there should be a large number of neurons involved in the representation of that item. Rule 4: Prior information and invariance should be built into the design of a neural network, thereby simplifying the network design by not having to learn them.

How to Build Prior Information into Neural Network Design Restricting the network architecture though the use of local connections knows as receptive fields. Constraining the choice of synaptic weights through the use of weight-sharing. Ex: Convolution Sum Convolution Network x1, …, x6 constitute the receptive field for hidden neuron 1 and so on for the other hidden neurons.

Artificial Intelligence (AI) Goal: Developing paradigms or algorithms that require machines to perform cognitive tasks. AI system must be capable of doing: Store knowledge Apply knowledge stored to solve problems Acquire new knowledge through experience Key components Representation Reasoning Learning Reasoning Representation Learning