2. Neural Network Architecture Session 2
Course : T0293 – NEURO COMPUTING
Year : 2013
Neural Network Architecture Session 2

3. Learning Outcomes At the end of this session, student will be able to:
Explain the knowledge representation of a neural network system and differentiate neural network architectures (LO1, LO2)
T Neuro Computing

4. Lecture Outline Single-Layer Feedforward Networks
Multilayer Feedforward Networks
Recurrent Networks
Knowledge Representation
T Neuro Computing

5. Network Architectures
Neurons (processing units) of a neural network are structured based on the learning algorithm used to train the system.
Learning algorithms (rules) in the design of neural networks is structured.
In general, there are three types of network architectures (structures):
Single-Layer Feedforward Networks
Multilayer Feedforward Networks
Recurrent Networks
T Neuro Computing

6. Single-Layer Feedforward Networks
Figure 2.1 Feedforwad network
with a single layer of neurons.
Information Flow
Single-layer feedforward network is the simplest form of a layered network.
There are two layers:
Input Layer
Output Layer (Computation Nodes)
It is feedforward, means the information flow from input to output and not vice versa.
Input layer of source nodes are not counted because no computation is performed.
Sources:
Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN:
T Neuro Computing

7. Multilayer Feedforward Networks
Figure 2.2 Fully connected feedforward network with one hidden layer and one output layer.
Information Flow
Multilayer feedforward networks has one or more hidden layers.
By adding hidden layers, the network is enabled to extract higher-order statistics from its input.
In this structure, the computation nodes are called hidden neurons or hidden units.
The example architecture in Figure 2.2 is referred to as a network:
10 source nodes
4 hidden neurons
2 output neurons
Fully Connected VS Partially Connected
Sources:
Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN:
T Neuro Computing

8. Recurrent Networks
Recurrent neural network is different from feedforward neural network because it has at least one feedback loop.
The presence of feedback loop has a profound impact on the learning capability of the network and its performance.
The feedback loops involve the use of particular branches composed of unit-time delay elements (denoted by z-1)
Structure at Figure 2.3:
No self-feedback loops in the network
No hidden neurons
Figure 2.3 Recurrent network with no self-feedback loops and no hidden neurons.
Sources:
Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN:
T Neuro Computing

9. Figure 2.4 Recurrent network with hidden neurons.
Structure at Figure 2.4:
Contains self-feedback loops in the network
Contains hidden neurons
The feedback connections originate from the hidden neurons as well as from the output neurons.
Figure 2.4 Recurrent network with hidden neurons.
Sources:
Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN:
T Neuro Computing

10. Knowledge Representation
“Knowledge refers to stored information or models used by a person or machine to interpret, predict, and appropriately respond to the outside world.” – Fischler and firschein
The primary characteristics of knowledge representation are twofold:
What information is actually made explicit
How the information is physically encoded for subsequent use
In short, knowledge representation is goal directed.
“Intelligence” machine, i.e., neural network systems, could be a good solution depends on a good representation of knowledge.
PROBLEMS: Knowledge representation form are highly diverse.
T Neuro Computing

11. Knowledge of the world consists of two kinds of information:
In order to achieve their goals, neural network systems have to represent real-world data (environment) and have to model sufficiently consistently with the real-world.
Knowledge of the world consists of two kinds of information:
The known world state, represent by facts about what is and what has been known (prior information).
Observations (measurements) of the world, obtained by means of sensors designed to probe the environment, in which the neural network is supposed to operate.
The examples (real-world data) could categories as Labeled and Unlabeled data.
A set of input-output pairs, with each pair consisting of an input signal and the corresponding desired response, is referred to as a set of training data or training sample.
The training sample is used to train a neural network system.
T Neuro Computing

12. Case Study (Problem) The handwritten-digit-recognition problem: Input
Image with black or white pixels. Each image representing one of 10 digits that are well separated from the background.
Desired Output
The “identity” of the particular digit whose image is presented to the network as the input signal.
The Training Samples
Large variety of handwritten digits that are representative of a real-world situation.
T Neuro Computing

13. Training Samples (or Data)
Case Study (Solution)
Design of a neural network:
Learning Phase
An appropriate architecture is selected for the neural network, where:
Input Layer
Consisting of source nodes equal in number to the pixels of an input image.
Output Layer
Consisting of 10 neurons (one for each digit).
Then, a subset of examples is used to train the network by means of a suitable algorithm.
Sources:
Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN:
Training Samples (or Data)
Neural Network System
is fed to
T Neuro Computing

14. Testing Phase
The recognition performance of the trained network is tested with data not seen before.
An input image is presented to the network, but the network is not told the identity of the digit which that particular image represents.
The performance of the neural network is asses by comparing the digit recognition reported by the network with the actual identity of the digit in question.
Successful performance on the test patterns is called generalization.
Sources:
Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN:
Unknown Data
Neural Network System
is tested by
recognized results
Results
T Neuro Computing

15. Summary
Single-layer feedforward network is the simplest form of a layered network.
Multilayer feedforward networks has one or more hidden layers.
Recurrent neural network is different from feedforward neural network because it has at least one feedback loop.
Knowledge representation is goal directed.
In order to achieve their goals, neural network systems have to represent real-world data (environment) and have to model sufficiently consistently with the real-world.
T Neuro Computing

16. PerceptronClassification Demo
A perceptron is computer code that models the behavior of a single biological neuron. Perceptrons were one of the very earliest types of machine-learning techniques and are the predecessors to neural networks.
T Neuro Computing

17. Classification T0293 - Neuro Computing
T Neuro Computing

18. Perceptron Training Method
loop until done foreach training data item compute output using weights and bias if the output is incorrect then adjust weights and bias compute error if error < smallest error so far smallest error so far = error save new weights and bias end if increment loop counter end foreach end loop return best weights and bias values found
T Neuro Computing

19. training
Although the training method doesn't have very many lines of code, it's quite clever. The heart of the method is the line that adjusts the weights when a training item is incorrectly classified:
weights[j] = weights[j] + (alpha * delta * trainData[i][j]);
The adjustment value has three terms: alpha, delta and the input value associated with the weight being adjusted. Delta is computed as desired - output, that is, the desired value stored in the Y array minus the output produced by the current weights and bias.
T Neuro Computing

20. The learning rate, alpha, is typically some small value such as 0
The learning rate, alpha, is typically some small value such as or , and throttles the magnitude of change when weights and bias values are adjusted. An advanced alternative is to allow alpha to change, starting out with a relatively large value and gradually decreasing. The idea is to have large adjustment jumps early on but then make the jumps finer-grained later. Another advanced alternative is to set alpha to a small random value each time it's used.
The version of the perceptron-training algorithm presented here iterates a fixed maxEpochs times (500 in the demo).
T Neuro Computing

21. TM1 (group)
Create a Neural Network program using Backpropagation using C#, and analyze it:
T Neuro Computing

22. References
Textbook
Haykin, S. (2009). Neural Networks and Learning Machines, 3rd ed. Pearson. ISBN:
Web
T Neuro Computing

23. END
T Neuro Computing