1. Introduction to
Neural Networks
Eric Wong
Martin Ho
Eddy Li
Kitty Wong

2. Presentation Outline 1) Introduction to Neural Networks
2) What is Neural Networks?
3) Perceptrons
4) Introduction to Backpropagation
5) Applications of Neural Networks
6) Summary
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

3. Background - Neural Networks can be : - Biological models
- Artificial models
- Desire to produce artificial systems capable of
sophisticated computations similar to the human brain.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

4. How Does the Brain Work ? (1)
NEURON
- The cell that perform information processing in the brain.
- Fundamental functional unit of all nervous system tissue.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

5. How Does the Brain Work ? (2)
Each consists of :
SOMA, DENDRITES, AXON, and SYNAPSE.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

6. Brain vs. Digital Computers (1)
Computers require hundreds of cycles to simulate
a firing of a neuron.
- The brain can fire all the neurons in a single step.
Parallelism
- Serial computers require billions of cycles to
perform some tasks but the brain takes less than
a second.
e.g. Face Recognition
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

7. Brain vs. Digital Computers (2)
Future : combine parallelism of the brain with the
switching speed of the computer.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

8. Definition of Neural Network
A Neural Network is a system composed of
many simple processing elements operating in
parallel which can acquire, store, and utilize
experiential knowledge.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

9. Neurons vs. Units (1) - Each element is a node called unit.
- Units are connected by links.
- Each link has a numeric weight.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

10. Neurons vs. Units (2)
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

11. Computing Elements A typical unit:
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

12. Presentation Outline 1) Introduction to Neural Networks
2) What is Neural Networks?
3) Perceptrons
4) Introduction to Backpropagation
5) Applications of Neural Networks
6) Summary
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

13. Planning in building a Neural Network
Decisions on:
- The number of units to use.
- The type of units required.
- Connection between the units.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

14. Learning a Task - Initializing the weights.
- Use of a learning algorithm.
- Set of training examples.
- Encode the examples as inputs.
- Convert output into meaningful results.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

15. Neural Network Example
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

16. Simple Computations - 2 components: Linear and Non-linear.
- Linear: Input function
- calculate weighted sum of all inputs.
- Non-linear: Activation function
- transform sum into activation level.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

17. Calculations Input function: Activation function:
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

18. A Computing Unit
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

19. Activation Functions
- Use different functions to obtain different models.
- 3 most common choices :
1) Step function
2) Sign function
3) Sigmoid function
- An output of 1 represents firing of a neuron down the
axon.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

20. 3 Activation Functions
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

21. Units in Action - Individual units representing Boolean functions
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

22. Network Structures Feed-forward : Links can only go in one direction.
Recurrent :
Links can go anywhere and form arbitrary
topologies.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

23. Feed-forward Networks
- Arranged in layers.
- Each unit is linked only in the unit in next layer.
No units are linked between the same layer, back to
the previous layer or skipping a layer.
- Computations can proceed uniformly from input to
output units.
- No internal state exists.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

24. Feed-Forward Example H3 H5 I1 O7 I2 H4 H6
t = -0.5
t = 1.5
W57 = 1 I1
W25 = 1 O7
t = 0.5
W16 = 1 I2
W67 = 1
W24= -1
t = -0.5
t = 1.5
W46 = 1 H4 H6
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

25. Multi-layer Networks and Perceptrons
- Have one or more layers of hidden units.
- With two possibly very large hidden layers, it is possible
to implement any function.
- Networks without hidden layer are called perceptrons.
- Perceptrons are very limited in what they can represent,
but this makes their learning problem much simpler.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

26. Recurrent Network (1)
- The brain is not and cannot be a feed-forward network.
- Allows activation to be fed back to the previous unit.
- Internal state is stored in its activation level.
- Can become unstable or oscillate.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

27. Recurrent Network (2) - May take long time to compute a stable output.
- Learning process is much more difficult.
- Can implement more complex designs.
- Can model certain systems with internal states.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

28. Presentation Outline 1) Introduction to Neural Networks
2) What is Neural Networks?
3) Perceptrons
4) Introduction to Backpropagation
5) Applications of Neural Networks
6) Summary
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

29. Perceptrons - First studied in the late 1950s.
- Also known as Layered Feed-Forward Networks.
- The only efficient learning element at that time was
for single-layered networks.
- Today, used as a synonym for a single-layer,
feed-forward network.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

30. Perceptrons
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

31. What can Perceptrons Represent ?
- Some complex Boolean function can be represented.
For example: Majority function - will be covered in the next
presentation.
- Limited in the Boolean functions they can represent.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

32. Linear Separability in Perceptrons
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

33. Learning Linearly Separable Functions (1)
What can these functions learn ?
Bad news:
- There are not many linearly separable functions.
Good news:
- There is a perceptron algorithm that will learn
any linearly separable function, given enough
training examples.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

34. Learning Linearly Separable Functions (2)
Most neural network learning algorithms, including the
perceptrons learning method, follow the current-best-
hypothesis (CBH) scheme.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

35. Learning Linearly Separable Functions (3)
- Initial network has a randomly assigned weights.
- Done by making small adjustments in the weights to reduce
the difference between the observed and predicted values.
- Main difference from the logical algorithms is need to repeat
the update phase several times in order to achieve
convergence.
- Updating process is divided into epochs, each epoch
updates all the weights of the process.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

36. Neural-Network-Learning
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

37. Presentation Outline 1) Introduction to Neural Networks
2) What is Neural Networks?
3) Perceptrons
4) Introduction to Backpropagation
5) Applications of Neural Networks
6) Summary
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

38. Introduction to Backpropagation
- In 1969 a method for learning in multi-layer network,
Backpropagation, was invented by Bryson and Ho.
- The Backpropagation algorithm is a sensible approach
for dividing the contribution of each weight.
- Works basically the same as perceptrons
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

39. Backpropagation Learning
There are two differences for the updating rule :
1) The activation of the hidden unit is used instead of the
input value.
2) The rule contains a term for the gradient of the activation
function.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

40. Backpropagation Algorithm(1)
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

41. Backpropagation Algorithm(2)
The ideas of the algorithm can be summarized as follows :
- Computes the error term for the output units using the
observed error.
- From output layer, repeat propagating the error term back
to the previous layer and updating the weights between the
two layers until the earliest hidden layer is reached.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

42. Examples of Backpropagation Learning
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

43. Examples of Feed-Forward Learning
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

44. Examples of Backpropagation Learning
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

45. Presentation Outline 1) Introduction to Neural Networks
2) What is Neural Networks?
3) Perceptrons
4) Introduction to Backpropagation
5) Applications of Neural Networks
6) Summary
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

46. Neural Network Applications
- May provide a model for massive parallel computation.
- More successful approach of “parallelizing” traditional
serial algorithms.
- Can compute any computable function.
- Can do everything a normal digital computer can do.
- Can do even more under some impractical assumptions.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

47. Neural Network in Practice
For classification and function approximation or mapping
problems which are:
- Tolerant of some imprecision.
- Have lots of training data available.
- Hard and fast rules cannot easily be applied.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

48. Neural Network Approaches
ALVINN - Autonomous Land Vehicle In a Neural Network
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

49. Neural Network Approaches
- Developed in 1993.
- Performs driving with
Neural Networks.
- An intelligent VLSI image
sensor for road following.
- Learns to filter out image
details not relevant to
driving.
Output units
Hidden layer
Input units
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

50. Neural Network Approaches
Input Array
Hidden Units
Output units
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

51. Actual Products Available
Enterprise Miner:
- Single multi-layered feed-forward neural networks.
- Provides business solutions for data mining.
Nestor:
- Uses Nestor Learning System (NLS).
- Several multi-layered feed-forward neural networks.
- Intel has made such a chip - NE1000 for VLSI.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

52. Enterprise Miner - Based on SEMMA (Sample, Explore, Modify, Model,
Access) methodology.
- Statistical tools include :
Clustering, decision trees, linear and logistic regression and neural networks.
- Data preparation tools include :
Outliner detection, variable transformation, random sampling, and partition of data sets (into training,
testing and validation data sets).
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

53. Nestor - With low connectivity within each layer.
- Minimized connectivity within each layer results in rapid
training and efficient memory utilization, ideal for VLSI.
- Composed of multiple neural networks, each specializing
in a subset of information about the input patterns.
- Real time operation without the need of special computers
or custom hardware.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

54. Presentation Outline 1) Introduction to Neural Networks
2) What is Neural Networks?
3) Perceptrons
4) Introduction to Backpropagation
5) Applications of Neural Networks
6) Summary
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

55. Summary - Neural network is a computational model that simulate
some properties of the human brain.
- The connections and nature of units determine the
behavior of a neural network.
- Perceptrons are feed-forward networks that can only
represent linearly separable functions.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

56. Summary - Given enough units, any function can be represented
by Multi-layer feed-forward networks.
- Backpropagation learning works on multi-layer
feed-forward networks.
- Neural Networks are widely used in developing
artificial learning systems.
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000

57. References
- Russel, S. and P. Norvig (1995). Artificial Intelligence - A
Modern Approach. Upper Saddle River, NJ, Prentice
Hall.
- Sarle, W.S., ed. (1997), Neural Network FAQ, part 1 of 7:
Introduction, periodic posting to the Usenet newsgroup
comp.ai.neural-nets,
URL: ftp://ftp.sas.com/pub/neural/FAQ.html
Presented by Martin Ho, Eddy Li, Eric Wong and Kitty Wong - Copyright© 2000