1. Backpropagation

2. Multilayer Perceptron
R – S1 – S2 – S3 Network

3. Example

4. Elementary Decision Boundaries
First Boundary:
Second Boundary:
First Subnetwork

5. Elementary Decision Boundaries
Third Boundary:
Fourth Boundary:
Second Subnetwork

6. Total Network

7. Function Approximation Example
Nominal Parameter Values

8. Nominal Response

9. Parameter Variations

10. Multilayer Network

11. Performance Index Training Set Mean Square Error Vector Case
Approximate Mean Square Error (Single Sample)
Approximate Steepest Descent

12. Application to Gradient Calculation
Chain Rule
Example
Application to Gradient Calculation

13. Gradient Calculation
Sensitivity
Gradient

14. Next Step: Compute the Sensitivities (Backpropagation)
Steepest Descent s m F ˆ n - 1 2 S =
Next Step: Compute the Sensitivities (Backpropagation)

15. Jacobian Matrix n m 1 + - 2 S

16. Backpropagation (Sensitivities)
The sensitivities are computed by starting at the last layer, and
then propagating backwards through the network to the first layer.

17. Initialization (Last Layer)

18. Summary
Forward Propagation
Backpropagation
Weight Update

19. Example: Function Approximation- e +
1-2-1
Network a

20. Network
1-2-1
Network a p

21. Initial Conditions

22. Forward Propagation

23. Transfer Function Derivatives

24. Backpropagation

25. Weight Update

26. Choice of Architecture
1-3-1 Network
i = 1
i = 2
i = 4
i = 8

27. Choice of Network Architecture
1-2-1
1-3-1
1-4-1
1-5-1

28. Convergence 5 1 5 3 3 4 2 4 2 1

29. Generalization
1-2-1
1-9-1