1. Outline Soft computing Fuzzy logic and fuzzy inference systems
2018/9/11
Outline
Soft computing
Fuzzy logic and fuzzy inference systems
Neural networks
Neuro-fuzzy integration: ANFIS
Derivative-free optimization
Genetic algorithms
Simulated annealing
Random search
Examples and demos
Specifically, this is the outline of the talk. Wel start from the basics, introduce the concepts of fuzzy sets and membership functions. By using fuzzy sets, we can formulate fuzzy if-then rules, which are commonly used in our daily expressions. We can use a collection of fuzzy rules to describe a system behavior; this forms the fuzzy inference system, or fuzzy controller if used in control systems. In particular, we can can apply neural networks?learning method in a fuzzy inference system. A fuzzy inference system with learning capability is called ANFIS, stands for adaptive neuro-fuzzy inference system. Actually, ANFIS is already available in the current version of FLT, but it has certain restrictions. We are going to remove some of these restrictions in the next version of FLT. Most of all, we are going to have an on-line ANFIS block for SIMULINK; this block has on-line learning capability and it ideal for on-line adaptive neuro-fuzzy control applications. We will use this block in our demos; one is inverse learning and the other is feedback linearization.

2. Neuro-Fuzzy and Soft Computing
Neural networks
Fuzzy inf. systems
Model space
Adaptive networks
Soft
Computing
Derivative-free optim.
Derivative-based optim.
Approach space

3. Fuzzy Sets Sets with fuzzy boundaries A = Set of tall people
2018/9/11
Fuzzy Sets
Sets with fuzzy boundaries
A = Set of tall people
Heights
(cm)
170
1.0
Crisp set A
Membership
function
Heights
(cm)
170
180 .5 .9
Fuzzy set A
1.0
A fuzzy set is a set with fuzzy boundary. Suppose that A is the set of tall people. In a conventional set, or crisp set, an element is either belong to not belong to a set; there nothing in between. Therefore to define a crisp set A, we need to find a number, say, 5??, such that for a person taller than this number, he or she is in the set of tall people. For a fuzzy version of set A, we allow the degree of belonging to vary between 0 and 1. Therefore for a person with height 5??, we can say that he or she is tall to the degree of 0.5. And for a 6-foot-high person, he or she is tall to the degree of .9. So everything is a matter of degree in fuzzy sets. If we plot the degree of belonging w.r.t. heights, the curve is called a membership function. Because of its smooth transition, a fuzzy set is a better representation of our mental model of all? Moreover, if a fuzzy set has a step-function-like membership function, it reduces to the common crisp set.

4. Membership Functions (MFs)
2018/9/11
Membership Functions (MFs)
About MFs
Subjective measures
Not probability functions
“tall” in Taiwan
MFs
“tall” in the US
“tall” in NBA .8
Here I like to emphasize some important properties of membership functions. First of all, it subjective measure; my membership function of all?is likely to be different from yours. Also it context sensitive. For example, I 5?1? and I considered pretty tall in Taiwan. But in the States, I only considered medium build, so may be only tall to the degree of .5. But if I an NBA player, Il be considered pretty short, cannot even do a slam dunk! So as you can see here, we have three different MFs for all?in different contexts. Although they are different, they do share some common characteristics --- for one thing, they are all monotonically increasing from 0 to 1. Because the membership function represents a subjective measure, it not probability function at all. .5 .1
180
Heights
(cm)

5. Fuzzy If-Then Rules Mamdani style
2018/9/11
Fuzzy If-Then Rules
Mamdani style
If pressure is high then volume is small
high
small
Sugeno style
If speed is medium then resistance = 5*speed
By using fuzzy sets, we can formulate fuzzy if-then rules that are commonly used in our daily expressions. Basically, we have two types of fuzzy rules. For Mamdani style, for instance, if pressure is high then volume is small, where igh?and mall?are described by fuzzy sets. For Sugeno style, if the speed of a moving object is medium then the resistance due to atmosphere is 5 times the speed. The basic difference between these two rules is in their THEN part, where Madman style has a fuzzy but Surgeon style has a linear equation. Madman style fuzzy rules were first proposed in the literature; they are more appealing to human intuition. Surgeon style fuzzy rules are proposed later, but they are more suited for mathematical design and analysis. In this talk, wel concentrate on Surgeon style fuzzy if-then rules.
medium
resistance = 5*speed

6. Fuzzy Inference System (FIS)
2018/9/11
Fuzzy Inference System (FIS)
If speed is low then resistance = 2
If speed is medium then resistance = 4*speed
If speed is high then resistance = 8*speed
MFs
low
medium
high .8 .3
A single fuzzy rule is not very interesting. But if we have a collection of fuzzy rules, we can use them to describe a system behavior. This leads to a fuzzy inference system. For instance, we can describe the resistance experienced by a moving object by the following three rules: .... Then given a crisp speed value, how do we find the resistance value from these three rules? It quite simple and can be done in three steps. In the first step, we find the membership grades for ow? edium? and igh? For instance, if speed is 2, the membership grades for ow? edium?and igh?are .3, .8, and .1, respectively. These numbers also represent how the given input condition peed = 2?satisfies the IF part of the rules. Sometimes these numbers are called the firing strengths of the rules. In the second step, we find the output of each rule, given speed is 2. In the third step, we apply a weighted average method to find the overall resistance, where the weighting factors are equal to the firing strengths of the rules. The whole process to derive the output from a given input condition is called fuzzy reasoning. For a two-input FIS, the process of fuzzy reasoning is better represented by the following diagram. .1 2
Speed
Rule 1: w1 = .3; r1 = 2
Rule 2: w2 = .8; r2 = 4*2
Rule 3: w3 = .1; r3 = 8*2
Resistance = S(wi*ri) / Swi
= 7.12

7. First-Order Sugeno FIS
2018/9/11
First-Order Sugeno FIS
Rule base
If X is A1 and Y is B1 then Z = p1*x + q1*y + r1
If X is A2 and Y is B2 then Z = p2*x + q2*y + r2
Fuzzy reasoning A1 B1 A2 B2
x=3 X Y
y=2 w1 w2
z1 =
p1*x+q1*y+r1
z =
z2 =
p2*x+q2*y+r2
w1+w2
w1*z1+w2*z2 P
In this talk, we are going to use first-order Sugeno fuzzy inference system exclusively, where the output equation of each rule is a linear equation. For example, if we have two fuzzy rules ...
We can express the process of fuzzy reasoning by this diagram. First we find the membership grades of the IF parts of the rules; the heights of the dashed line represent these values. Since the pre-conditions in the IF part are connected by AND, so we use multiplication to find the firing strength of each rule. For instance, firing strength w1 for rule 1 is the product of the heights of these two dashed lines. Similar for w2. Once we have w1 and w2, the overall output is again derived by weighted average.

8. Neural Networks Supervised Learning Unsupervised Learning Others
2018/9/11
Neural Networks
Supervised Learning
Multilayer perceptrons
Radial basis function networks
Modular neural networks
LVQ (learning vector quantization)
Unsupervised Learning
Competitive learning networks
Kohonen self-organizing networks
ART (adaptive resonant theory)
Others
Hopfield networks

9. Single-Layer Perceptrons
2018/9/11
Single-Layer Perceptrons
Network architecture x1 w1 w0 w2 x2
y = signum(Swi xi + w0) w3 x3
Dwi = k t xi
Learning rule

10. Single-Layer Perceptrons
2018/9/11
Single-Layer Perceptrons
Example: Gender classification h v w1 w2 w0
Network Arch.
y = signum(hw1+vw2+w0)
-1 if female
1 if male = y
Training data
h (hair length)
v (voice freq.)

11. Multilayer Perceptrons (MLPs)
Network architecture x1 x2 y1 y2
hyperbolic tangent
or logistic function
Learning rule:
Steepest descent (Backprop)
Conjugate gradient method
All optim. methods using first derivative
Derivative-free optim.

12. Multilayer Perceptrons (MLPs)
Example: XOR problem
Training data x1 x2 y
Network Arch.
x1 x2 y x1 x2 x1 x2 y

13. MLP Decision Boundaries
XOR
Interwined
General
1-layer: Half planes A B
2-layer: Convex A B
3-layer: Arbitrary

14. Adaptive Networks Architecture: Goal: Basic training method: x z y
2018/9/11
Adaptive Networks x z y
Architecture:
Feedforward networks with diff. node functions
Squares: nodes with parameters
Circles: nodes without parameters
Goal:
To achieve an I/O mapping specified by training data
Basic training method:
Backpropagation or steepest descent

15. Derivative-Based Optimization
Based on first derivatives:
Steepest descent
Conjugate gradient method
Gauss-Newton method
Levenberg-Marquardt method
And many others
Based on second derivatives:
Newton method

16. Fuzzy Inference system
2018/9/11
Fuzzy Modeling x1 xn
. . .
Unknown target system y
Fuzzy Inference system y*
Given desired i/o pairs (training data set) of the form
(x1, ..., xn; y), construct a FIS to match the i/o pairs
Given an input condition, it very easy to derive the overall output. However, our task is more than that. What we want to do is to construct an appropriate fuzzy inference system that can match a desired input/output data set of a unknown target system. This is called fuzzy modeling and it a branch of nonlinear system identification. In general, fuzzy modeling involves two step. The first step is structure identification, where we have to find suitable numbers of fuzzy rules and membership functions. In FLT, this is accomplished by subtractive clustering proposed by Steve Chiu in Rockwell Science Research Center. The second step is parameter identification, where we have to find the optimal parameters for membership functions as well as output linear equations. In the FLT, this is done by ANFIS, which was proposed by me back in Since ANFIS uses some neural network training techniques, it is also classified as one of the neuro-fuzzy modeling approaches.
Two steps in fuzzy modeling
structure identification --- input selection, MF numbers
parameter identification --- optimal parameters

17. Neuro-Fuzzy Modeling Basic approach of ANFIS Adaptive networks
2018/9/11
Neuro-Fuzzy Modeling
Basic approach of ANFIS
Adaptive networks
Generalization
Specialization
Neural networks
Fuzzy inference
systems
Our approach of using ANFIS as a neuro-fuzzy modeling tool is like this. First we generalize neural networks?architectures to obtain adaptive networks, and then we do a specialization to derive fuzzy inference systems represented by adaptive networks, and that ANFIS. During the processes of generalization and specialization, the backpropagation techniques used for training neural networks can be carried over directly, so we can train ANFIS using the same techniques. (In fact backpropagation is too slow, so we have some other technique to speed up training in ANFIS.)
ANFIS

18. ANFIS Fuzzy reasoning ANFIS (Adaptive Neuro-Fuzzy Inference System) A1
2018/9/11
ANFIS
Fuzzy reasoning A1 B1 A2 B2 w1 w2
z1 =
p1*x+q1*y+r1
z2 =
p2*x+q2*y+r2
z =
w1+w2
w1*z1+w2*z2 x y
This slide explains the basic architecture of an ANFIS. In the upper part, we have to graphical representation of the process of fuzzy reasoning. For each operation of fuzzy reasoning, we can put it into a node in an adaptive network. For instance, given two input x and y, first we need to find the membership grades; this is represented by the four blocks in the first layer and each of them generates a membership grades. The firing strengths are computed by nodes in the second layer;
ANFIS (Adaptive Neuro-Fuzzy Inference System) A1 A2 B1 B2 P S / x y w1 w2
w1*z1
w2*z2
Swi*zi
Swi z

19. Four-Rule ANFIS Input space partitioningB1 A2 B2 x y
ANFIS (Adaptive Neuro-Fuzzy Inference System) A1 A2 B1 B2 S / x y w1 w4
w1*z1
w4*z4
Swi*zi
Swi z P

20. ANFIS: Parameter ID Hybrid training method x z y S / S nonlinear
parameters
linear
parameters w1 A1 P
w1*z1 x A2 P S
Swi*zi B1 P / z y B2 P
w4*z4
Swi w4 S
forward pass
backward pass
MF param.
(nonlinear)
fixed
steepest descent
Coef. param.
(linear)
least-squares
fixed

21. Parameter ID: Gauss-Newton Method
Synonyms:
linearization method
extended Kalman filter method
Concept:
general nonlinear model: y = f(x, q)
linearization at q = qnow:
y = f(x, qnow)+a1(q1 - q1,now)+a2(q2 - q2,now) + ...
LSE solution:
qnext = qnow + h(A A) A B T -1 T

22. Param. ID: Levenberg-Marquardt
Formula:
qnext = qnow + h(A A + lI) A B
Effects of l:
l small Gauss-Newton method
l big steepest descent
How to update l:
greedy policy make l small
cautious policy make l large T -1 T

23. Param. ID: Comparisons Steepest descent (SD) Hybrid learning (SD+LSE)
treats all parameters as nonlinear
Hybrid learning (SD+LSE)
distinguishes between linear and nonlinear
Gauss-Newton (GN)
linearizes and treat all parameters as linear
Levenberg-Marquardt (LM)
switches smoothly between SD and GN

24. ANFIS: Structure ID Input selection Input space partitioning
To select relevant input for efficient modeling
Grid partitioning
Tree partitioning
Scatter partitioning
C-means clustering
mountain method
hyperplane clustering
CART method

25. Derivative-Free Optimization
Genetic algorithms (GAs)
Simulated annealing (SA)
Random search
Downhill simplex search
Tabu search

26. Genetic Algorithms Motivation Look at what evolution brings us? Vision
Hearing
Smelling
Taste
Touch
Learning and reasoning
Can we emulate the evolutionary process with today's fast computers?

27. Genetic Algorithms Terminology: Fitness function Polulation
Encoding schemes
Selection
Crossover
Mutation
Elitism

28. Genetic Algorithms Binary encoding Crossover Mutation
Chromosome
(11, 6, 9)
Gene
Crossover
Crossover point
Mutation
Mutation bit

29. Genetic Algorithms Flowchart Current generation Next generation
. . .
. . .
Elitism
Selection
Crossover
Mutation
Current
generation
Next
generation

30. Genetic Algorithms Example: Find the max. of the “peaks” function
z = f(x, y) = 3*(1-x)^2*exp(-(x^2) - (y+1)^2) - 10*(x/5 - x^3 - y^5)*exp(-x^2-y^2) -1/3*exp(-(x+1)^2 - y^2).

31. Genetic Algorithms Derivatives of the “peaks” function
dz/dx = -6*(1-x)*exp(-x^2-(y+1)^2) - 6*(1-x)^2*x*exp(-x^2-(y+1)^2) - 10*(1/5-3*x^2)*exp(-x^2-y^2) + 20*(1/5*x-x^3-y^5)*x*exp(-x^2-y^2) - 1/3*(-2*x-2)*exp(-(x+1)^2-y^2)
dz/dy = 3*(1-x)^2*(-2*y-2)*exp(-x^2-(y+1)^2) + 50*y^4*exp(-x^2-y^2) + 20*(1/5*x-x^3-y^5)*y*exp(-x^2-y^2) + 2/3*y*exp(-(x+1)^2-y^2)
d(dz/dx)/dx = 36*x*exp(-x^2-(y+1)^2) - 18*x^2*exp(-x^2-(y+1)^2) - 24*x^3*exp(-x^2-(y+1)^2) + 12*x^4*exp(-x^2-(y+1)^2) + 72*x*exp(-x^2-y^2) - 148*x^3*exp(-x^2-y^2) - 20*y^5*exp(-x^2-y^2) + 40*x^5*exp(-x^2-y^2) + 40*x^2*exp(-x^2-y^2)*y^5 -2/3*exp(-(x+1)^2-y^2) - 4/3*exp(-(x+1)^2-y^2)*x^2 -8/3*exp(-(x+1)^2-y^2)*x
d(dz/dy)/dy = -6*(1-x)^2*exp(-x^2-(y+1)^2) + 3*(1-x)^2*(-2*y-2)^2*exp(-x^2-(y+1)^2) + 200*y^3*exp(-x^2-y^2)-200*y^5*exp(-x^2-y^2) + 20*(1/5*x-x^3-y^5)*exp(-x^2-y^2) - 40*(1/5*x-x^3-y^5)*y^2*exp(-x^2-y^2) + 2/3*exp(-(x+1)^2-y^2)-4/3*y^2*exp(-(x+1)^2-y^2)

32. Genetic Algorithms GA process: Initial population 5th generation

33. Genetic Algorithms
Performance profile

34. Simulated Annealing
Analogy

35. Simulated Annealing Terminology:
Objective function E(x): function to be optiimized
Move set: set of next points to explore
Generating function: to select next point
Acceptance function h(DE, T): to determine if the selected point should be accept or not. Usually h(DE, T) = 1/(1+exp(DE/(cT)).
Annealing (cooling) schedule: schedule for reducing the temperature T

36. Simulated Annealing Flowchart Select a new point xnew in the move sets
via generating function
Compute the obj. function E(xnew)
Set x to xnew with prob. determined
by h(DE, T)
Reduce temperature T

37. Simulated Annealing Example: Travel Salesperson Problem (TSP)
How to transverse n cities once and only
once with a minimal total distance?

38. Simulated Annealing Move sets for TSP Translation Inversion Switching12 12 10 10 3 3
Translation 1
Inversion 1 6 6 7 7 2 2 11 9 11 9 8 8 4 5 4 5 12 12 10 10 3
Switching 3 1 1 6 6 7 7 2 2 11 9 11 9 8 8 4 5 4 5

39. Simulated Annealing A 100-city TSP using SA Initial random path
During SA process
Final path

40. Simulated Annealing
100-city TSP with penalities when crossing the circle
Penalty = 0
Penalty = 0.5
Penalty = -0.3

41. Conclusions
Contributing factors to successful applications of neuro-fuzzy and soft computing:
Sensor technologies
Cheap fast microprocessors
Modern fast computers

42. References and WWW Resources
“Neuro-Fuzzy and Soft Computing”, J.-S. R. Jang, C.-T. Sun and E. Mizutani, Prentice Hall, 1996
“Neuro-Fuzzy Modeling and Control”, J.-S. R. Jang and C.-T. Sun, the Proceedings of IEEE, March 1995.
“ANFIS: Adaptive-Network-based Fuzzy Inference Systems,”, J.-S. R. Jang, IEEE Trans. on Systems, Man, and Cybernetics, May 1993.
Internet resources:
This set of slides is available at
WWW resouces about neuro-fuzzy and soft-computing