1. Advances in Neuro-Fuzzy Systems and their Applications
An Invited Lecture at
AICTE-ISTE Sponsored Short Term Training Programme on
Advances in Neuro-Fuzzy Systems and their Applications at
A D Patel Institute of Technology,
New Vidyanagar, Anand

2. Neuro-Fuzzy Systems Part 1
Dr. Priti Srinivas Sajja
P G Department of Computer Science and Technology
Sardar Patel University
Vallabh Vidyanagar

3. Introduction and Contact Information:
Name: Dr. Priti Srinivas Sajja
Communication:
Mobile :
Website :priti.sajja.info
Academic qualifications : Ph. D in Computer Science
Thesis title: Knowledge-Based Systems for Socio-Economic
Rural Development
Subject area of specialization : Artificial Intelligence
Publications : 84 in International and National Books, Chapters and Papers
Academic position : Associate Professor at
Department of Computer Science
Sardar Patel University
Vallabh Vidyanagar
March 20, 2007

4. Lecture Plan: Part 1: Part 2: AI and Soft Computing
Introduction to Fuzzy Logic
Introduction to Neural Network
Part 2:
Neuro fuzzy systems: fusion
Advantages and requirements
Approaches and structures
Applications of neuro-fuzzy systems
Tools and resources
References
March 20, 2007

5. Artificial Intelligence
“Artificial Intelligence(AI) is the study of how to make computers do things at which, at the moment, people are better”
-Elaine Rich, Artificial Intelligence, Mcgraw Hill Publications, 1986
March 20, 2007

6. Artificial Intelligence:
AI involves
Studying the thought process of humans
Deals with representing those processes via machines.
March 20, 2007

7. AI implementation leads to:
Intelligence become permanent
Speedy problem solving
Ease of duplication
Less expensive
Ease of documentation etc.
March 20, 2007

8. Knowledge-Based Systems:
Knowledge-Based Systems (KBS) are Productive Artificial Intelligence Tools working in a narrow domain.
March 20, 2007

9. How Knowledge is organized?:
Volume
Complexity &
Sophistication
Wisdom(experience)
Knowledge(synthesis)
Information(analysis)
Data
Data Pyramid
Source: Tuthill & Leavy, modified
March 20, 2007

10. Explanation/ Reasoning
Structure of KBS:
Knowledge Base
Explanation/ Reasoning
Self Learning
Inference Engine
User Interface
March 20, 2007

11. Soft computing techniques
Evolutionary
algorithms
Neuro-
-computing
Rough
sets
Uncertain
variables
Probabilistic
techniques
Soft
computing
Fuzzy
logic
March 20, 2007

12. Fuzzy numbers and logic:
Crisp 8
Fuzzy 8
March 20, 2007

13. Fuzzy Logic and Systems:
Humans routinely and subconsciously place things into classes whose meaning and significance are well understood but whose boundaries are not well defined.
Hot season, large car, young boy and rich people are the examples for the same.
March 20, 2007

14. Membership functions:
Temperature is low 1
Crisp Set: A set of temperature T which consists all temperature reading between 0* c to 40* c.
That is if t=27*c the tT
But it is said that “the temperature is very low” then one can not exactly claim that low temp t is a member of T
0.5
Low(t)=0.98 if t=10 *c
March 20, 2007

15. Another example….Set of tall people…
Heights
5’10’’
1.0
Crisp set
Membership
function
Heights
5’10’’
6’2’’ .5 .9
Fuzzy set
1.0
MFs
Heights
5’10’’ .5 .8 .1
“tall” in Asia
“tall” in the US
“tall” in xyz
March 20, 2007

16. Air conditioning machine example:
cold
Hot
Comfortable
integrated rules
March 20, 2007

17. Fuzzy Control systems:
Process
Fuzzifier / Defuzzifier
Input
Output
Fuzzy control rules, sets, membership function definitions
fuzzy
crisp
March 20, 2007

18. Advantages of fuzzy logic:
Linguistic values used making it simpler to way human think.
Allows the solution to previously unsolved problems.
Rapid prototyping is possible as knowledge is not required before starting work.
March 20, 2007

19. Advantages of fuzzy logic :
Cheaper to make than conventional system as easier to design
Increased robustness.
Simpler knowledge acquisition and representation.
A few rules are used to describe great complexity.
March 20, 2007

20. Connectionist system:
Objective: Not to mimic brain functionality but to receive inspiration from the fact about how brain is working.
Characterized by:
A large number of very simple neuron like processing elements.
A large number of weighted connection between the elements. This weights encode the knowledge of a network.
Highly parallel, distributed control.
An emphasis on learning internal representation automatically.
March 20, 2007

21. Human Brain
March 20, 2007

22. Neuron
March 20, 2007

23. Neuron
March 20, 2007

24. Model of an artificial neuron
March 20, 2007

25. Modeling Connectionist Systems(ANN):
Hopefiled network
Perceptron
Multi-layer feed forward back propogation
March 20, 2007

26. How are these features achieved?
A simple Hopefield network is shown here.
Active
Inactive -1 +1 +3 -2 +2
March 20, 2007

27. In a Hopefield network, all processing units/elements are in two states either active or inactive.
Units are connected to each other with weighted, symmetric connections.
A positively weighted connection indicates that the units tend to active each other.
A negative connection allows an active unit to deactivate a neighbouring unit.
March 20, 2007

28. A Simple Hopfield Network
Active
Inactive -1 +1 +3 -2 +2
A random unit is chosen.
If any of its neighbours are active, the unit computes the sum of weights on the connections to those active neighbours.
If the sum is positive, the unit becomes active else new random unit is chosen.
This process will continue till the network become stable. That is no unit can change its status. This process is known as parallel relaxation.
March 20, 2007

29. Perceptron With Adjustable Threshold1 X1
W1 W2 X2 
ƒƒƒ X3 …. W3 WN XN
March 20, 2007

30. Perceptron With Many Inputs and Outputs ƒ 1  ƒ X1 X2  ƒ X3 …. …. …. XN  ƒ
March 20, 2007

31. Consider the following figure:
This problem is linearly separatable
Given values of x1 and x2, we want to train a perceptron to output 1 if it thinks the input belongs to the class of white dots and 0 if it thinks the input belongs to the class of filled dots
Decision Surface
March 20, 2007

32. A Perception Learning to Solve a Classification Problem:
K wo w1 w2
March 20, 2007

33. A Multi Layer Network- XOR Problem
non linearly separatable X1 X2 1  ƒ
-1.5
1.0
-9.0
-0.5
March 20, 2007

34. Fully connected,multi layered,feed-forward network structureOc
Output units
W2ij 1 h1 h2 h3 hB
Hidden Layer
w1ij
Input Layer 1 x1 x2 x3 x4 xA
……This network has three layers but there may be many.
March 20, 2007

35. Application of Neural Networks
Neural Networks may be divided into the following categories based on the complexity of the problem and the network’s behavior:
Pattern recognizers and associative memories
Pattern transformers
Connectionist Speech [3-layer backpropagation n/w]
Connectionist Vision [Hopfield n/w-parallel relaxation]
Combinatorial Problems
Other Applications: compress images, to classify sonar signals, to drive a vehicle along a road
Dynamic inferences
Still at a primitive stage
March 20, 2007

36. Connectionist AI and Symbolic AI
Search – Parallel relaxation.
Knowledge Representation – very large number of real-valued connection Structures often stored as distributed patterns of activation.
Learning – Backpropagation, Boltzmann machines, reinforcement learning, unsupervised learning.
Symbolic
Search – State space traversal.
Knowledge Representation – Predicate logic, semantic networks, frames, scripts.
Learning – Macro-operators, version spaces, explanation-based learning, discovery.
March 20, 2007

37. Thanks
End of Part 1
March 20, 2007

38. Neuro-Fuzzy Systems Part 2
NEURO-FUZZY Computing
(for More Intelligent System)

39. Knowledge Representation
Combining Neural and Fuzzy:
Neural Nets
Fuzzy Logic
Knowledge Representation
Implicit, the system cannot be easy interpreted or modified (-)
Explicit, verification and optimization easy and efficient (+++)
Trainability
Trains itself by learning from data sets (+++)
None, you have to define everything explicitly (-)
Get “best of both worlds”: Explicit Knowledge Representation from Fuzzy Logic with Training Algorithms from Neural Nets
March 20, 2007

40. Combining Neural and Fuzzy
The key benefit of fuzzy logic is simple "if-then" relations to describe systems behaviour.
This leads to simpler solution in less design time.
However, the designer has to derive the "if-then" rules from the data sets manually, which requires a major effort with large data sets.
When data sets contain knowledge about the system to be designed, a neural net promises a solution as it can train itself from the data sets.
March 20, 2007

41. However, only few commercial applications of neural nets exist
However, only few commercial applications of neural nets exist. This is a contrast to fuzzy logic, which is a very common design technique in Asia and Europe.
As neural net solutions remain a “black box” , it is impossible to interpret or manually change it
Selection of the appropriate net model and setting the parameters of the learning algorithm is still a "black art" and requires long experience.
Of the aforementioned reasons, the lack of an easy way to verify and optimize a neural net solution is probably the major limitation.
March 20, 2007

42. That is both neural nets and fuzzy logic are powerful design techniques that have its strengths and weaknesses.
Neural nets can learn from data sets while fuzzy logic solutions are easy to verify and optimize.
A clever combination of the two technologies delivers best of both worlds.
March 20, 2007

43. Hybrid Systems
Neuro-fuzzy
Genetic neural
Fuzzy genetic
Fuzzy neuro
genetic
Knowledge-based
Systems
Probabilistic reasoning
Approximate reasoning
Case based reasoning
Data Driven
Systems
Machine
Intelligence
Neural network
system
Evolutionary
computing
Fuzzy logic
Rough sets
Non-linear
Dynamics
Chaos theory
Rescaled range
analysis (wavelet)
Fractal analysis
Pattern recognition
and learning
Machine Intelligence: A core concept for grouping various advanced technologies with Learning
March 20, 2007

44. Possible Integrations:
Fuzzy Logic + ANN
ANN + GA
Fuzzy Logic + ANN + GA
Fuzzy Logic + ANN + GA + Rough Set
Neuro-fuzzy hybridization is the most visible
integration realized so far.
ANN: Artificial Neural Network
GA: Genetic Algorithms
March 20, 2007

45. Comparison of Expert Systems, Fuzzy Systems,
Neural Networks and Genetic Algorithms
March 20, 2007

46. Cooperative neuro-fuzzy approach:
Reference:
Neuro Fuzzy Systems: State-of-the-art Modeling Techniques
Ajith Abraham
School of Computing & Information Technology
Monash University, Churchill 3842, Australia
March 20, 2007

47. Concurrent neuro-fuzzy approach:
Reference:
Neuro Fuzzy Systems: State-of-the-art Modeling Techniques
Ajith Abraham
School of Computing & Information Technology
Monash University, Churchill 3842, Australia
March 20, 2007

48. Basic structure of a neural expert system
Inference Engine
Neural Knowledge Base
Rule Extraction
Explanation Facilities
User Interface
Rule: IF - THEN
Training Data
New
Data
User
March 20, 2007

49. The neural knowledge base
-0.8
-0.2
-0.1
-1.1
2.2
0.0
-1.0
2.8
-1.6
-2.9
-1.3
Bird
Plane
Glider +1
Wings
Tail
Beak
Feathers
Engine - 1
-0.7
1.9
Rule 2 3
1.0
Neurons in the network are connected by links, each of which has a numerical weight attached to it.
The weights in a trained neural network determine the strength or importance of the associated neuron inputs.
March 20, 2007

50. If we set each input of the input layer to either +1 (true), 1 (false), or 0 (unknown), we can give a semantic interpretation for the activation of any output neuron.
For example, if the object has Wings (+1), Beak (+1) and Feathers (+1), but does not have Engine (1), then we can conclude that this object is Bird (+1):
March 20, 2007

51. We can similarly conclude that this object is not Plane:
and not Glider:
March 20, 2007

52. By attaching a corresponding question to each input
neuron, we can enable the system to prompt the user
for initial values of the input variables:
Neuron: Wings
Question: Does the object have wings?
Neuron: Tail
Question: Does the object have a tail?
Neuron: Beak
Question: Does the object have a beak?
Neuron: Feathers
Question: Does the object have feathers?
Neuron: Engine
Question: Does the object have an engine?
March 20, 2007

53. An inference can be made if the known net
weighted input to a neuron is greater than the
sum of the absolute values of the weights of
the unknown inputs.
where i  known, j  known and n is the number
of neuron inputs.
March 20, 2007

54. An example of a multi-layer knowledge base
March 20, 2007

55. More general form of this network
March 20, 2007

56. Neuro-fuzzy system Fuzzification Fuzzy Rule Output Defuzzification
Input
March 20, 2007

57. Each layer in the neuro-fuzzy system is associated
with a particular step in the fuzzy inference process.
Layer 1 is the input layer. Each neuron in this layer transmits external crisp signals directly to the next layer. That is,
Layer 2 is the fuzzification layer. Neurons in this layer represent fuzzy sets used in the antecedents of fuzzy rules. A fuzzification neuron receives a crisp input and determines the degree to which this input belongs to the neuron’s fuzzy set.
March 20, 2007

58. The activation function of a membership neuron is set to the function that specifies the neuron’s fuzzy set. One may use triangular sets, and therefore, the activation functions for the neurons in Layer 2 are set to the triangular membership functions. A triangular membership function can be specified by two parameters {a, b} as follows:
March 20, 2007

59. Layer 3 is the fuzzy rule layer
Layer 3 is the fuzzy rule layer. Each neuron in this layer corresponds to a single fuzzy rule. A fuzzy rule neuron receives inputs from the fuzzification neurons that represent fuzzy sets in the rule antecedents. For instance, neuron R1, which corresponds to Rule 1, receives inputs from neurons A1 and B1.
In a neuro-fuzzy system, intersection can be implemented by the product operator. Thus, the output of neuron i in Layer 3 is obtained as:
March 20, 2007

60. This operation can be implemented by the probabilistic OR. That is,
Layer 4 is the output membership layer. Neurons in this layer represent fuzzy sets used in the consequent of fuzzy rules.
An output membership neuron combines all its inputs by using the fuzzy operation union.
This operation can be implemented by the probabilistic OR. That is,
The value of C1 represents the integrated firing strength of fuzzy rule neurons R3 and R6.
March 20, 2007

61. We can use the sum-product composition method.
Layer 5 is the defuzzification layer. Each neuron in this layer represents a single output of the neuro-fuzzy system. It takes the output fuzzy sets clipped by the respective integrated firing strengths and combines them into a single fuzzy set.
Neuro-fuzzy systems can apply standard defuzzification methods, including the centroid technique.
We can use the sum-product composition method.
March 20, 2007

62. The sum-product composition calculates the crisp output as the weighted average of the centroids of all output membership functions. For example, the weighted average of the centroids of the clipped fuzzy sets C1 and C2 is calculated as,
March 20, 2007

63. Neuro-fuzzy systems: summary
The combination of fuzzy logic and neural networks constitutes a powerful means for designing intelligent systems.
Domain knowledge can be put into a neuro-fuzzy system by human experts in the form of linguistic variables and fuzzy rules.
When a representative set of examples is available, a neuro-fuzzy system can automatically transform it into a robust set of fuzzy IF-THEN rules, and thereby reduce our dependence on expert knowledge when building intelligent systems.
March 20, 2007

64. Applications and Examples….
March 20, 2007

65. Flexible neuro-fuzzy system
L. Rutkowski and K. Cpałka „Flexible Neuro-Fuzzy Systems”, IEEE Trans. Neural Networks, vol. 14, pp , May 2003
March 20, 2007

66. Knowledge-based ANN for learning and rule extraction
Combine the strengths of different AI techniques, e.g. ANN and rule-based systems or fuzzy logic
FuNN (Kasabov et al, 1997)
Learning from data and rule extraction, e.g.:
R1: IF x1 is Small (DI11) and x2 is Small (DI21) THEN y is Small (CF1),
R2: IF x1 is Large (DI12) and x2 is Large (DI22) THEN y is Large (CF2).
In order to combine the strengths of different AI techniques hybrid techniques have been developed.
An example is the fuzzy neural network FuNN developed in our Knowledge Engineering Laboratory (KEL).
It facilitates both learning from data (as the ANN do) and fuzzy rule representation (as fuzzy logic systems do).
A simple two-input/one-output structure of a fuzzy neural network FuNN is shown on the figure. It consists of five layers of neurons and four layers of connections. The parameters in brackets define the functioning of each layer. The thick lines represent strong connections and also depict a structure that represents two complex IF-THEN fuzzy rules.
March 20, 2007

67. Prototype rules extracted from DENFIS and EFuNN after model and data integration
Takagi-Sugeno fuzzy rules (DENFIS):
Rule 1: IF x1 is (-0.05, 0.05, 0.14) and x2 is (0.15,0.25,0.35) THEN y = x x2
Rule 2: IF x1 is (0.02, 0.11, 0.21) and x2 is (0.45,0.55, 0.65) THEN y = x x2
Rule 3: IF x1 is (0.07, 0.17, 0.27) and x2 is (0.08,0.18,0.28) THEN y = x x2
Rule 4: IF x1is (0.26, 0.36, 0.46) and x2 is (0.44,0.53,0.63) THEN y = x x2
Rule 5: IF x1is (0.35, 0.45, 0.55) and x2 is (0.08,0.18,0.28) THEN y = x x2
Rule 6: IF x1is (0.52, 0.62, 0.72) and x2 is (0.45,0.55,0.65) THEN y = x x2
Rule 7: IF x1is (0.60, 0.69,0.79) and x2 is (0.10,0.20,0.30) THEN y = x x2
New rules:
Rule 8: IF x1is (0.65,0.75,0.85) and x2 is (0.70,0.80,0.90) THEN
y = x1+0.51x2
Rule 9: IF x1is (0.86,0.95,1.05) and x2 is (0.71,0.81,0.91) THEN
y = x1+0.37x2 
Zade-Mamdani fuzzy rules (ECF, EFuNN):
Rule 1: IF x1 is (Low 0.8) and x2 is (Low 0.8) THEN y is (Low 0.8), radius R1=0.24; N1ex= 6
Rule 2: IF x1 is (Low 0.8) and x2 is (Medium 0.7) THEN y is (Small 0.7), R2=0.26, N2ex= 9
Rule 3: IF x1 is (Medium 0.7) and x2 is (Medium 0.6) THEN y is (Medium 0.6), R3 = 0.17,N3ex=17
Rule 4: IF x1 is (Medium 0.9) and x2 is (Medium 0.7) THEN y is (Medium 0.9), R4 = 0.08, N4ex=10
Rule 5: IF x1 is (Medium 0.8) and x2 is (Low 0.6) THEN y is (Medium 0.9), R5= 0.1, N5ex = 11
Rule 6: IF x1 is (Medium 0.5) and x2 is (Medium 0.7) THEN y is (Medium 0.7), R6= 0.07,N6ex= 5
New rules:
Rule 7: IF x1 is (High 0.6) and x2 is (High 0.7) THEN y is (High 0.6), R7 = 0.2, N7ex = 12
Rule 8: IF x1 is (High 0.8) and x2 is (Medium 0.6) THEN y is (High 0.6), R8=0.1,N8ex= 5
Rule 9: IF x1 is (High 0.8) and x2 is (High 0.8) THEN y is (High3 0.8), R9= 0.1, N9ex = 6
March 20, 2007

68. NeuCom
Facilitates data analyses, data understanding, model creation and knowledge discovery
Data management and data ontology
Data analysis and feature extraction (statistical, PCA, clustering, SNR, …)
Data modeling and rule extraction (classification, prediction, optimisation)
Image recognition
Module integration
Free inspection copy from: or
March 20, 2007

69. Dynamic Evolving Neuro-Fuzzy System DENFIS for time series prediction , identification and control
Modeling, prediction and knowledge discovery from dynamic time series
Publication: Kasabov, N., and Song, Q., DENFIS: Dynamic Evolving Neural-Fuzzy Inference System and its Application for Time Series Prediction, IEEE Transactions on Fuzzy Systems, 2002, April
March 20, 2007

70. A Neuro-Fuzzy Approach as Medical Diagnostic Interface R. Brause, F
A Neuro-Fuzzy Approach as Medical Diagnostic Interface R. Brause, F. Friedrich J.W.Goethe-University, Frankfurt a. M., Germany
March 20, 2007

71. Hybrid view of the proposed model
A fuzzy agent to input vague parameters into multi-layer connectionist expert system: An application for stock market Priti Srinivas Sajja P1 P2 P3 P4 P5 P6 P7 P8
… … … … …..
Parameters Parameters Input Hidden Output
(fuzzy) (normalised) layer layer(s) layer …… ∑
Output
Fuzzy Agent
fP1
fP2
fP3
fP4
fP5
fP6
fP7
fP8
Hybrid view of the proposed model
March 20, 2007

72. Thanks
March 20, 2007