© Negnevitsky, Pearson Education, 2005 1 Lecture 10 Introduction Introduction Neural expert systems Neural expert systems Evolutionary neural networks.

© Negnevitsky, Pearson Education, 2005 1 Lecture 10 Introduction Introduction Neural expert systems Neural expert systems Evolutionary neural networks Evolutionary neural networks Fuzzy evolutionary systems Fuzzy evolutionary systems Summary Summary Hybrid intelligent systems:

© Negnevitsky, Pearson Education, 2005 2 Introduction A hybrid intelligent system is one that combines at least two intelligent technologies. For example, combining a fuzzy system with evolutionary computation. A hybrid intelligent system is one that combines at least two intelligent technologies. For example, combining a fuzzy system with evolutionary computation. The combination of probabilistic reasoning, fuzzy logic, neural networks and evolutionary computation forms the core of soft computing, an emerging approach to building hybrid intelligent systems capable of reasoning and learning in an uncertain and imprecise environment. One character of soft computing is the using of words but not numbers in systems. The combination of probabilistic reasoning, fuzzy logic, neural networks and evolutionary computation forms the core of soft computing, an emerging approach to building hybrid intelligent systems capable of reasoning and learning in an uncertain and imprecise environment. One character of soft computing is the using of words but not numbers in systems.

© Negnevitsky, Pearson Education, 2005 3 Although words are less precise than numbers, precision carries a high cost. We use words when there is a tolerance for imprecision. Soft computing exploits the tolerance for uncertainty and imprecision to achieve greater tractability and robustness, and lower the cost of solutions. Although words are less precise than numbers, precision carries a high cost. We use words when there is a tolerance for imprecision. Soft computing exploits the tolerance for uncertainty and imprecision to achieve greater tractability and robustness, and lower the cost of solutions. We also use words when the available data is not precise enough to use numbers. This is often the case with complex problems, and while “hard” computing fails to produce any solution, soft computing is still capable of finding good solutions. We also use words when the available data is not precise enough to use numbers. This is often the case with complex problems, and while “hard” computing fails to produce any solution, soft computing is still capable of finding good solutions.

© Negnevitsky, Pearson Education, 2005 4 Lotfi Zadeh is reputed to have said that a good hybrid would be “British Police, German Mechanics, French Cuisine, Swiss Banking and Italian Love”. But “British Cuisine, German Police, French Mechanics, Italian Banking and Swiss Love” would be a bad one. Likewise, a hybrid intelligent system can be good or bad – it depends on which components constitute the hybrid. So our goal is to select the right components for building a good hybrid system. Lotfi Zadeh is reputed to have said that a good hybrid would be “British Police, German Mechanics, French Cuisine, Swiss Banking and Italian Love”. But “British Cuisine, German Police, French Mechanics, Italian Banking and Swiss Love” would be a bad one. Likewise, a hybrid intelligent system can be good or bad – it depends on which components constitute the hybrid. So our goal is to select the right components for building a good hybrid system.

© Negnevitsky, Pearson Education, 2005 6 Let’s compare neural networks with expert systems: Let’s compare neural networks with expert systems: Expert systems rely on logical inferences and decision trees and focus on modelling human reasoning. Neural networks rely on parallel data processing and focus on modelling a human brain. Expert systems rely on logical inferences and decision trees and focus on modelling human reasoning. Neural networks rely on parallel data processing and focus on modelling a human brain. Expert systems treat the brain as a black-box. Neural networks look at its structure and functions, particularly at its ability to learn. Expert systems treat the brain as a black-box. Neural networks look at its structure and functions, particularly at its ability to learn. Knowledge in a rule-based expert system is represented by IF-THEN production rules. Knowledge in neural networks is stored as synaptic weights between neurons. Knowledge in a rule-based expert system is represented by IF-THEN production rules. Knowledge in neural networks is stored as synaptic weights between neurons. Neural expert systems

© Negnevitsky, Pearson Education, 2005 7 In expert systems, knowledge can be divided into individual rules and the user can see and understand the piece of knowledge applied by the system. In expert systems, knowledge can be divided into individual rules and the user can see and understand the piece of knowledge applied by the system. In neural networks, one cannot select a single synaptic weight as a discrete piece of knowledge. Here knowledge is embedded in the entire network; it cannot be broken into individual pieces, and any change of a synaptic weight may lead to unpredictable results. A neural network is, in fact, a black-box for its user. In neural networks, one cannot select a single synaptic weight as a discrete piece of knowledge. Here knowledge is embedded in the entire network; it cannot be broken into individual pieces, and any change of a synaptic weight may lead to unpredictable results. A neural network is, in fact, a black-box for its user.

© Negnevitsky, Pearson Education, 2005 8 Can we combine advantages of expert systems and neural networks to create a more powerful and effective expert system? A hybrid system that combines a neural network and a rule-based expert system is called a neural expert system (or a connectionist expert system).

© Negnevitsky, Pearson Education, 2005 10 The heart of a neural expert system is the inference engine. It controls the information flow in the system and initiates inference over the neural knowledge base. Based on the neural knowledge base (a special neural network), inference derive the outputs based on the inputs. The most important thing about neural expert system is that it can derive the rules from the neural network. That’s why the it is called a neural expert system but not a neural network

© Negnevitsky, Pearson Education, 2005 12 If we set each input of the input layer to either +1 (true),  1 (false), or 0 (unknown), we can give a semantic interpretation for the activation of any output neuron. For example, we can have following IF-THEN rule: if the object has Wings (+1), Beak (+1) and Feathers (+1), but does not have Engine (  1), then we can conclude that this object is Bird (+1):

© Negnevitsky, Pearson Education, 2005 14 By attaching a corresponding question to each input neuron, we can enable the system to prompt the user for initial values of the input variables: Neuron: Wings Question: Does the object have wings? Neuron: Tail Question: Does the object have a tail? Neuron: Beak Question: Does the object have a beak? Neuron: Feathers Question: Does the object have feathers? Neuron: Engine Question: Does the object have an engine? Based on these inputs, we can determine the type of objects.

© Negnevitsky, Pearson Education, 2005 15 More interestingly, we do not need all the answers of questions to make the decision. An inference can be made if the known net weighted input to a neuron is greater than the sum of the absolute values of the weights of the unknown inputs. where i  known, j  known and n is the number of neuron inputs.

© Negnevitsky, Pearson Education, 2005 16 Enter initial value for the input Feathers:  +1 Example: KNOWN = 1  2.8 = 2.8 UNKNOWN =  0.8  +  0.2  +  2.2  +  1.1  = 4.3 KNOWN  UNKNOWN KNOWN = 1  2.8 + 1  2.2 = 5.0 UNKNOWN =  0.8  +  0.2  +  1.1  = 2.1 KNOWN  UNKNOWN Enter initial value for the input Beak:  +1 CONCLUDE: Bird is TRUE

© Negnevitsky, Pearson Education, 2005 18 Although neural networks are used for solving a variety of problems, they still have some limitations. Although neural networks are used for solving a variety of problems, they still have some limitations. One of the most common is associated with neural network training. The back-propagation learning algorithm cannot guarantee an optimal solution. In real-world applications, the back-propagation algorithm might converge to a set of sub-optimal weights from which it cannot escape. As a result, the neural network is often unable to find a desirable solution to a problem at hand. One of the most common is associated with neural network training. The back-propagation learning algorithm cannot guarantee an optimal solution. In real-world applications, the back-propagation algorithm might converge to a set of sub-optimal weights from which it cannot escape. As a result, the neural network is often unable to find a desirable solution to a problem at hand. Evolutionary neural networks

© Negnevitsky, Pearson Education, 2005 19 Another difficulty is related to selecting an optimal topology for the neural network. The “right” network architecture for a particular problem is often chosen by means of heuristics, and designing a neural network topology is still more art than engineering. Another difficulty is related to selecting an optimal topology for the neural network. The “right” network architecture for a particular problem is often chosen by means of heuristics, and designing a neural network topology is still more art than engineering. Genetic algorithms are an effective optimisation technique that can guide both weight optimisation and topology selection. Genetic algorithms are an effective optimisation technique that can guide both weight optimisation and topology selection.

© Negnevitsky, Pearson Education, 2005 21 The second step is to define a fitness function for evaluating the chromosome’s performance. This function must estimate the performance of a given neural network. We can apply here a simple function defined by the sum of squared errors. The second step is to define a fitness function for evaluating the chromosome’s performance. This function must estimate the performance of a given neural network. We can apply here a simple function defined by the sum of squared errors. The training set of examples is presented to the network, and the sum of squared errors is calculated. The smaller the sum, the fitter the chromosome. The genetic algorithm attempts to find a set of weights that minimises the sum of squared errors. The training set of examples is presented to the network, and the sum of squared errors is calculated. The smaller the sum, the fitter the chromosome. The genetic algorithm attempts to find a set of weights that minimises the sum of squared errors.

© Negnevitsky, Pearson Education, 2005 22 The third step is to choose the genetic operators – crossover and mutation. A crossover operator takes two parent chromosomes and creates a single child with genetic material from both parents. Each gene in the child’s chromosome is represented by the corresponding gene of the randomly selected parent. The third step is to choose the genetic operators – crossover and mutation. A crossover operator takes two parent chromosomes and creates a single child with genetic material from both parents. Each gene in the child’s chromosome is represented by the corresponding gene of the randomly selected parent. A mutation operator selects a gene in a chromosome and adds a small random value between –1 and 1 to each weight in this gene. A mutation operator selects a gene in a chromosome and adds a small random value between –1 and 1 to each weight in this gene.

© Negnevitsky, Pearson Education, 2005 25 Can genetic algorithms help us in selecting the network architecture? The architecture of the network (i.e. the number of neurons and their interconnections) often determines the success or failure of the application. Usually the network architecture is decided by trial and error; there is a great need for a method of automatically designing the architecture for a particular application. Genetic algorithms may well be suited for this task.

© Negnevitsky, Pearson Education, 2005 26 The basic idea behind evolving a suitable network architecture is to conduct a genetic search in a population of possible architectures. The basic idea behind evolving a suitable network architecture is to conduct a genetic search in a population of possible architectures. We must first choose a method of encoding a network’s architecture into a chromosome. We must first choose a method of encoding a network’s architecture into a chromosome.

© Negnevitsky, Pearson Education, 2005 27 The connection topology of a neural network can be represented by a square connectivity matrix. The connection topology of a neural network can be represented by a square connectivity matrix. Each entry in the matrix defines the type of connection from one neuron (column) to another (row), where 0 means no connection and 1 denotes connection for which the weight can be changed through learning. Each entry in the matrix defines the type of connection from one neuron (column) to another (row), where 0 means no connection and 1 denotes connection for which the weight can be changed through learning. To transform the connectivity matrix into a chromosome, we need only to string the rows of the matrix together. To transform the connectivity matrix into a chromosome, we need only to string the rows of the matrix together. Encoding the network architecture

© Negnevitsky, Pearson Education, 2005 30 Evolutionary computation is also used in the design of fuzzy systems, particularly for generating fuzzy rules and adjusting membership functions of fuzzy sets. Evolutionary computation is also used in the design of fuzzy systems, particularly for generating fuzzy rules and adjusting membership functions of fuzzy sets. In this section, we introduce an application of genetic algorithms to select an appropriate set of fuzzy IF-THEN rules for a classification problem. In this section, we introduce an application of genetic algorithms to select an appropriate set of fuzzy IF-THEN rules for a classification problem. For a classification problem, a set of fuzzy IF-THEN rules is generated from numerical data. For a classification problem, a set of fuzzy IF-THEN rules is generated from numerical data. First, we use a grid-type fuzzy partition of an input space. First, we use a grid-type fuzzy partition of an input space. Fuzzy evolutionary systems

© Negnevitsky, Pearson Education, 2005 32 Fuzzy partition Black and white dots denote the training patterns of Class 1 and Class 2, respectively. Black and white dots denote the training patterns of Class 1 and Class 2, respectively. The grid-type fuzzy partition can be seen as a rule table. The grid-type fuzzy partition can be seen as a rule table. The linguistic values of input x1 (A 1, A 2 and A 3 ) form the horizontal axis, and the linguistic values of input x2 (B 1, B 2 and B 3 ) form the vertical axis. The linguistic values of input x1 (A 1, A 2 and A 3 ) form the horizontal axis, and the linguistic values of input x2 (B 1, B 2 and B 3 ) form the vertical axis. At the intersection of a row and a column lies the rule consequent. At the intersection of a row and a column lies the rule consequent.

© Negnevitsky, Pearson Education, 2005 33 In the rule table, each fuzzy subspace can have only one fuzzy IF-THEN rule, and thus the total number of rules that can be generated in a K  K grid is equal to K  K.

© Negnevitsky, Pearson Education, 2005 34 Fuzzy rules that correspond to the K  K fuzzy partition can be represented in a general form as: where x p is a training pattern on input space X1  X2, P is the total number of training patterns, C n is the rule consequent (either Class 1 or Class 2), and is the certainty factor that a pattern in fuzzy subspace A i B j belongs to class C n. CF A i B j CnCn

© Negnevitsky, Pearson Education, 2005 35 To determine the rule consequent and the certainty factor, we use the following procedure: To determine the rule consequent and the certainty factor, we use the following procedure: Step 1: Partition an input space into K  K fuzzy subspaces, and calculate the strength of each class of training patterns in every fuzzy subspace. Each class in a given fuzzy subspace is represented by its training patterns. The more training patterns, the stronger the class  in a given fuzzy subspace, the rule consequent becomes more certain when patterns of one particular class appear more often than patterns of any other class. Each class in a given fuzzy subspace is represented by its training patterns. The more training patterns, the stronger the class  in a given fuzzy subspace, the rule consequent becomes more certain when patterns of one particular class appear more often than patterns of any other class.

© Negnevitsky, Pearson Education, 2005 37 Step 2: Determine the rule consequent and the certainty factor in each fuzzy subspace. As the rule consequent is determined by the strongest class we need to find class C m such that,

© Negnevitsky, Pearson Education, 2005 39 The certainty factor can be interpreted as follows: If all the training patterns in fuzzy subspace A i B j belong to the same class, then the certainty factor is maximum and it is certain that any new pattern in this subspace will belong to this class. If all the training patterns in fuzzy subspace A i B j belong to the same class, then the certainty factor is maximum and it is certain that any new pattern in this subspace will belong to this class. If, however, training patterns belong to different classes and these classes have similar strengths, then the certainty factor is minimum and it is uncertain that a new pattern will belong to any particular class. If, however, training patterns belong to different classes and these classes have similar strengths, then the certainty factor is minimum and it is uncertain that a new pattern will belong to any particular class.

© Negnevitsky, Pearson Education, 2005 40 This means that patterns in a fuzzy subspace can be misclassified. Moreover, if a fuzzy subspace does not have any training patterns, we cannot determine the rule consequent at all. This means that patterns in a fuzzy subspace can be misclassified. Moreover, if a fuzzy subspace does not have any training patterns, we cannot determine the rule consequent at all. If a fuzzy partition is too coarse, many patterns may be misclassified. On the other hand, if a fuzzy partition is too fine, many fuzzy rules cannot be obtained, because of the lack of training patterns in the corresponding fuzzy subspaces. If a fuzzy partition is too coarse, many patterns may be misclassified. On the other hand, if a fuzzy partition is too fine, many fuzzy rules cannot be obtained, because of the lack of training patterns in the corresponding fuzzy subspaces.

© Negnevitsky, Pearson Education, 2005 41 Training patterns are not necessarily distributed evenly in the input space. As a result, it is often difficult to choose an appropriate density for the fuzzy grid. To overcome this difficulty, we use multiple fuzzy rule tables.

© Negnevitsky, Pearson Education, 2005 42 Multiple fuzzy rule tables Fuzzy IF-THEN rules are generated for each fuzzy subspace of multiple fuzzy rule tables, and thus a complete set of rules for our case can be specified as: 2 2 + 3 3 + 4 4 + 5 5 + 6 6 = 90 rules.

© Negnevitsky, Pearson Education, 2005 43 Once the set of rules S ALL is generated, a new pattern, x = (x1, x2), can be classified by the following procedure: Step 1: In every fuzzy subspace of the multiple fuzzy rule tables, calculate the degree of compatibility of a new pattern with each class.

© Negnevitsky, Pearson Education, 2005 46 The number of multiple fuzzy rule tables required for an accurate pattern classification may be large. The number of multiple fuzzy rule tables required for an accurate pattern classification may be large. Consequently, a complete set of rules can be enormous. Consequently, a complete set of rules can be enormous. Meanwhile, these rules have different classification abilities, and thus by selecting only rules with high potential for accurate classification, we reduce the number of rules. Meanwhile, these rules have different classification abilities, and thus by selecting only rules with high potential for accurate classification, we reduce the number of rules.

© Negnevitsky, Pearson Education, 2005 47 The problem of selecting fuzzy IF-THEN rules can be seen as a combinatorial optimisation problem with two objectives. The problem of selecting fuzzy IF-THEN rules can be seen as a combinatorial optimisation problem with two objectives. The first, more important, objective is to maximise the number of correctly classified patterns. The first, more important, objective is to maximise the number of correctly classified patterns. The second objective is to minimise the number of rules. The second objective is to minimise the number of rules. Genetic algorithms can be applied to this problem. Genetic algorithms can be applied to this problem. Can we use genetic algorithms for selecting fuzzy IF-THEN rules ?

© Negnevitsky, Pearson Education, 2005 48 A basic genetic algorithm for selecting fuzzy IF- THEN rules includes the following steps: A basic genetic algorithm for selecting fuzzy IF- THEN rules includes the following steps: Step 1: Randomly generate an initial population of chromosomes. The population size may be relatively small, say 10 or 20 chromosomes. Each gene in a chromosome corresponds to a particular fuzzy IF-THEN rule in the rule set defined by S ALL. e.g. the chromosome can be specified by a 90-bit string. Each bit can assume 1, -1, 0. e.g. the chromosome can be specified by a 90-bit string. Each bit can assume 1, -1, 0. 1 for a particular rule belongs to set S, -1 for not belonging, and 0 for dummy rule. 1 for a particular rule belongs to set S, -1 for not belonging, and 0 for dummy rule. Step 2: Calculate the performance, or fitness, of each individual chromosome in the current population.

© Negnevitsky, Pearson Education, 2005 49 The problem of selecting fuzzy rules has two objectives: to maximise the accuracy of the pattern classification and to minimise the size of a rule set. The problem of selecting fuzzy rules has two objectives: to maximise the accuracy of the pattern classification and to minimise the size of a rule set. The fitness function has to accommodate both these objectives. This can be achieved by introducing two respective weights, w P and w N, in the fitness function: The fitness function has to accommodate both these objectives. This can be achieved by introducing two respective weights, w P and w N, in the fitness function: where P S the number of patterns classified successfully, P ALL is the total number of patterns presented to the classification system, N S and N ALL are the numbers of fuzzy IF-THEN rules in set S and set S ALL, respectively.

© Negnevitsky, Pearson Education, 2005 51 Step 3: Select a pair of chromosomes for mating. Parent chromosomes are selected with a probability associated with their fitness; a better fit chromosome has a higher probability of being selected. Step 4: Create a pair of offspring chromosomes by applying a standard crossover operator. Parent chromosomes are crossed at the randomly selected crossover point. Step 5: Perform mutation on each gene of the created offspring. The mutation probability is normally kept quite low, say 0.01. The mutation is done by multiplying the gene value by –1.

© Negnevitsky, Pearson Education, 2005 52 Step 6: Place the created offspring chromosomes in the new population. Step 7: Repeat Step 3 until the size of the new population becomes equal to the size of the initial population, and then replace the initial (parent) population with the new (offspring) population. Step 9: Go to Step 2, and repeat the process until a specified number of generations (typically several hundreds) is considered. The number of rules can be cut down to less than 2% of the initially generated set of rules.

© Negnevitsky, Pearson Education, 2005 1 Lecture 10 Introduction Introduction Neural expert systems Neural expert systems Evolutionary neural networks.

Similar presentations

Presentation on theme: "© Negnevitsky, Pearson Education, 2005 1 Lecture 10 Introduction Introduction Neural expert systems Neural expert systems Evolutionary neural networks."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

© Negnevitsky, Pearson Education, 2005 1 Lecture 10 Introduction Introduction Neural expert systems Neural expert systems Evolutionary neural networks.

Similar presentations

Presentation on theme: "© Negnevitsky, Pearson Education, 2005 1 Lecture 10 Introduction Introduction Neural expert systems Neural expert systems Evolutionary neural networks."— Presentation transcript:

Similar presentations

About project

Feedback