1. 1 Abstract This study presents an analysis of two modified fuzzy ARTMAP neural networks. The modifications are first introduced mathematically. Then, the performance of the systems is studied on benchmark examples with noiseless data. It is shown that each modified ARTMAP system achieves classification accuracy superior to that of standard fuzzy ARTMAP, while retaining comparable complexity of the internal code. 1.In the first modified ARTMAP system, a graded choice-by- difference (CBD) signal function takes the choice signal T j to be dependent on the input position, even when the input lies within the category box R j. Namely, an input near the center of the box R j generates a larger signal T j than an input near the boundary of the box. In order to ensure that the same input would choose the same category if it were immediately re-presented (direct access), the ART match rule was also modified, to correspond to the new choice rule. The resulting graded signal function system creates more accurate decision boundaries, especially when these boundaries are not parallel to the input space axes.

2. 2 2.In the second modified ARTMAP system, all category boxes R j are point boxes. This simplified network learns with a fast- commit/no-recode rule, which does not allow any learning at node j once category j has been established. In addition, vigilance (  ) is set to zero, which eliminates the matching system. Each input that makes a predictive error creates a new category, which is encoded as the input itself. The classification accuracy obtained with this point-box system is better than that of the other studied systems. However, the point-box system has a potential drawback in that its memory requirements may be high for large databases. To alleviate this problem, a strategy for on-line elimination of redundant categories is proposed and evaluated. This strategy can be interpreted as a rule for on-line forgetting of certain stored memories. Its application leads to a significant reduction in memory requirements while retaining classification accuracy.

3. 3 ARTMAP Neural network for supervised learning. Output class Input Signal function Category choice b

4. 4 ART1: Weber law (1987) Fuzzy ARTMAP: Choice-by-difference (CBD, 1994) NEW: Graded signal function that improves behavior for certain types of data 1. Signal Functions

5. 5 (1994) CBD Signal Function DIAGONAL Circle-in-Square (CIS) 94.41 % correct 40.7 coding nodes 91.70 % correct 15.2 coding nodes SIMULATIONS: Average of 10 runs, 1 training epoch Training points: 1,000 (DIAG) or 10,000 (CIS), Testing points: 10,000

6. 6 NEW Graded Signal Function DIAGONALCircle-in-Square (CIS) 96.73 % correct (  =0.5) 46.4 coding nodes 95.26 % correct (  =1) 17.8 coding nodes Smoother boundaries Improved % correct but slightly more nodes

7. 7 Choice Signal with CBD and Graded Signal Function in Two-dimensional Input Space (1994) CBD signal function Signal constant for inputs within a   R j. Flat top (  =0) NEW Graded signal function Signal position-dependent for inputs within R j. Peak-height determined by parameter 

8. 8 Decision Boundaries Between Overlapping Category Boxes with CBD and Graded Signal Function R1R1 R2R2 CBD Graded

9. 9 Graded Signal Function Implementation: Match Input Match function Reset or Resonance ? ARTMAP design principle Repeated presentation of the same input should lead to a choice of correct node without search. Need to modify match function. Output class Input b TjTj

10. 10 2. Point ARTMAP Input Minimum algorithm with critical properties of Adaptive Resonance Theory Input a Labeled coding points Output class Point ARTMAP ARTMAP Output class b

11. 11 Point ARTMAP - Learning Cycle a Input a a Output class a … do nothing 1. New input presentation leads to a choice of closest coding point 2a. If chosen coding node matches the output class... 2b. If it does not...… store current input into memory a

12. 12 Point ARTMAP Circle-in-Square (CIS) 98.72 % correct 275 coding nodes DIAGONAL 97.8 % correct 59.2 coding nodes

13. 13 Point ARTMAP - Results Best performance Fastest learning When training stops at a given fuzzy ARTMAP accuracy, memory sizes are comparable Potential danger of many coding nodes HOW to restrict network size while assuring improvement of its performance as more patterns are presented? Possible solution: Compute continually for each node a measure of its usefulness/criticality and eliminate least useful nodes as needed.

14. 14 How to compute usefulness of nodes? Required general properties: –local, fast, simple computation –computed only for one (or a few) nodes per input presentation Criterion for acceptability of any usefulness rule: M N  number of training patterns from a given training set, in response to which the network reaches size N. Usefulness rule must assure that the network code learned in response to M>M N inputs is better than that for M N inputs. When to eliminate a node? Many possible choices, in this study “hard limit:” –Define the maximum network size N –Once network size N is reached, one of the existing nodes (the least useful one) is eliminated every time a new node is created Usefulness Rule

15. 15 Usefulness Rule - Definition General definition: Usefulness   error [if eliminated] Implemented rule: Usefulness updated every time node wins the competition and gives correct prediction (step 2a). Usefulness increases if the node is critical, i.e., if elimination of the current winner would lead to a predictive error. Usefulness decreases if the node is non-critical, i.e., the network would give a correct prediction even without it. Initial usefulness is zero. a

16. 16 Point ARTMAP with On-line Elimination Circle-in-Square (CIS) with network size frozen after 10,000 iterations (275 nodes)

17. 17 Development of Internal Code in Point ARTMAP with On-line Elimination 10,000 train. points 100,000 200,000 500,000 400,000 300,000 Learned code after presentation of training set of different sizes

18. 18 Point ARTMAP with On-Line Elimination - Discussion Properties: ability to improve coding with time without growing in size ability to correct a learned error ability to adapt in a non-stationary environment can be understood as rule for optimal forgetting In a noisy environment: without on-line elimination, the system will grow without limits current elimination rule can lead to incorrect behavior goal - find a rule that will secure optimum performance

19. 19 Summary Grdaded signal function is an extension of the CBD signal function that distinguishes between points within category boxes. Point ARTMAP is a minimum version of a system for supervised learning based on Adaptive Resonance Theory. It is fast and very simple, with tendency to proliferate stored categories. This deficiency can be alleviated by several local pruning rules. Both systems, especially Point ARTMAP, performed very well on benchmark problems with noiseless data. Additional testing on noisy data is needed.

20. 20 Appendix 1: Simulations with Diagonal data 00.20.40.60.81 80 84 88 92 96 100 % Correct predictions steepness parameter  100 training points 1000 training points 10000 training points 00.20.40.60.81 Point ARTMAP 0 20 40 60 80 Number of coding nodes 177.3 Point ARTMAP 100 training points 1000 training points 10000 training points fuzzy ARTMAP fuzzy ARTMAP Simulations of fuzzy ARTMAP with standard CBD (  ) and with graded CBD function (  ), and of Point ARTMAP on the diagonal benchmark problem

21. 21 Appendix 2: Simulations with Circle-in-the- square data steepness parameter  00.20.40.60.81 80 84 88 92 96 100 % Correct predictions 100 training points 1000 training points 10000 training points Point ARTMAP fuzzy ARTMAP 00.20.40.60.81 0 20 40 60 80 275 Point ARTMAP fuzzy ARTMAP Number of coding nodes 100 training points 1000 training points 10000 training points Simulations of fuzzy ARTMAP with standard CBD (  ) and with graded CBD function (  ), and of Point ARTMAP on the circle-in- the-square benchmark problem

22. 22 Fig 4: Choice signal for standard vs. graded CBD in one input dimension RjRj w 1,j 1-w 1,j TjTj Standard CBDGraded CBD Fig 5: Decision boundaries between two category boxes with standard and graded CBD R1R1 R2R2 Standard CBD Graded CBD

23. 23 ARTMAP Neural network for supervised learning. Input class Input Signal function Category choice

24. 24 Point ARTMAP Input class Input Minimum algorithm with critical properties of Adaptive Resonance Theory Input a Labeled coding points Input class Coding boxes reduced to coding points Point ARTMAP ARTMAP

25. 25 Point ARTMAP - Algorithm a Coding boxes reduced to coding points ARTMAP Input

26. 26 Usefulness Rule - Criteria Required properties of the usefulness rule local computation fast/simple computed only for one (or a few) nodes at a time Criterion for acceptability of the rule: Let the maximum network size be restricted by number N Let M N be the number of training patterns from a given training set, in response to which the network reaches size N Once maximum network size is reached, every time a new node needs to be created, the least useful one (based on proposed rule) is eliminated The proposed rule is acceptable if the network code learned in response to M>M N inputs is better than for M N inputs.