1. A Comparison of Rule-Based versus Exemplar-Based Categorization Using the ACT-R Architecture Matthew F. RUTLEDGE-TAYLOR, Christian LEBIERE, Robert THOMSON, James STASZEWSKI and John R. ANDERSON Carnegie Mellon University, Pittsburgh, PA, USA 19th Annual ACT-R Workshop: Pittsburgh, PA, USA

2. Overview Categorization theories Facility Identification Task – Study examples of four different facilities – Categorize unseen facilities ACT-R Models – Rule-based versus Exemplar-based – Three different varieties of each based on information attended Model Results – Rule-based models are equivalent to exemplar-based models in terms of hit-rate performance Discussion

3. Categorization theories Rule-based theories (Goodman, Tenenbaum, Feldman & Griffiths, 2008) – Exceptions, e.g. RULEX (Nosofsky & Palmeri, 1995) – Probabilistic membership (Goodman et al., 2008) Prototype theories (Rosch, 1973) – Multiple prototype theories Exemplar theories (Nosofsky, 1986) – WTA vs weighted similarity ACT-R has been used previously to compare and contrast exemplar-based and rule-based approaches to categorization (Anderson & Betz, 2001)

4. Facility Identification Task Building (IMINT) Hardware MASINT1 MASINT2 SIGINT Notional Simulated Imagery Four kinds of facilities Probabilistic feature composition

5. Facility Identification Task Probabilistic occurrences of features Facility AFacility BFacility CFacility D Building1HighMidHighMid Building2HighMidHigh Building3HighMid High Building4High Mid Building5LowHighMidHigh Building6LowHigh Building7LowHigh Mid Building8LowMid High MASINT1FewManyFewMany MASINT2FewMany Few SIGINTManyFewManyFew HardwareFew Many

6. Three comparisons Human data versus model data – Hit-rate accuracy Exemplar model versus rule-based model – Blended retrieval of facility chunk, VS – Retrieval of one or more rules that manipulate a probability distribution Cognitive phenotypes: versions of both exemplar and rule-based models that attend to different data – Feature counts – Buildings that are present – Both

7. Three participant phenotypes Phenotype #1: Assumes buildings are key – Attentive to specific buildings in the image – Ignores the MASINT, SIGINT, and Hardware Phenotype #2: Assumes the numbers of each feature type is key – Attentive to counts of each facility feature – Ignores the types of buildings (just counts them) Phenotype #3: Attends to both specific buildings and feature counts

8. Facility Identification Phenotype #1 Specific Buildings only: SA model Building #2 Building #3 Building #6 Building #7 2 3 6 7

9. Facility Identification Phenotype #2 Feature type counts only: PM model Buildings 4 Hardware 1 MASINT1 6 MASINT2 2 SIGINT 5

10. Facility Identification Phenotype #3 SA and PM Building #2 Building #3 Building #6 Building #7 Hardware 1 MASINT1 6 MASINT2 2 SIGINT 5 2 3 6 7

11. ACT-R Exemplar based model Implicit statistical learning – Commits tokens of facilities to declarative memory Slots for facility type (A, B, C or D) Slots for sums of each feature type Slot for presence (or absence) of each building (IMINT) Categorization – Retrieval request made to DM based on facility features in target – Category slot values of retrieved chunk is used as categorization decision of the model

12. Facility chunk

13. ACT-R: Chunk activation A i = B i + S i + P i + Ɛ i A i is the net activation, B i is the base-level activation, S i is the effect of spreading activation, P i is the effect of the partial matching mismatch penalty, and Ɛ i is magnitude of activation noise.

14. Spreading Activation All values in all included buffers, spread activation to DM All facility features stored held in the visual buffer spread activation to all chunks in DM Primary retrieval factor for phenotype #1 (buildings)

15. Spreading Activation Visual Buffer Facility Chunk Declarative Memory Facility Chunk b1 nil b2 building2 b3 building3 b4 nil b5 nil b6 building6 b7 building7 category d b1 nil b2 building2 b3 nil b4 nil b5 nil b6 building6 b7 building7 category d b1 nil b2 building2 b3 building3 b4 building4 b5 nil b6 building6 b7 nil category a b1 building1 b2 nil b3 building3 b4 nil b5 building5 b6 nil b7 nil category d

16. Partial Matching The partial match is on a slot by slot basis For each chunk in DM, the degree to which each slot mismatches the corresponding slot in the retrieval cue determines the mismatch penalty Primary retrieval factor for phenotype #2 (counts)

17. Partial Matching Retrieval Buffer Facility Chunk Declarative Memory Facility Chunk buildings b4 Masint1 m6 Masint2 n2 Sigints s5 hardware h1 buildings b4 Masint1 m7 Masint2 n0 Sigints s7 hardware h2 buildings b5 Masint1 m4 Masint2 n1 Sigints s5 hardware h2 buildings b5 Masint1 m1 Masint2 n8 Sigints s5 hardware h0 category d category c Dissimilar values = high penalty Similar values = low penalty Equal values = no penalty category d

18. Heat Map on Counts of Features

19. Results of Exemplar Based Model PM only – 0.462 SA only – 0.665 PM + SA – 0.720 Human Participant Accuracy: 0.535 – Performance and interviews suggests Mix of phenotypes, with #2 (PM-like) most prevalent Employment of some explicit rules

20. ACT-R Rule Based Model Applied a set of rules to the unidentified target facility Accumulated a net probability distribution over the four possible facility categories Facility with greatest probability is the forced choice category response by the model

21. ACT-R Rule Based Model Two kinds of rules – SA-like: applies to presence of buildings – PM-like: applies to feature counts Rules implemented as chunks in DM Sets of dedicated productions for retrieving relevant rules High confidence in choice of rules – Based on analysis of probabilities of features

22. ACT-R Rule Based Model Example building rule - Ifis present then facility A is 1.38 times more likely (than if not present) Example count rule - If there are 5 MASINT1 then facility A is 3 times more likely (than if more or less) - Note: Count rules apply if count total in target is within a threshold difference of number in rule

23. Rule chunks

24. ACT-R Rule Based Model Three versions of the rules based model – Only apply building rules: similar to SA exemplar model – Only apply count rules: similar to PA exemplar model – Apply both building and count rules: similar to combined exemplar model

25. ACT-R Rule Based Model Results Building rules only: 0.657 Count rules only: 0.476 Both building and count rules: 0.755 StrategyRule-basedExemplar% Difference SA / Buildings0.6570.6550.30 PM / Counts0.4760.4622.94 Combined0.7550.7204.64

26. Discussion Agreement between rule-based and exemplar models, implemented in ACT-R, supports the equivalence of these approaches – They exploit the same available information The performance equivalence between the two establishes that functional Bayesian inferencing can be accomplished in ACT-R either through: – explicit, rule application – implicit, subsymbolic processes of the activation calculus, that support the exemplar model ACT-R learning mechanisms of the subsymbolic system in ACT-R is Bayesian in nature (Anderson, 1990; 1993) Blending allows ACT-R to implement importance sampling (Shi, et al., 2010)

27. Acknowledgements This work is supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of the Interior (DOI) contract number D10PC20021. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained hereon are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI, or the U.S. Government.

28. Blended Retrieval Standard retrieval – One previously existing chunk is retrieved Effectively, WTA closest exemplar Blending – One new chunk which is a blend of matching chunks is retrieved (created) – All slots not specified in the retrieval cue are assigned blended values – The contribution each exemplar chunk makes to blended slot values is proportional to the activation of the chunk