1. Methodology of Simulations n CS/PY 399 Lecture Presentation # 19 n February 21, 2001 n Mount Union College

2. Simulating Systems (Ch. 2, Plunkett & Elman) n To study complex systems of interacting components, we can: n 1) Study real systems –can we do this without disturbing the system to the point that results are flawed? n 2) Build a model of the system and simulate actual system operation –does model include all important aspects of the actual system?

3. Advantages of Modeling and Simulation n correlation between model components and brain structure –we can include more relevant details than in a verbal description of system operation n non-linear nature of Neural Networks makes purely mathematical analysis difficult or impossible n surprising behavior can be uncovered –leads to new experimental hypotheses

4. Advantages of Modeling and Simulation n It is easier to dissect a model than a brain! –model will work after dissection n We can begin to discover why some behavior arises from connected neuron clusters by studying node activations

5. Simulation as Experiment n Several factors must be defined before we use the computer to simulate a network n Define problem or goal –purpose of the experiment –e.g., to see if a network can learn letter sequences

6. Simulation as Experiment n Formulate a hypothesis –e.g., a feed-forward network can learn letter sequences n Design a mechanism for testing the hypothesis –e.g., see if training data can train network with a global error rate < ε n Evaluate results of simulation

7. Decisions when Designing Connectionist Simulations n Task –what is the desired behavior (output) for the network –defined by the training environment input/output patterns in training set –network is taught by example does not learn rules; learns to recognize patterns is this what happens with humans? –e.g., multiplication tables

8. Decisions when Designing Connectionist Simulations n Stimulus Representation –our models will use a numerical representation of input stimuli –vector notation used for multiple inputs defines an n-dimensional space, for n inputs –output also numeric

9. Decisions when Designing Connectionist Simulations n Network Architecture –number and arrangement of nodes –one large, completely connected network? simplest to design –several sub-networks connected in some configuration? this is more like brain structure –feed-forward vs. recurrent connections –these decisions will powerfully affect the network’s capabilities

10. Completely-Connected Network

11. Layered Feedforward Network

12. Recurrent Network

13. Analyzing the Simulation after Training n Global Error –output produced vs. output desired averaged over all patterns –perfect learning: error = 0 –can be misleading, especially if there are relatively few patterns that give bad results e.g. Quicksort average vs. worst-case behavior –avg. time for 1 million names:  20,000,000 ops. –worst-case:  1,000,000,000,000 (1 trillion) ops.

14. Analyzing the Simulation after Training n Individual Pattern Error –looking at specific interesting patterns too many patterns, in general, to examine all of them –also, we may look at novel patterns (not part of the training set) spontaneous generalization, etc. –what has the network learned? rote information vs. generalized knowledge

15. Analyzing the Simulation after Training n Inspecting weights and internal network settings –to see how the network is arriving at conclusions –comparison of starting weights with final trained weight set is often illuminating –after training, it is often useful to examine specific hidden node activations for a selected input pattern what part of the network is triggered?

16. Methodology of Simulations n CS/PY 399 Lecture Presentation # 19 n February 21, 2001 n Mount Union College