Perception and Thought as Constraint Satisfaction Processes Jay McClelland Symsys 100 April 27, 2010

Perception as Constraint Satisfaction For the first figure, one tends to see it a nothing until the ‘solution’ emerges. –The pieces and the whole simultaneously support each other For the second, there are two alternative solutions, each involving a re- interpretation of every part of the larger whole. –Again the parts and the whole are mutually consistent

Ubiquity of the Constraint Satisfaction Problem In sentence processing –I saw the grand canyon flying to New York –I saw the sheep grazing in the field In comprehension –Margie was sitting on the front steps when she heard the familiar jingle of the “Good Humor” truck. She remembered her birthday money and ran into the house. In reaching, grasping, typing…

Puzzles and Problems that Can Involve Constraint Satisfaction Find a word that you can combine with each of the next three words to make a compound word: Pine, crab, tree Make a single word from the phrase below: A rope ends it The two string problem: –You are in an office. There is a desk with a printer, some paper, a stapler, binder clips, pens and pencils. Two strings are hanging from a ceiling, about 12 feet apart in a room with a workbench. You can’t reach the second one while holding the first. –What can you do to bring them both together? You know how much gold weighs per cubic centimeter, and you want to test whether the king’s golden crown is pure gold. But you don’t know how many cc’s of gold are in the crown. How can you find out? ‘Seeing’ Mate in 3(?)

How Geniuses Think 1 Barwise and Etchemendy, two Stanford logicians, agree –they believe that nearly all great discoveries by logicians and mathematicians arise through imagery –after they make a discovery, they then attempt to verify it formally they attempt to develop a proof for their insight whereas the insight may have happened quickly, the proof may take years to develop –once a proof has been established, it is typically reported in a publication that says nothing about the imagery that led to the discovery 1. Slide from L. Barsalou, Stanford Psychology Ph.D.

Finding Perceptual Solutions It appears that our brains can search for alternative solutions until one pops out. How are such solutions found? –One answer is that the process occurs through a noisy, interactive activation process.

The interactive Activation Model: a Gradual Mutual Constraint Satisfaction Process Units represent hypotheses about the visual input at several levels and positions. –Features –Letters –Words Connections code contingent relations: –Excitatory connections for consistent relations –Inhibitory connections for inconsistent relations –Lateral inhibition for competition among mutually inconsistent possibilities within levels. Connections run in both directions –So that the network tends to evolve toward a state of activation in which everything is consistent.

Interactive Activation Simultaneously Identifies Words and Letters Stimulus input comes first to letter level, but as it builds up, it starts to influence the word level. Letter input from all four positions makes work the most active word unit (there is no word worr). Although the bottom up input to the letter level supports K and R equally in the fourth letter position, feedback from the word level supports K, causing it to become more active, and lateral inhibition then suppresses activation of R.

Goodness and Constraint Satisfaction Consider a network with symmetric connections, i.e. for all pairs of units i, j: w ij = w ji Provide external input e i to some of the units. Define the Goodness of a state of the network as: G(s) =  i>j w ij a i a j +  i e i a i As the network settles it tends toward states of higher goodness Examples: –Three-unit network –Necker Cube Network –Interactive activation network Noise in the settling process allows networks to jump out of local Goodness maxima. If we gradually reduce the noisiness, we can guarantee finding the best solution.

The Relationship Between Goodness and Probability in the Boltzmann Machine A ‘Boltzmann Machine’ is a stochastic neural network in which units’ activations are set to 0 or 1 with probability: P(a i = 1) = e net i /T /(1+e net i /T ) When T is large, the Boltzmann machine can jump from state to state easily. If one gradually reduces T, one can reach an equilibrium in which the probability of being in a particular state is given by: P(s) = e G(s)/T /(  s’ e G(s’)/T ) Here s indexes one state and s’ ranges over all possible states

More on Goodness and Probability If the weights, biases, and external inputs are set appropriately, then the quantity e G(s)/T /(  s’ e G(s’)/T ) represents the posterior probability that the state corresponds to the correct interpretation of the input. It follows that: –When T = 1 the probability of being in a state matches the probability that the state is the correct one. –When T = 0, only the best state (or the N equally good states) are possible In short, neural networks are capable of achieving optimal perceptual inference over entire ensembles of hypotheses, if the weights, biases, and external inputs are set appropriately.

Interactivity in the Brain Bidirectional Connectivity Interactions between MT and ‘lower’ visual areas Subjective Contours in V1 Distributed Constraint Satisfaction in Binocular Rivalry

Effect of Cooling MT on neural activation in lower visual areas Investigated effects of cooling MT on neuronal responses in V1, V2, and V3 to a bar on a background grid of lower contrast. MT cooling typically produces a reversible reduction in firing rate to V1/V2/V3 cells’ preferred stimulus (figure). Top down effect is greatest for stimuli of low contrast. If the stimulus is easy to see, top- down influence from MT has little effect. Response decrease due to cooling in MT

Lee & Nguyen (PNAS, 2001, 98, ) They asked the question: Do V1 neurons participate in the formation of a representation of the illusory contour seen in the upper panel (but not in the lower panel)? They recorded from neurons in V1 tuned to the illusory line segment, and varied the position of the illusory segment with respect to the most responsive position of the neuron.

Response to the illusory contour is found at precisely the expected location.

Temporal Response to Real and Illusory Contours Neuron’s receptive field falls right over the middle of the real or illusory line defining the bottom edge of the square

The patterns seen in the physiology are comparable to those seen in the interactive activation model in that the effect of direct input is manifest first, followed somewhat later by contextual influences, presumably mediated in the physiology by neurons sensitive to the overall configuration of display elements. direct context

Distributed Alternation of Brain Activity in Binocular Rivalry

Models of Binocular Rivalry Binocular rivalry can be explained by constraint satisfaction models in which neurons and/or connections that are active gradually become fatigued. As a result the state that happens to dominate initially eventually weakens, and is replaced by the alternative state. This state eventually weakens, and the other neurons/connections recover in the meantime, so that the first state will re-activate as the second one weakens. And the process continues…