1. Discourse & Situational Context
Lectures I. Overview 2. Simulation Semantics 3. ECG and Best-fit Analysis 4. Compositionality 5. Simulation, Counterfactuals, and Inference
Discourse & Situational Context
Constructions
Utterance
Analyzer:
incremental,
competition-based, psychologically plausible
Semantic Specification:
image schemas, bindings, action schemas
Simulation

2. Evidence for Simulation Semantics
BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE
Evidence for common mechanisms for recognition and action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Boccino 2002) and from motor imagery (Jeannerod 1996)
IMPLEMENTATION:
x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network.
RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

3. Active representations
Many inferences about actions derive from what we know about executing them
X-net representation based on stochastic Petri nets captures dynamic, parameterized nature of actions
Used for acting, recognition, planning, and language
walker at goal
Walking: bound to a specific walker with a direction or goal
consumes resources (e.g., energy)
may have termination condition (e.g., walker at goal)
ongoing, iterative action
energy
Certain words are closely associated with biological phenomena.
Simple representation based on petri nets. Action/event representation has in common with motor control the need to refer to
process states and transitions;
and resource consumption/production; parameters.
Part of learning/understanding words like “push”, “walk” clearly involves grounded knowledge about how to perform the action, as well as quite complex/concrete inferences based on execution. (e.g....)
Note: these representations might be parameterized: “shove”, “walk slowly”, “walk home”
and NOTE: this model of action is accurate cross-linguistically, even if some specific conditions on word meaning varies from language to language.
This may seem complex, but in fact VERY early children seem to have no problem performing and understanding words like this. And more complicated ones too!
walker=Harry
goal=home

4. Task Interpret simple discourse fragments/blurbs
France fell into recession. Pulled out by Germany
Economy moving at the pace of a Clinton jog.
US Economy on the verge of falling back into recession after moving forward on an anemic recovery.
Indian Government stumbling in implementing Liberalization plan.
Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice.
The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.

5. Basic Result
A neurally plausible, computational model and an implementation that is able to cash out the observation that motion, manipulation and spatial concepts are used to convey important and subtle information about abstract domains such as International Economics.
In 1991, India set out on a path of Liberalization. After making rapid strides in the first few years, the Government policy hit a first of a series of roadblocks in By 1998, the new BJP Government had reoriented the Government’s policy ..

6. I/O as Feature Structures
Indian Government stumbling in implementing liberalization plan

7. Basic Primitives
An fine-grained executing model of action and events (X-schemas)
A factorized representation of state (DBN’s)
A model of metaphor maps that project bindings from source to target domains.

9. Features of Representation
Inherently action based, with fine grained distinctions in resource usage, and temporal evolutions.
Can deal with concurrent actions, durations, hierarchical action sets, and stochastic actions (selection and effects).
Highly responsive to a changing environment with uncertain evolutions.
Can model complex domain constraints in a factorized representation that can compute complex ramifications as well as prior beliefs and possible predictions.

10. The Target Domain Simple knowledge about Economics Key Requirement:
Factual (US is a market economy)
Correlational (High Growth => High Inflation)
Key Requirement:
Must combine background knowledge of economics with inherent structure and constraints of the target domain with inferential products of metaphoric (and other) projections from multiple source domains.
Must be able to compute the global impact of new observations (from direct input as well as metaphoric inferences)

11. Metaphor Maps
Static Structures that project bindings from source domain f- struct to target domain Belief net nodes by setting evidence on the target network.
Different types of maps
PMAPS project X- schema Parameters to abstract domains
OMAPS connect roles between source and target domain
SMAPS connect schemas from source to target domains.
ASPECT is an invariant in projection.

12. Results
Model was implemented and tested on discourse fragments from a database of 30 newspaper stories in international economics from standard sources such as WSJ, NYT, and the Economist. Results show that motion terms are often the most effective method to provide the following types of information about abstract plans and actions.
Information about uncertain events and dynamic changes in goals and resources. (sluggish, fall, off-track, no steam)
Information about evaluations of policies and economic actors and communicative intent (strangle-hold, bleed).
Communicating complex, context-sensitive and dynamic economic scenarios (stumble, slide, slippery slope).
Commincating complex event structure and aspectual information (on the verge of, sidestep, giant leap, small steps, ready, set out, back on track).
ALL THESE BINDINGS RESULT FROM REFLEX, AUTOMATIC INFERENCES PROVIDED BY X-SCHEMA BASED INFERENCES.

13. States are DBN
Dynamic Bayesian Networks (D(T)BNs) are an extension of Bayesian networks for modeling dynamic systems.
In a DBN, the state at time t is represented by a set of random variables. The state at time t is dependent on the states at previous time steps.
Typically, we assume that each state only depends on the immediately preceding state (first-order Markovian), and thus we need to represent the transition distribution P(Zt+1 | Zt).
This can be done using a two-time-slice Bayesian network fragment (2-TBN) Bt+1,
variables from Zt+1 whose parents are variables from Zt and/or Zt+1, and variables from Zt without any parents.
Typically, we also assume that the process is stationary, i.e., the transition models for all time slices are identical:

15. An Active Model of Events
Computationally, actions and events are coded in active representations called x-schemas which are extensions to Stochastic Petri nets.
x-schemas are fine-grained action and event representations that can be used for monitoring and control as well as for inference.

16. Preconditions, resources, fine control structure are important aspects of events

18. X-schema Extensions to Petri Nets

19. A Walk X-schema

20. Logical Action Theories
Connection to ARD (or other Action Languages):
The representation can be used to encode a causal model for a domain description D (in the Syntax of ARD) in that it satisfies all the causal laws in D. Furthermore, a value proposition of the form C after A is entailed by D iff all the terms in C are in Si; the state that results after running the projection algorithm on the action set A. (IJCAI 99)
Executing representation,
frame axioms are encoded in the topology of the network and transition firing rules respect them.
Planning as backward reachability or computing downward closure (IJCAI 99, WWW2002)
Links to linear logic. Perhaps a model of stochastic linear logic? (SRI CSL TR 2001).

21. Event Structure in Language
Fine-grained
Rich Notion of Contingency Relationships.
Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals
Event Structure Metaphor:
Phenomena: Abstract Actions are conceptualized in Motion and Manipulation terms. Invariants in projection.

22. A Climb X-schema

23. Common Patterns Posture = Up Energy Available Ready Dest = top(obj)
START
FINISH
Posture = Up
Energy Available
Ready
Dest = top(obj)
Loop
BEGIN
Execute(subschema)
END
At Dest
Done
Ground ok

25. A Schema Controller iterate Ready Start Process Finish Done interrupt
resume
Cancel
Suspend
An active controller that sends signals to the embedded schema and transitions based on signals from the embedded schema.
Useful for higher level coordination of actions.

26. A Generic Process Schema
iterate
Ready
Start
Process
Finish
Done
interrupt
resume
Cancel
Suspend
Part of Conceptual Structure.
Generalizes over actions and events. Has internal state and models evolution of processes.

27. Aspects of (Climb) Iterate Ready Start Process Finish Done resume
interrupt
Suspend
Cancel
BINDINGS
Energy
Ready
Standing
Hold
Find hold
Pull(self)
Stabilize
On top

28. About to + (Climb) (Prospective)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Suspend
Cancel
BINDINGS
Energy
Ready
Standing
Hold
Find hold
Pull(self)
Stabilize
On top

29. Be + (Climb)-ING (Progressive)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Suspend
Cancel
BINDINGS
Energy
Ready
Standing
Hold
Find hold
Pull(self)
Stabilize
On top

30. Have + (Climb)-ed (Perfect)
Ready
Done
Start
Process
Finish
Suspend
Cancel
interrupt
resume
Iterate
BINDINGS
Energy
Ready
Standing
Hold
Find hold
Pull(self)
Stabilize
On top

31. Phasal Aspect Maps to the Controller
Iterative (repeat)
Inceptive (start, begin)
Iterate
Ready
Start
Process
Finish
Done
interrupt
resume
Cancel
Suspend
Completive (finish, end)
Resumptive(resume)

32. Inference using the Controller
Different Bindings give rise to different interpretations.
Dowty’s Imperfective Paradox
He was walking to the store He was walking.
does not imply does imply
He walked to the store He walked.

33. Inherent Aspect
Much richer than traditional Linguistic Characterizations (VDT)
Action patterns
one-shot, repeated, periodic, punctual
decomposition: concurrent, alternatives, sequential
Goal based schema enabling/disabling
Generic control features;
interruption, suspension, resumption
Resource usage

34. Inherent Aspect Selects/Disables Controller Transitions

35. Viewpoint Aspect (Perfective/Imperfective)

36. A Precise Notion of Contingency Relations
Activation:
Executing one schema causes the enabling, start or continued execution of another schema. Concurrent and sequential activation.
Inhibition:
Inhibitory links prevent execution of the inhibited x-schema by activating an inhibitory arc. The model distinguishes between concurrent and sequential inhibition, mutual inhibition and aperiodicity.
Modification:
The modifying x-schema results in control transition of the modified xschema. The execution of the modifying x-schema could result in the
interruption, termination, resumption of the modified x-schema.

43. SHRUTI: a connectionist cognitive architecture
Connectionist model of knowledge representation and reasoning -- and a useful modeling framework Representation via focal clusters 2. Temporal synchrony variable binding
relational predicate
entity
type

48. Resources and actions

49. Basic Features Fine grained model of actions and events
Interruption, hierarchy, concurrency, synchronization, iteration
Models resources, preconditions, state changes
Active representation
Feedback loops
Forward and backward
Extensions allow hybrid system models

50. Interaction of Aspect with Tense
Reichenbach’s system uses three pointers
Speech Time (S)
Reference Time (R)
Event Time (E)
Tense is a partial ordering relation between the pointers
Simple Past E < R, E < S
Perfect E < R < S

52. Probabilistic Relation Inference
Scalable Representation of
States, domain knowledge, ontologies
(Avi Pfeffer 2000, Koller et al. 2001)
Merges relational database technolgy with Probabilistic reasoning based on Graphical Models.
Domain entities and relational entities
Inter-entity relations are probabilistic functions
Can capture complex dependencies with both simple and composite slot (chains).
Inference exploits structure of the domain

53. Model Review Basic Mechanism 3 2 [1] 1 [1] Resource arc
Precondition arc
Inhibition arc
[1]

54. Model Review
Firing Semantics 3 1 2

55. Model Review
Result of Firing 1
DEADLOCK! 2

57. States
Factorized Representation of State uses Dynamic Belief Nets (DBN’s)
Probabilistic Semantics
Structured Representation

58. A Simple DBN for the Economics Domain
Economic State
[recession,nogrowth,lowgrowth,higrowth]
Policy
[Liberalization, Protectionism]
Goal
[Free Trade, Protection ]
Outcome
[Success, failure]
Difficulty
[present, absent] T0 T1

59. Dynamic Bayes Nets
Dynamic Bayesian Networks (D(T)BNs) are an extension of Bayesian networks for modeling dynamic systems.
In a DBN, the state at time t is represented by a set of random variables. The state at time t is dependent on the states at previous time steps.
Typically, we assume that each state only depends on the immediately preceding state (first-order Markovian), and thus we need to represent the transition distribution P(Zt+1 | Zt).
This can be done using a two-time-slice Bayesian network fragment (2-TBN) Bt+1,
variables from Zt+1 whose parents are variables from Zt and/or Zt+1, and variables from Zt without any parents.
Typically, we also assume that the process is stationary, i.e., the transition models for all time slices are identical:

60. SHRUTI
SHRUTI does inference by connections between simple computation nodes
Nodes are small groups of neurons
Nodes firing in sync reference the same object