1. Monitoring Dynamical Systems: Combining Hidden Markov Models and Logic
Brian C. Williams J/6.834J October 21, 2002
9/13/00
copyright Brian Williams, 2000
courtesy of JPL
Brian C. Williams, copyright 2000

2. Outline Model-based Programming Estimation of Dynamic Systems
3/6/00
Outline
Model-based Programming
Estimation of Dynamic Systems
Mode Reconfiguration
9/13/00
copyright Brian Williams, 2000

3. act WORLD Plant actions observations P(s) Diagnostic Agent:
3/6/00
WORLD
Plant
actions
observations
P(s)
sense
act
Diagnostic Agent:
Monitors & Diagnoses
Repairs & Avoids
Probes and Tests
AGENT
Consistency-based
From Symptoms
9/13/00
copyright Brian Williams, 2000

4. Model-based Programs Interact Directly with State
Embedded programs interact with plant sensors/actuators:
Read sensors
Set actuators
Model-based programs interact with plant state:
Read state
Write state
Embedded Program S
Plant
Obs
Cntrl
Model-based Embedded Program S
Plant
Programmer must map between state and sensors/actuators.
Model-based executive maps between sensors, actuators to states.
9/13/00
copyright Brian Williams, 2000

5. 3/6/00
Example: The model-based program sets engine = thrusting, and the deductive controller . . .
Oxidizer tank
Fuel tank
Plans actions
to open
six valves
Deduces that
thrust is off, and the engine is healthy
Deduces that a valve
failed - stuck closed
A programmer uses a model-based program to specify the behavior of an autonomous system as a set of possible state trajectories.
The model-based executive uses the plant model to deal with the complexity of reasoning about interactions between this state and the sensors and actuators:
The mission planner and control sequencer selects a set of state trajectories, according to the program, which best matches the current situation.
The deductive controller deduces the current state from the sensors, and plans control actions to move to the desired state.
For example, given the desired state of “engine thrusting,” the deductive controller:
Deduces from sensor information that the engine is not thrusting, that all components are most likely healthy, and that all valves are closed.
Deduces that a most cost effective way to achieve thrust is to command the valves open on the left, as indicated, and to power down all extraneous devices. It plans, executes and monitors actions to achieve this.
Deduces based on acceleration and pressure data that an inlet valve on the left is stuck closed, and that “engine thrusting” has failed.
Deduces a valve setting for the right engine that achieves thrust, and plans, executes and monitors a set of actions that successfully achieves this setting.
Determines that valves
on the backup engine will achieve thrust, and plans needed actions.

6. Control Sequencer Deductive Controller Generates target goal states
3/6/00
RMPL Model-based Program
Control Sequencer
Deductive Controller
Commands
Observations
Plant
Titan Model-based Executive
State goals
State estimates
Control Program
Generates target goal states
conditioned on state estimates
Executes concurrently
Preempts
Queries (hidden) states
Asserts (hidden) state
System Model
Mode
Estimation
Reconfiguration
Tracks
likely plant states
Tracks least cost goal states
Closed
Valve
Open
Stuck
open
closed
Close
0. 01
0.01
inflow = outflow = 0
Model-based programs interact directly with hidden plant states:
they read and write (hidden) states.
they describe possible state evolutions at a high level.
A Model-based Executive:
Chooses the state evolutions that best match the current situation.
Using the Mission Planner (Kirk)
Maps between states and sensors/actuators.
using the Control Sequencer and Deductive Controller (Titan/Livingstone)
Is beginning to bridge the gap to continuous estimation and control
Hybrid estimation and control (Moriarty)

7. Control Sequencer Deductive Controller Generates target goal states
3/6/00
Mode Estimation:
Select a most likely set of component mode transitions that are consistent with the model and observations
Mode Reconfiguration:
Select a least cost set of component mode transitions that entail the current goal, and are consistent
RMPL Model-based Program
Control Sequencer
Deductive Controller
Commands
Observations
Plant
Titan Model-based Executive
State goals
State estimates
Control Program
Generates target goal states
conditioned on state estimates
Executes concurrently
Preempts
Queries (hidden) states
Asserts (hidden) state
System Model
Mode
Estimation
Reconfiguration
Tracks
likely plant states
Tracks least cost goal states
Closed
Valve
Open
Stuck
open
closed
Close
0. 01
0.01
inflow = outflow = 0
Model-based programs interact directly with hidden plant states:
they read and write (hidden) states.
they describe possible state evolutions at a high level.
A Model-based Executive:
Chooses the state evolutions that best match the current situation.
Using the Mission Planner (Kirk)
Maps between states and sensors/actuators.
using the Control Sequencer and Deductive Controller (Titan/Livingstone)
Is beginning to bridge the gap to continuous estimation and control
Hybrid estimation and control (Moriarty)
arg min Pt(m’)
s.t. M(m’) ^ O(m’) is consistent
arg max Rt(m’)
s.t. M(m’) entails G(m’)
s.t. M(m’) is consistent

8. Control Sequencer Deductive Controller Generates target goal states
3/6/00
OpSat:
arg min f(x)
s.t. C(x) is satisfiable
D(x) is unsatisfiable
RMPL Model-based Program
Control Sequencer
Deductive Controller
Commands
Observations
Plant
Titan Model-based Executive
State goals
State estimates
Control Program
Generates target goal states
conditioned on state estimates
Executes concurrently
Preempts
Queries (hidden) states
Asserts (hidden) state
System Model
Mode
Estimation
Reconfiguration
Tracks
likely plant states
Tracks least cost goal states
Closed
Valve
Open
Stuck
open
closed
Close
0. 01
0.01
inflow = outflow = 0
Model-based programs interact directly with hidden plant states:
they read and write (hidden) states.
they describe possible state evolutions at a high level.
A Model-based Executive:
Chooses the state evolutions that best match the current situation.
Using the Mission Planner (Kirk)
Maps between states and sensors/actuators.
using the Control Sequencer and Deductive Controller (Titan/Livingstone)
Is beginning to bridge the gap to continuous estimation and control
Hybrid estimation and control (Moriarty)
arg min Pt(m’)
s.t. M(m’) ^ O(m’) is consistent
arg max Rt(m’)
s.t. M(m’) entails G(m’)
s.t. M(m’) is consistent

9. Outline Model-based Programming Estimation of Dynamic Systems
3/6/00
Outline
Model-based Programming
Estimation of Dynamic Systems
Hidden Markov Models
Probabilistic, Concurrent Constraint Automata.
Mode Reconfiguration
9/13/00
copyright Brian Williams, 2000

10. Estimating Dynamic Systems
3/6/00
Estimating Dynamic Systems S T
Given prior commands and observations:
Infer distribution of most likely states
Infer most likely trajectories
9/13/00
copyright Brian Williams, 2000

11. Hidden Markov Model S, A, : Finite States, Actions & Observations
T(s,a,s’): State transition function
O(s’,a,o): Observation function
9/13/00
copyright Brian Williams, 2000

12. HMM Belief State Update
b(s) belief distribution over S.
9/13/00
copyright Brian Williams, 2000

13. Example: Where is Robbie?
9/13/00
copyright Brian Williams, 2000

14. Example: Where is Robbie?
Actions: North, South, East, West
Observations: Wall, No-Wall
Transition Function:
If action pushes against wall, Then stay in same place with P = 1.0.
Else action succeeds with P = stays in same place with P = 0.25
Observation Function:
If next to wall, observe wall with P = 1.0
If one square from wall, observe wall with P = 0.25
9/13/00
copyright Brian Williams, 2000

15. Example: Where is Robbie?4
9/13/00
copyright Brian Williams, 2000

16. Example: Where is Robbie? Moving North Predicts …7 4 1
9/13/00
copyright Brian Williams, 2000

17. Example: Where is Robbie? Observing No-Wall … (Roughly)9 12
9/13/00
copyright Brian Williams, 2000

18. Outline Model-based Programming Estimation of Dynamic Systems
3/6/00
Outline
Model-based Programming
Estimation of Dynamic Systems
Hidden Markov Models
Probabilistic, Concurrent Constraint Automata.
Mode Reconfiguration
9/13/00
copyright Brian Williams, 2000

19. Models: Concurrent Probabilistic Constraint Automata
3/6/00
Devices modes
Probabilistic transitions between modes
State constraints for each mode
One automata per component
Vlv = closed =>
Outflow = 0;
vlv=open =>
Outflow = Mz+(inflow);
vlv=stuck open =>
vlv=stuck closed=>
Closed
Open
Stuck
open
closed
Open
Close
Cost 5
Prob .9
Discuss Xerox Diagnosis modeling lesson
models are compositional, broadly applicable & highly reusable.
commonsense models are easy to articulate at concept stage and insensitive to design variations.
single model covers most functions
Unknown
9/13/00
copyright Brian Williams, 2000

20. Mode Estimation Find most likely reachable states
consistent with observations.
Left engine on
Observe
“no thrust”
Enumerate by decreasing probability.
9/13/00
copyright Brian Williams, 2000

21. Dynamic Mode Estimation
b(s) belief distribution over S.
Mode Estimation:
Distribution on modes, rather than states.
HMM encoded through concurrent automata and constraints.
Assume automata transitions are conditionally independent.
O(s’,a,o) derived from model, similar to GDE
9/13/00
copyright Brian Williams, 2000

22. Mode Estimation
Find feasible next states, given current state distribution, commands and next state observations.
Current
Si  Sui j
SOi+1 k
Si+1
Previous
Observed
Predicted S
9/13/00
copyright Brian Williams, 2000

23. Outline Model-based Programming Estimation of Dynamic Systems
3/6/00
Outline
Model-based Programming
Estimation of Dynamic Systems
Hidden Markov Models
Probabilistic, Concurrent Constraint Automata.
Mode Reconfiguration
9/13/00
copyright Brian Williams, 2000

24. Find reachable next states that entail goal.
Mode Reconfiguration
Find reachable next states that entail goal.
Current state
Goal
“nominal thrust”
Enumerate by increasing immediate cost.
9/13/00
copyright Brian Williams, 2000

25. 3/6/00
Conflicts Focus MR
Find least cost commands that achieve the current goal in the next state. 
Si  Suj n gi
Goal
n(Si  Suj)
Nominal
Next
Current
9/13/00
copyright Brian Williams, 2000

26. Conflicts Focus MR Goal: Achieve Thrust
A conflict, C, is an assignment to a subset of the control variables
that entails the negation of the goal.
9/13/00
copyright Brian Williams, 2000

27. Conflicts Focus MR Goal: Achieve Thrust
A conflict, C, is an assignment to a subset of the control variables
that entails the negation of the goal.
9/13/00
copyright Brian Williams, 2000

28. Conflicts Focus MR Goal: Achieve Thrust
A conflict, C, is an assignment to a subset of the control variables
that entails the negation of the goal.
9/13/00
copyright Brian Williams, 2000

29. Conflicts Focus MR Goal: Achieve Thrust
A conflict, C, is an assignment to a subset of the control variables
that entails the negation of the goal.
9/13/00
copyright Brian Williams, 2000

30. Performance on Cassini
Number of components: 80
Number of clauses: 11,101
9/13/00
copyright Brian Williams, 2000

31. Model-based Embedded Systems
To survive decades embedded systems orchestrate complex regulatory and immune systems.
Future systems will be programmed with models, describing themselves and their environments.
Runtime kernels will be agile, deducing and planning by solving optimization problems with propositional constraints.
OpSat problems are solved efficiently by combining conflict-recognition and A* search to leap over leading inconsistent solutions.
9/13/00
copyright Brian Williams, 2000