1. Announcements  Project 3 will be posted ASAP: hunting ghosts with sonar

2. 2

3. 3

4. CS 188: Artificial Intelligence Dynamic Bayes Nets and Particle Filters Instructor: Stuart Russell University of California, Berkeley

5. Today  Dynamic Bayes nets  Relationship to HMMs  Application: ICU monitoring  Particle filtering  And maybe some cool demos

6. Dynamic Bayes Nets

7. Dynamic Bayes Nets (DBNs)  We want to track multiple variables over time, using multiple sources of evidence  Idea: Repeat a fixed Bayes net structure at each time  Variables from time t can condition on those from t-1 G1aG1a E1aE1a E1bE1b G1bG1b G2aG2a E2aE2a E2bE2b G2bG2b t =1t =2 G3aG3a E3aE3a E3bE3b G3bG3b t =3

8. DBNs and HMMs  Every HMM is a single-variable DBN  Every discrete DBN is an HMM  HMM state is Cartesian product of DBN state variables  Sparse dependencies => exponentially fewer parameters  E.g., 20 state variables, 3 parents each; DBN has 20 x 2 3 = 160 parameters, HMM has 2 20 x 2 20 =~ 10 12 parameters

9. Exact Inference in DBNs  Variable elimination applies to dynamic Bayes nets  Offline: “unroll” the network for T time steps, then eliminate variables to find P(X T |e 1:T )  Online: eliminate all variables from the previous time step; store factors for current time only  Problem: largest factor contains all variables for current time (plus a few more) t =1t =2t =3 G1aG1a E1aE1a E1bE1b G1bG1b G2aG2a E2aE2a E2bE2b G2bG2b G3aG3a E3aE3a E3bE3b G3bG3b G3bG3b [Demo: pacman sonar ghost DBN model (L15D6)]

10. Video of Demo Pacman Sonar Ghost DBN Model

11. Application: ICU monitoring  State: variables describing physiological state of patient  Evidence: values obtained from monitoring devices  Transition model: physiological dynamics, sensor dynamics  Query variables: pathophysiological conditions (a.k.a. bad things) 11

12. Toy DBN: heart rate monitoring parameter variable state variable sensor variable sensor state variable

14. Vasculature Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge Brain Neurotransmitters Heart Blood flow

15. Vasculature Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Cardiac contrac- tility Venous tone Arterio- lar tone Cardiac preload Capillary pressure Barorecep- tor discharge Brain Neurotransmitters Heart Blood flow Heart rate Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure

16. Vasculature Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge MAP sensor model Heart rate sensor model Central venous pressure sensor model Brain Neurotransmitters Heart Blood flow

17. Vasculature Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge ABP sensor model Heart rate sensor model Central venous pressure sensor model Brain Neurotransmitters Heart Blood flow

18. Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Blood [volume] Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge Intracranial physiology Tissues- NOS [perfusion] MAP sensor model GI/Liver [perfusion] Heart rate sensor model Central venous pressure sensor model Blood [transu- dation] Setpoint inputs from ANS, CNS, intracranial, blood Pulm. [intra- thoracic press.] Brain Neurotransmitters Heart Blood flow Vasculature

19. Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge Intracranial physiology Tissues- NOS [perfusion] MAP sensor model GI/Liver [perfusion] Heart rate sensor model Central venous pressure sensor model Blood [transu- dation] Setpoint inputs from ANS, CNS, intracranial, blood Pulm. [intra- thoracic press.] Brain Neurotransmitters Heart Blood flow Vasculature Blood [volume]

20. Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Blood [volume] Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge Intracranial physiology Tissues- NOS [perfusion] MAP sensor model GI/Liver [perfusion] Heart rate sensor model Central venous pressure sensor model PK [conc. of phenyl- ephrine] Blood [transu- dation] Setpoint inputs from ANS, CNS, intracranial, blood Pulm. [intra- thoracic press.] Brain Neurotransmitters Heart Blood flow Vasculature

21. Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Blood [volume] Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge Intracranial physiology Tissues- NOS [perfusion] MAP sensor model GI/Liver [perfusion] Heart rate sensor model Central venous pressure sensor model PK [conc. of phenylephrine] Blood [transu- dation] Setpoint inputs from ANS, CNS, intracranial, blood Pulm. [intra- thoracic press.] Brain Neurotransmitters Heart Blood flow Vasculature

22. Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Blood [volume] Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge Intracranial physiology Tissues- NOS [perfusion] MAP sensor model GI/Liver [perfusion] Heart rate sensor model Central venous pressure sensor model PK [conc. of phenyl- ephrine] Blood [transu- dation] Setpoint inputs from ANS, CNS, intracranial, blood Pulm. [intra- thoracic press.] Medullary cardiovascular center Cardiac parasympathetic output Cardiac sympathetic output Card. M2 Card. β1 Card. β2 Vasc. α1 Vasc. α2 Vasc. β2 Heart rate Cardiac contrac- tility Venous tone Blood [volume] Arterio- lar tone Cardiac preload Capillary pressure Cardiac stroke volume Cardiac output Vascular resistance Mean arterial blood pressure Barorecep- tor discharge Intracranial physiology Tissues- NOS [perfusion] MAP sensor model GI/Liver [perfusion] Heart rate sensor model Central venous pressure sensor model PK [conc. of phenyl- ephrine] Blood [transu- dation] Setpoint inputs from ANS, CNS, intracranial, blood Pulm. [intra- thoracic press.] Brain Neurotransmitters Heart Blood flow Vasculature

23. Human physiology v0.1

24. Other people have done a lot of work! 10% of Guyton 1972

26. ICU data: 22 variables, 1min ave

27. 27

28. ICP closeup

29. 1s diastolic/systolic/mean ABP

30. Blood pressure measurement

34. ALARM

37. Particle Filtering

38. We need a new algorithm!  When |X| is more than 10 6 or so (e.g., 3 ghosts in a 10x20 world), exact inference becomes infeasible  Likelihood weighting fails completely – number of samples needed grows exponentially with T X1X1 X0X0 X2X2 X3X3 E1E1 E2E2 E3E3

39. We need a new idea!  The problem: sample state trajectories go off into low-probability regions, ignoring the evidence; too few “reasonable” samples  Solution: kill the bad ones, make more of the good ones  This way the population of samples stays in the high-probability region  This is called resampling or survival of the fittest t=2t=7

40. Particle Filtering 0.00.1 0.0 0.2 0.00.20.5  Represent belief state by a set of samples  Samples are called particles  Time per step is linear in the number of samples  But: number needed may be large  This is how robot localization works in practice

41. Representation: Particles  Our representation of P(X) is now a list of N particles (samples)  Generally, N << |X|  Storing map from X to counts would defeat the point  P(x) approximated by number of particles with value x  So, many x may have P(x) = 0!  More particles, more accuracy  Usually we want a low-dimensional marginal  E.g., “Where is ghost 1?” rather than “Are ghosts 1,2,3 in {2,6], [5,6], and [8,11]?”  For now, all particles have a weight of 1 Particles: (3,3) (2,3) (3,3) (3,2) (3,3) (3,2) (1,2) (3,3) (2,3)

42. Particle Filtering: Propagate forward  A particle in state x t is moved by sampling its next position directly from the transition model:  x t+1 ~ P(X t+1 | x t )  Here, most samples move clockwise, but some move in another direction or stay in place  This captures the passage of time  If enough samples, close to exact values before and after (consistent) Particles: (3,3) (2,3) (3,3) (3,2) (3,3) (3,2) (1,2) (3,3) (2,3) Particles: (3,2) (2,3) (3,2) (3,1) (3,3) (3,2) (1,3) (2,3) (3,2) (2,2)

43.  Slightly trickier:  Don’t sample observation, fix it  Similar to likelihood weighting, weight samples based on the evidence  W = P(e t | x t )  Normalize the weights: particles that fit the data better get higher weights, others get lower weights Particle Filtering: Observe Particles: (3,2) w=.9 (2,3) w=.2 (3,2) w=.9 (3,1) w=.4 (3,3) w=.4 (3,2) w=.9 (1,3) w=.1 (2,3) w=.2 (3,2) w=.9 (2,2) w=.4 Particles: (3,2) (2,3) (3,2) (3,1) (3,3) (3,2) (1,3) (2,3) (3,2) (2,2)

44. Particle Filtering: Resample  Rather than tracking weighted samples, we resample  N times, we choose from our weighted sample distribution (i.e., draw with replacement)  Now the update is complete for this time step, continue with the next one Particles: (3,2) w=.9 (2,3) w=.2 (3,2) w=.9 (3,1) w=.4 (3,3) w=.4 (3,2) w=.9 (1,3) w=.1 (2,3) w=.2 (3,2) w=.9 (2,2) w=.4 (New) Particles: (3,2) (2,2) (3,2) (2,3) (3,3) (3,2) (1,3) (2,3) (3,2)

45. Summary: Particle Filtering  Particles: track samples of states rather than an explicit distribution Particles: (3,3) (2,3) (3,3) (3,2) (3,3) (3,2) (1,2) (3,3) (2,3) Propagate forwardWeightResample Particles: (3,2) (2,3) (3,2) (3,1) (3,3) (3,2) (1,3) (2,3) (3,2) (2,2) Particles: (3,2) w=.9 (2,3) w=.2 (3,2) w=.9 (3,1) w=.4 (3,3) w=.4 (3,2) w=.9 (1,3) w=.1 (2,3) w=.2 (3,2) w=.9 (2,2) w=.4 (New) Particles: (3,2) (2,2) (3,2) (2,3) (3,3) (3,2) (1,3) (2,3) (3,2) [Demos: ghostbusters particle filtering (L15D3,4,5)] Consistency: see proof in AIMA Ch. 15

46. Particle filtering on umbrella model 46

47. Robot Localization  In robot localization:  We know the map, but not the robot’s position  Observations may be vectors of range finder readings  State space and readings are typically continuous so we cannot usually represent or compute an exact posterior  Particle filtering is a main technique

48. Particle Filter Localization (Sonar) [Video: global-sonar-uw-annotated.avi]

49. Robot Mapping  SLAM: Simultaneous Localization And Mapping  We do not know the map or our location  State consists of position AND map!  Main techniques: Kalman filtering (Gaussian HMMs) and particle methods DP-SLAM, Ron Parr [Demo: PARTICLES-SLAM-mapping1-new.avi]

50. Particle Filter SLAM – Video 1 [Demo: PARTICLES-SLAM-mapping1-new.avi]

51. Particle Filter SLAM – Video 2 [Demo: PARTICLES-SLAM-fastslam.avi]