1. NW Computational Intelligence Laboratory 1 Designing A Contextually Discerning Controller By Lars Holmstrom

2. NW Computational Intelligence Laboratory 2 Main Points To Describe The Architecture And Training Methodology To Discuss What Is Mean By Context And Contextual Discernment To Demonstrate This Methodology Using The Pole- Cart System

3. NW Computational Intelligence Laboratory 3 Architecture Of A Contextually Discerning Controller Contextually Aware Controller Contextual Discerner R(t) C D (t) u(t)

4. NW Computational Intelligence Laboratory 4 What Is Meant By Context? Conceptual Definition –A Set Of Parameters Capable Of Representing Changes In The Dynamics Of A Plant Mathematical Definition –A Point On A Manifold Of Functions That Can Be Indexed By A Specific Point In An Associated Continuous Coordinate Space

5. NW Computational Intelligence Laboratory 5 The Set Of Sine Functions Specified By Coordinates Corresponding To Amplitude, Frequency, And DC-Offset 0.5*sin(0.7*x)+0.3

6. NW Computational Intelligence Laboratory 6 Contextual Discernment To Discern The True Coordinates Of A Function That We Know Lies On A Specified Manifold But Has Unknown Manifold Coordinates CDCD CACA Current Discerned Context Unknown Actual Context ∆C D0 C D1 ∆C D1 C D2 ∆C D2 C D3 C D0

7. NW Computational Intelligence Laboratory 7 Contextual Discernment System Function On Manifold At Location C A CDN (Context Discerning Network) D(t)=y A (t)-y D (t) x(t) C D (t) z -1 + ∆C D (t) Function On Manifold At Location C D y D (t) y A (t)

8. NW Computational Intelligence Laboratory 8 Training The Contextual Discernment System Function On Manifold At Location C A CDN (Context Discerning Network) D(t)=y A (t)-y D (t) x(t) C D (t) z -1 + ∆C D (t) Function On Manifold At Location C D y D (t) y A (t) ∆C D (t) = u(t) C D (t) = R(t) C D (t) + ∆C D (t) = R(t+1) U(t)=(y A (t)-y D (t)) 2 Critic C D (t) x(t) λ(t) Used To Train Critic C D (t+1) “Plant”

9. NW Computational Intelligence Laboratory 9 Training The Critic and CDN With DHP The Approach Is Similar To That For Training A Controller With DHP, But With The Substitutions: R(t) = C D (t) u(t) = ∆C D (t) R(t+1) = C D (t)+∆C D (t) Action Training Signal: λ(t+1) Critic Training Signal: λ(t) - dU(t)/dC D (t) + λ(t+1)(I+d∆C D /dC D )

10. NW Computational Intelligence Laboratory 10 R A (t) Training The CDN To Discern Plant Context Function On Manifold At Location C A Function On Manifold At Location C D CDN (Context Discerning Network) D(t)=y A (t)-y D (t) x(t) C D (t) z -1 + ∆C D (t) y D (t) y A (t) x(t) U(t)=(y A (t)-y D (t)) 2 Critic C D (t) λ(t) Plant (With Context Parameters C A ) Plant Model (With Context Parameters C D ) u(t) R A (t) D(t)=R A (t+1)-R D (t+1) U(t)=(R A (t+1)-R D (t+1)) 2 R D (t+1) R A (t+1) Used To Train Critic

11. NW Computational Intelligence Laboratory 11 Context And State Variables For Training CDN Using An Analytic Pole-Cart Plant C(t) –Mass Of Cart Ranged From [3 5]kg –Length Of Pole Ranged From [0.5 3]m R(t) –Position Of Cart Ranged From [-2 2]m –Velocity Of Cart Ranged From [-2 2]m/s –Angle Of Pole Ranged From [-π/2 π/2] rads –Angular Velocity Ranged From [-2 2]rads/s u(t) –Control Ranged From [-5 5]Newtons

12. NW Computational Intelligence Laboratory 12 Results Of Contextual Discernment Testing

13. NW Computational Intelligence Laboratory 13 Architecture Of A Contextually Discerning Controller Contextual Discerner Contextually Aware Controller R(t) C(t) u(t)

14. NW Computational Intelligence Laboratory 14 Designing A Contextually Aware Controller Train A Controller Using DHP In The Standard Fashion Except… Add The Contextual Variables For The Plant To The State Vector Used In The Training Process Vary These Contextual Variables Through The Training Process

15. NW Computational Intelligence Laboratory 15 Architecture Of A Contextually Discerning Controller Contextual Discerner Contextually Aware Controller R A (t) C D (t) u(t) z -1 u(t-1)

16. NW Computational Intelligence Laboratory 16 Future Tasks Improve Contextual Discerner For The Pole-Cart System Train A Contextually Aware Controller For The Pole-Cart System Test The Ability Of The Contextually Discerning Controller In A Simulated “Real-Time” Environment