1. Data Mining for Hierarchical Model Creation G. Michael Youngblood and Diane J. Cook IEEE Transactions on Systems, Man, and Cybernetics, Part C, 37(4):561-572, 2007 Reporter: Hsin-chan Tsai

2. Outline Markov decision process (MDS) Software architecture Kolmogorov (or descriptive) complexity Data mining for Model Creation

3. An intelligent agent agent environment Perception (Sensors) action (controller)

4. Markov decision process (MDS ) Same property as Naïve Bayesian Probability A B C 0.4 0.6

5. MavLab

6. Not in the Lab Lab entry to the desk Take a breakAlternate workstation work Leave the Lab from the desk end Going on breakGoing off break Return from alternate workstation end 1.0 Going alternate workstation 0.35 0.25 0.4 0.8 0.2 0.25 0.75 1.0

7. Software architecture Decision Information Communication Physical Device Physical Device Physical Device Physical Device Physical Device Hardware Interface Hardware Interface Hardware Interface Operating System OS Service Software Interface Software Interface Software Interface Logical Proxy Logical Proxy Logical Proxy Prediction (ALZ) Prediction (ED) MavCore Database Aggregator Decision Maker (ProPHet) Arbitration (Arbiter) CORBA Zero Conf point to pointNaming / Discovery

8. Decision Decision Maker  provides partially observable hierarchical based decision tasks Arbitration  A selected action filter  To prevent violation of security constraints or user preferences

9. MavHome

10. Data mining for Model Creation Mining sequential Patterns Using Episode discovery (ED) Predicting Activities Using ALZ ProPHeT Algorithm

11. Kolmogorov complexity for ED 01 重複 32 次  01 100000  2 8 Minimum description length (MDL) 0101010101010101010101010101010101010101010101010101010101010101 1100100001100001110111101110110011111010010000100101011110010110 64bits 1.1. 2.2. 2 64

12. Minimum description length Input sequence O The sets of patterns θ The Description length (DL) Compression MDL(O)=arg max θ {Γ(θ|O)}

13. 1100100001100001110111101110110011111010010000100101011110010110 Θ = 110 (9)  3 bit 總 length 數 DL(O, Θ ) = DL( Θ )+DL(O| Θ )  (3)+(64-3*9+1*9)=49 Original length DL(O)  64 bit Compression Γ= DL(O)/ DL(O,Θ)  64 / 49 = 1.31

14. Θ = 01 (32)  2 bit Original length DL(O)  64 bit DL(O,Θ) = DL(Θ)+DL(O|Θ)  (2)+(32)=34 Compression Γ= DL(O)/ DL(O,Θ)  64 / 34 = 1.88 > 1.31 better 0101010101010101010101010101010101010101010101010101010101010101

15. ALZ prediction aaababbbbaabccddcdaaaa  a-10,aa-5,aaa-2,ab-3,abc-1  b-6,ba-2,bb-3,bc-1,bcc-1  c-3,cc-1,ccd-1,cd-2,cda-1,cdd-1  d-3,da-1,daa-1,dc-1,dcd-1

16. ProPHeT Algorithm Input: Episode provide by ED for all abstract nodes Create abstract node S n for all episode instances if (instance contains low-level events) create product node S m else if S m first instance in set then insert S n into the location of S m create abstract node S m Assign horizontal trasition values between each node store observation history with each transition Create end node e n Assign horizontal transition value from last node in instance sequence to e n end for Assign the vertical tansition values Connect abstract node S n to root node S 1 Connect all abstract node to an end node directly linked to the root node end for

17. Example Episode: {123,154,143,257,705} 123’s instance {143, 257} 705’s instance {154} 123 143257 123 143257154 end 705 end 154 end root

18. Learning from the hierarchical model Reinforcement learning Q( s, a) – the utility value to incrementally estimate for state / action pair.  If the agent executes action a in state s  After each action the utility is updated as Q( s, a)  Q( s, a) + α[r + γQ( s’, a’) - Q( s, a)] Select highest expected utility to execute

19. Experimental results

20. ProPHeT-generated hierarchical model (figure 8) 1 652371312935135155547571140178117648 e_1_0 1414262222364681205016823 e_2_65237 691273354008516 e_2_13129 0.2 0.6 0.8 0.1 1.0 0.4 0.2 0.8 1.0 0.5 1.0

21. Inhabitant interaction reduction in MavLab (top) and MavPad (bottom)

22. ProPHeT-generated hierarchical model 1 5427127 2 0.11 0.22 0.33 e_1_0 2463944549 e_2_54 0 49261649615 e_2_27127 2444545466240644895194926214 e_2_2 1.0 0.17 0.5

23. Example based on MavHome Not in Lab (1) Lab entry to desk (28485) Alternate workstation Work (237845) Take a break (456370) Leave lab from desk (34566) end Go to alternate workstation (62435) Return from alternate workstation (67348) end V24 ON c12 OFF … v22 OFF V28 ON v25 OFF V30 ON v28 OFF a16 ON NO ACT end a14 ON

24. MavLab virtual inhabitant automation performance

25. MavLab virtual inhabitant interaction reducing during long-term experimentation