Data Mining for Hierarchical Model Creation G. Michael Youngblood and Diane J. Cook IEEE Transactions on Systems, Man, and Cybernetics, Part C, 37(4): , 2007 Reporter: Hsin-chan Tsai

Outline Markov decision process (MDS) Software architecture Kolmogorov (or descriptive) complexity Data mining for Model Creation

An intelligent agent agent environment Perception (Sensors) action (controller)

Markov decision process (MDS ) Same property as Naïve Bayesian Probability A B C

MavLab

Not in the Lab Lab entry to the desk Take a breakAlternate workstation work Leave the Lab from the desk end Going on breakGoing off break Return from alternate workstation end 1.0 Going alternate workstation

Software architecture Decision Information Communication Physical Device Physical Device Physical Device Physical Device Physical Device Hardware Interface Hardware Interface Hardware Interface Operating System OS Service Software Interface Software Interface Software Interface Logical Proxy Logical Proxy Logical Proxy Prediction (ALZ) Prediction (ED) MavCore Database Aggregator Decision Maker (ProPHet) Arbitration (Arbiter) CORBA Zero Conf point to pointNaming / Discovery

Decision Decision Maker  provides partially observable hierarchical based decision tasks Arbitration  A selected action filter  To prevent violation of security constraints or user preferences

MavHome

Data mining for Model Creation Mining sequential Patterns Using Episode discovery (ED) Predicting Activities Using ALZ ProPHeT Algorithm

Kolmogorov complexity for ED 01 重複 32 次   2 8 Minimum description length (MDL) bits

Minimum description length Input sequence O The sets of patterns θ The Description length (DL) Compression MDL(O)=arg max θ {Γ(θ|O)}

Θ = 110 (9)  3 bit 總 length 數 DL(O, Θ ) = DL( Θ )+DL(O| Θ )  (3)+(64-3*9+1*9)=49 Original length DL(O)  64 bit Compression Γ= DL(O)/ DL(O,Θ)  64 / 49 = 1.31

Θ = 01 (32)  2 bit Original length DL(O)  64 bit DL(O,Θ) = DL(Θ)+DL(O|Θ)  (2)+(32)=34 Compression Γ= DL(O)/ DL(O,Θ)  64 / 34 = 1.88 > 1.31 better

ALZ prediction aaababbbbaabccddcdaaaa  a-10,aa-5,aaa-2,ab-3,abc-1  b-6,ba-2,bb-3,bc-1,bcc-1  c-3,cc-1,ccd-1,cd-2,cda-1,cdd-1  d-3,da-1,daa-1,dc-1,dcd-1

ProPHeT Algorithm Input: Episode provide by ED for all abstract nodes Create abstract node S n for all episode instances if (instance contains low-level events) create product node S m else if S m first instance in set then insert S n into the location of S m create abstract node S m Assign horizontal trasition values between each node store observation history with each transition Create end node e n Assign horizontal transition value from last node in instance sequence to e n end for Assign the vertical tansition values Connect abstract node S n to root node S 1 Connect all abstract node to an end node directly linked to the root node end for

Example Episode: {123,154,143,257,705} 123’s instance {143, 257} 705’s instance {154} end 705 end 154 end root

Learning from the hierarchical model Reinforcement learning Q( s, a) – the utility value to incrementally estimate for state / action pair.  If the agent executes action a in state s  After each action the utility is updated as Q( s, a)  Q( s, a) + α[r + γQ( s’, a’) - Q( s, a)] Select highest expected utility to execute

Experimental results

ProPHeT-generated hierarchical model (figure 8) e_1_ e_2_ e_2_

Inhabitant interaction reduction in MavLab (top) and MavPad (bottom)

ProPHeT-generated hierarchical model e_1_ e_2_ e_2_ e_2_

Example based on MavHome Not in Lab (1) Lab entry to desk (28485) Alternate workstation Work (237845) Take a break (456370) Leave lab from desk (34566) end Go to alternate workstation (62435) Return from alternate workstation (67348) end V24 ON c12 OFF … v22 OFF V28 ON v25 OFF V30 ON v28 OFF a16 ON NO ACT end a14 ON

MavLab virtual inhabitant automation performance

MavLab virtual inhabitant interaction reducing during long-term experimentation