1. A Complex Event Recognition Architecture Will Fitzgerald Kalamazoo College R. James Firby I/NET, Inc.

2. A Complex Event Recognition Architecture Protecting us from the Metal Horde! Will Fitzgerald R. James Firby

3. What is … A Complex Event? A Complex Event? Complex events are hierarchical, discrete, time-stamped structures inferred from multi-channel, asynchronous signals. Complex events are hierarchical, discrete, time-stamped structures inferred from multi-channel, asynchronous signals.

4. What is … A Complex Event Recognition Architecture? A Complex Event Recognition Architecture? A description or implementation of typical patterns and recognition algorithms for complex events. A description or implementation of typical patterns and recognition algorithms for complex events.

5. A real example… Water Recovery System at NASA's Johnson Space Center Water Recovery System at NASA's Johnson Space Center Four complex subsystems, Four complex subsystems, About 200 sensors and actuators, About 200 sensors and actuators, Each subsystem asynchronously signals data. Each subsystem asynchronously signals data.

6. Detecting Safe Mode When a problem is detected internally, the Water Recovery System attempts to go into safe mode, which occurs when the four subsystems are safed. When a problem is detected internally, the Water Recovery System attempts to go into safe mode, which occurs when the four subsystems are safed. Safing of the four subsystems happen asynchronously. Safing of the four subsystems happen asynchronously. Safing detection for each subsystem differs from one another. Safing detection for each subsystem differs from one another. On recognizing that the WRS has gone into safe mode, signal an event that all subsystems have been safed. On recognizing that the WRS has gone into safe mode, signal an event that all subsystems have been safed.

7. Another example… To get directions to a location on the on- board map, the user says: Go here and Taps the display location within 200 ms. within 200 ms. (CNN photo)

8. Parsing the world Dynamic Predictive Memory Architecture (DPMA) Dynamic Predictive Memory Architecture (DPMA) KR and Semantic parsing KR and Semantic parsing Task execution and dialogue management Task execution and dialogue management complex, dynamic environments complex, dynamic environments Do similar techniques apply to … Do similar techniques apply to … multi-channel, asynchronous sensors? multi-channel, asynchronous sensors? multi-modal interface input? multi-modal interface input?

9. A Complex Event Recognition Architecture What assumptions are reasonable to make about the form of input data? What assumptions are reasonable to make about the form of input data? What useful general patterns are there in the data? What useful general patterns are there in the data? What recognition algorithms do we need? What recognition algorithms do we need?

10. NLP Assumptions Input to Natural Language Processing systems are typically assumed to be: Input to Natural Language Processing systems are typically assumed to be: Discrete events of one type (words) Discrete events of one type (words) Single channel Single channel Totally ordered by position; duration irrelevant Totally ordered by position; duration irrelevant 12345 timeflieslikeanarrow

11. More generally… Events of various types Events of various types Over multiple channels and asynchronous Over multiple channels and asynchronous Duration of event often important Duration of event often important Hierarchical model still useful Hierarchical model still useful 000005010015020025030 putthesehere click and drag click and drag tap

12. Assumptions about Events Discrete: Individually distinict, non- continuous data (could be discretized). Discrete: Individually distinict, non- continuous data (could be discretized). Time-stamped: Event carries the start and end times (defining the event duration, which could be instanteneous). Time-stamped: Event carries the start and end times (defining the event duration, which could be instanteneous). Typed: Events form distinct types (e.g., words vs. taps). Typed: Events form distinct types (e.g., words vs. taps). Structured: Event may internal, hierarchical structure (complex). Structured: Event may internal, hierarchical structure (complex).

13. Standard Event Patterns Are there patterns of events which are particularly useful to identify? Are there patterns of events which are particularly useful to identify? Are there recognition algorithms to identify those patterns? Are there recognition algorithms to identify those patterns? Yes. Yes. ONE and BINDING ONE and BINDING IN-ORDER, ALL, ONE-OF IN-ORDER, ALL, ONE-OF Allen patterns Allen patterns WITHIN and WITHOUT WITHIN and WITHOUT

14. ONE and BINDING patterns ONE: The simple pattern of looking for a single event (of a particular type). ONE: The simple pattern of looking for a single event (of a particular type). BINDING: ONE pattern plus collecting and constraining state. BINDING: ONE pattern plus collecting and constraining state. Essentially event- driven programming; the stimulus in S-R. Essentially event- driven programming; the stimulus in S-R. ON-CLICK ON-CLICK A ONE pattern if just looking for the click A BINDING pattern if x,y coordinates are significant.

15. IN-ORDER patterns Events will occur in order Events will occur in order That is, saying two events, A and B, occur in order, the start time of B is the end time of A. That is, saying two events, A and B, occur in order, the start time of B is the end time of A. (IN-ORDER A B C D) (IN-ORDER A B C D) First an event of type A, then B, etc. First an event of type A, then B, etc.

16. IN-ORDER as NLP Combined with BINDING and signaling of subpatterns this is essentially a classic natural language processing pattern. Combined with BINDING and signaling of subpatterns this is essentially a classic natural language processing pattern. S NP VP NP DET N VP V NP The boy saw the girl. [S [NP [DET the][N boy]] [VP [V saw] [NP [DET the] [N girl]]]]

17. ALL Patterns Events will all occur, but in any order Events will all occur, but in any order With this, we leave (our) standard NLP approaches. With this, we leave (our) standard NLP approaches. For example, user will choose from all of the sets of options. For example, user will choose from all of the sets of options. For example, all subsystems will be safed, but in any order. For example, all subsystems will be safed, but in any order.

18. ALL patterns and contradiction The problem: user or system undoing an event that has already been seen (interpreting events as state changes). The problem: user or system undoing an event that has already been seen (interpreting events as state changes). Example: Class will start when all the students, Alice, Bob, Charles, Dominique, have arrived. Example: Class will start when all the students, Alice, Bob, Charles, Dominique, have arrived.

19. Consider this sequence for (ALL A B C D): 1. Charles arrives. 2. Alice and Bob arrive together. 3. Alice starts to sing. 4. Charles leaves. 5. Dominique arrives. 6. Charles arrives. Order is not relevant; Alices singing is not relevant; but Charless leaving undoes his earlier arrival.

20. ONE-OF Pattern Look for any of a set of event forms Look for any of a set of event forms Example: Office hours begin as soon as one of the professors A,B,C or D arrives. Example: Office hours begin as soon as one of the professors A,B,C or D arrives. (ONE-OF A B C D) (ONE-OF A B C D)

21. Time-based patterns Allen relationships Allen relationships WITHIN patterns WITHIN patterns WITHOUT patterns WITHOUT patterns

22. Allen Patterns James Allen described the relationships between two intervals. James Allen described the relationships between two intervals. Allen patterns look for temporal relationships between 2 events or an event and an interval. Allen patterns look for temporal relationships between 2 events or an event and an interval. 1. contains 2. finishes 3. starts 4. before 5. meets 6. overlaps 7. equal 8. overlapped by 9. after 10. met by 11. started by 12. finished by 13. during A contains B A … overlaps B

23. WITHIN and WITHOUT WITHIN patterns reflect that the duration of an event is no longer than a certain amount of time. WITHIN patterns reflect that the duration of an event is no longer than a certain amount of time. E.g., an ALL pattern wrapped in a WITHIN pattern. E.g., an ALL pattern wrapped in a WITHIN pattern. WITHOUT patterns reflect that an interval of time will pass without the occurrence of an event. WITHOUT patterns reflect that an interval of time will pass without the occurrence of an event. E.g., Sherlock Holmess significance of the barking dog. E.g., Sherlock Holmess significance of the barking dog.

24. Pattern Combination Go here and a tap within 200 ms. Go here and a tap within 200 ms. (within (all (in-order go here) (tap ?x ?y)) 200 ms) (CNN photo)

25. Safe mode recognizer (define-recognizer (safing-complete) (pattern (pattern '(all '(all (safing (system pbbwp) (status on)) (safing (system pbbwp) (status on)) (safing (system ro) (status on)) (safing (system ro) (status on)) (safing (system aes) (status on)) (safing (system aes) (status on)) (safing (system pps) (status on)))) (safing (system pps) (status on)))) (on-complete (st end) (on-complete (st end) (signal-event '(all-safed) st end))) (signal-event '(all-safed) st end))) Some details elided…

26. Parsing Algorithms The parsing algorithms and recognizer semantics are more fully described in the paper. The parsing algorithms and recognizer semantics are more fully described in the paper.

27. Implementation Details

28. Conclusions Standard patterns of events. Standard patterns of events. Standard recognizers for these patterns. Standard recognizers for these patterns. Good for monitoring complex (internal) system state. Good for monitoring complex (internal) system state. Useful for recognizing patterns of complex events over multiple modes, over time. Useful for recognizing patterns of complex events over multiple modes, over time.

29. Acknowledgments Work done under NASA SBIR contract NAS9-00122. Work done under NASA SBIR contract NAS9-00122. We would like to especially acknowledge collaborators at NASA, including Debra Schreckenghost, Pete Bonasso, Carrol Thronesbery and others. We would like to especially acknowledge collaborators at NASA, including Debra Schreckenghost, Pete Bonasso, Carrol Thronesbery and others. Pulp Images from Pulp of the Day: groups.yahoo.com/group/pulpoftheday Pulp Images from Pulp of the Day: groups.yahoo.com/group/pulpoftheday

30. Questions? Questions?