1. Gartner Application Integration and Web Services Summit Notes accompany this presentation. Please select Notes Page view. Mani Chandy, Caltech Event-Driven Applications: Where they Apply and How they are Built K. Mani Chandy California Institute of Technology mani@cs.caltech.edu Special thanks to Roy Schulte and David Luckham.

2. Mani Chandy, Caltech EDA Business Value Respond to events – threats and opportunities.

3. Mani Chandy, Caltech Take Away 1: EDA Characteristics 1. Monitors and correlates events outside and inside the enterprise. 2. No communication IMPLIES reality matches model.

4. Mani Chandy, Caltech Take Away 2: You’re ready for EDA now 3. Your company is already event driven. 4. You already have the components of the architecture in your enterprise stack, and it’s complementary to SOA. Key Question for you: Do the incremental benefits of IT for EDA exceed its incremental costs?

5. Mani Chandy, Caltech Organisms Sense and Respond: EDA 1) Streams of data: sight, sound, touch, smell, taste 2) Central nervous system detects the rare event that is a threat or an opportunity 3) Organism responds appropriately to threat or opportunity 4) Too many false positives, a false negative, or too many delayed or inappropriate responses results in death

6. Mani Chandy, Caltech Group Sense and Respond: EDA Three enterprises: lions, hyenas, zebras Critical conditions: fuse information from inside and outside the enterprise

7. Mani Chandy, Caltech External Awareness: Questions for you Does your enterprise monitor its competitors? Government agencies? Do people in your enterprise correlate information from multiple sources? e.g., correlate flood at a supplier’s factory with deadlines for critical customers.

8. Mani Chandy, Caltech Responding to Unexpected Situations Question for you: A fire has just occurred in a factory that is going to effect customers severely. Which of two scenarios represents your enterprise? 1. The CEO doesn’t expect VP Mfg to say anything unless the CEO asks. 2. The CEO expects VP Mfg to tell the CEO.

9. Mani Chandy, Caltech No Communication => Reality = Model 1. CEO has an expectation – a model – of the CFO: No communication implies reality matches the model. 2. Space shuttle director has a model for the engineer responsible for shuttle foam insulation. No communication implies reality matches the model.

10. Mani Chandy, Caltech Enterprise-Wide Situational Awareness Military situational awareness Houston Denver Edmonton London Sydney NY, NY Corporate situational awareness Trader cockpit Risk manager Houston Corporate VP, risk Risk management cockpit Scheduler cockpit

11. Mani Chandy, Caltech Take-Away Points 1. EDA: Global situational awareness 2. EDA: Respond when reality deviates from model. Your enterprise is: 3. Already event-directed. 4. Has the software components including SOA Question for you: incremental costs vs. benefits? 4:05 PM

12. Mani Chandy, Caltech What is EDA? System that manages and executes rules of the form: WHEN reality deviates significantly from expectations THEN initiate appropriate response.

13. Mani Chandy, Caltech EDA Characteristics Aggregate events across multiple sources Analyze Detect events across extended environment in real-time Sense Invoke distributed services in real-time Respond

14. Mani Chandy, Caltech EDA: Software Characteristics 1. Asynchronous coupling. The timing, place, and characteristics of threats and opportunities are not determined by you. So sensors are responsible for pushing information; responders are not responsible for pulling it. 2. Defensive programming to the extreme. Data may be unstructured and inaccurate. Protocols may be unspecified.

15. Mani Chandy, Caltech Defensive Programming to the Extreme Airline A Airline B Monitoring One airline can make few assumptions about another airline. EDA should be very robust; or it is very brittle The robustness comes at a price EDA is at the limit of coupling “looseness”

16. Mani Chandy, Caltech Defensive Programming to the Extreme Division A Division B Monitoring One division can make few assumptions about another division. EDA should be very robust The robustness comes at a price EDA is at the limit of coupling “looseness”

17. Mani Chandy, Caltech EDA Structure: Sense, Analyze, Respond

18. Mani Chandy, Caltech Defensive Programs: Sensors & Responders PROGRAM OUTWARD-FACING COMPONENTS EXTREMELY DEFENSIVELY SENSORS RESPONDERS

19. Mani Chandy, Caltech Less Defensive: Event Processing Network PROGRAM INWARD-FACING COMPONENTS LESS DEFENSIVELY

20. Mani Chandy, Caltech Compare EDA Requirements with SOA SOA: Components are collaborators –Accounting “client” calls a “sales pipeline expectation” method on a sales “service” which returns with a report SOA: Time of interaction determined by client SOA: Service protocols and schemas are well defined. SOA: Units obtain global situational awareness by invoking multiple services.

21. Mani Chandy, Caltech EDA: Software Characteristics - Review 1. Asynchronous coupling. The timing, place, and characteristics of threats and opportunities are not determined by you. So sensors are responsible for pushing information; responders are not responsible for pulling it. 2. Defensive programming to the extreme. Data may be unstructured and inaccurate. Protocols may be unspecified. 3. Sense – Analyze – Respond. 4. When-Then rules.

22. Mani Chandy, Caltech An Event-Driven Architecture Database Interaction DB Electronic Markets Applications Market Interaction News Handler Ticker Handler Stock tickers News feeds Application Handler When-Then Rule Mgmt System Configuration Monitoring Text Analysis Parametric Analysis Time series Analysis ESBESB Web Service Web sites Approximate Matching Alert Engine End Users Screen Scrape Hostile Web sites Complex Event Patterns Scheduler

23. Mani Chandy, Caltech Take-Away 2 (repeated) Your enterprise already has most of the components of the architecture: –EAI tools; Messaging, ESB or event distributors; Databases; Alerts engines; Web Services; Rules engines EDA is compatible with and complements SOA. Your enterprise is already event-directed. Your key question: Do the incremental benefits exceed the incremental costs? 4:15 PM

24. Mani Chandy, Caltech Inevitable Down-Sides of EDA Effort required to specify and tune rules. Errors: 1. False positives: Response to non-event Drowning in Events: “turn that darn thing off!” 2. False negatives: Non-response to event Why are these down-sides inevitable?

25. Mani Chandy, Caltech Inevitable EDA Down-Sides: Errors When-then rules are imperfectly specified. When clauses evaluated incorrectly or late. Changing rules to reduce false positives increases false negatives (and vice-versa).

26. Mani Chandy, Caltech Inevitable EDA Down-Sides: Drown in Events Most responses alert people. Perception: false positives cost much less than false negatives. Too many false positives: “so, turn that thing off.” Correct evaluation should be based on total costs of all the false positives and negatives over time.

27. Mani Chandy, Caltech Inevitable Down-Sides of EDA: No Rules People are busy. Learning tools for specifying rules takes time. So, rules aren’t specified, or are specified in a cursory fashion. Destroys business value of EDA.

28. Mani Chandy, Caltech Rule Development Solutions 1. Have IT (or some central org.) specify rules. Doesn’t work. 2. Work with business users to specify rule templates; individuals fill in templates. 3. Have role-based rule repositories.

29. Mani Chandy, Caltech Is Anything Missing in your Software Stack? Event Process Agents 1. Machine can “learn” the critical condition from positive and negative examples 2. Users can specify critical conditions –SQL-like queries –Fuzzy matches –Statistical operators –Regular expressions –CEP

30. Mani Chandy, Caltech Estimating Performance Requirements Delay from occurrence of condition to initiation of response: Minutes? Sub-seconds? Number of data sources: Tens, Hundreds? Numbers of rule templates: Tens, Hundreds? Numbers of users? Numbers of rules? Observation: Many enterprises overshoot: they estimate greater performance requirements than they need.

31. Mani Chandy, Caltech Yes Use Your Existing Software Stack If Delay from occurrence of condition to initiation of response: Tens of seconds Number of data sources: Ten, Numbers of rule templates: Ten, Numbers of users? 100s Numbers of rules? 1000s This is the more common case.

32. Mani Chandy, Caltech Speed of EDA uptake Enterprises are event-driven. So, why is EDA uptake slow? 1. Simple event processing is widely used. Enterprise service buses and EAI tools allow when clauses on single documents (events). 2. Multiple event streams are correlated in some applications But these apps are not seen as part of an EDA paradigm.

33. Mani Chandy, Caltech Speed of EDA uptake Correlation across multiple event streams in: 1. Financial trading; IT Infrastructure management; Plant control; Defense Why these spaces? 1. Small designated groups responsible for responding to critical events. 2. Clear additional value 3. Performance

34. Mani Chandy, Caltech Should you build EDA? Your company is already event driven; you have many of the software components. Question: Do incremental benefits exceed incremental costs?

35. Mani Chandy, Caltech Should you build EDA? Do you have: 1. Applications: Apps that sense and respond to the environment and that benefit from automation? Evolving middleware that can benefit from asynchronous coupling? 2. Responsibility in a single group or LOB? 3. Performance requirements not met? 4. Small numbers of data sources and rule templates satisfying large numbers of users?

36. Mani Chandy, Caltech Should you build EDA? Are costs of false positives and negatives smaller than benefits of EDA? Do you have a senior manager in a LOB who will make all the hard business work happen? (Building effective EDA is 95% effort by business people and only 5% effort in technology.)

37. Mani Chandy, Caltech Future Talks How not to build EDA. Lessons from the trenches! Steps and gotchas in building EDA.

38. Mani Chandy, Caltech Take-Away Points and THANKS! 1. EDA: Global situational awareness 2. EDA: Unit “knows” reality matches model until the unit gets a message. Your enterprise: 3. Is already event-directed. 4. Has the software components including SOA Your Question: incremental costs vs. benefits? 4:45 PM

39. Mani Chandy, Caltech Getting Business Users to Specify Rules is Hard Different roles have different sets of rules. Who specifies the rules? How many rule templates? Tens, hundreds? How many rules? Hundreds, thousands? How are rules specified? Language, visual UI, positive and negative examples? Fill in templates? Who can turn a rule off? What if rules are at cross-purposes?