1. Interoperability

2. Introductions & Session objectives Operational Intelligence Context Explore some of the parts – CEP – ESB – Cloud Agenda

3. Operational Intelligence Federated activities Telco billing dispute, Process customer returns, Data intensive Interpret and react (sense & respond) Automate and self-correct Optimisation & agility

4. “Process Mining” What is really happening? Compliance: Does it comply to regulation? Performance: Are there any bottlenecks? Prediction: Will the SLA be breached Improvement: How can it be improved http://prom.win.tue.nl/research/wiki/_media/presentations/gartner-keynote-bpm2009-wvda.pdfProcess Mininghttp://prom.win.tue.nl/research/wiki/_media/presentations/gartner-keynote-bpm2009-wvda.pdfProcess Mining Professor Wil van der Aalst, University of Eindhoven

5. The Value of Timely Analytics Present Time of interest Web Analytics – Ad placement, Financial Services, Smart Grids, Monitoring – Systems mgmt, Health Care, Manufacturing, etc. years monthsdayshrsminsec $ value of analytics Forecasting in Enterprises Historical Trend Analysis

6. Current Products for Analytics Traditional DW Analytics Active DW analytics Present Time of interest 100000 10000 1000 100 Custom-built solutions that carry huge development and customization costs Facts/sec. yearsmonthsdayshrsminsec Load time in ETL ET time in ETL Load barrier is dictated by current choices of the solution, e.g., loading into databases, persisting into files. This is intrinsic because in current approaches no processing can be done till the data is loaded.

7. Recap: What are Event-Driven Apps? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less Data RateHundreds of events/secTens of thousands of events/sec or more request response Event output stream input stream

8. Events Events expose different temporal characteristics – Point in time events – Interval events with fixed duration – Interval events with initially unknown duration Rich payloads capture all properties of an event t1 t4 t3 t2 t5 Time  Payload/ value  a b c d e

9. Event Processing Architecture 9 CEP Engine Output Adapters Input Adapters Event Standing Queries Applications & Event sources Event targets Devices, Sensors Web servers Event stores & Databases Stock tickers & News feeds Event C_IDC_NAMEC_ZIP Event stores & databases Business process & ESB KPI Dashboards, SharePoint UI Trading stations Event CEP Application at Runtime Static reference data IDE.NET C# LINQ CEP Application Development Cloud

10. Relational Database Applications Financial trading Applications Core scenarios – event driven Aggregate Data Rate (Events/sec.) Latency 010100100010000100000~1million Months Days hours Minutes Seconds 100 ms < 1ms Manufacturing Applications Monitoring Applications CEP Target Scenarios Data Warehousing Applications Web Analytics Applications 10 Operational Analytics Applications, e.g., Logistics, etc.

11. Example Scenarios Data Stream Stream Data Store & Archive Event Processing Engine Data Stream Asset Specs & Parameters Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Web Analytics: Click-stream data Online customer behavior Page layout 100,000 events /sec Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Threshold queries Event correlation from multiple sources Pattern queries Threshold queries Event correlation from multiple sources Pattern queries Lookup Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec

12. Industry trends Data acquisition costs are negligible Raw storage costs are small and continue to decrease Processing costs are non-negligible Data loading costs continue to be significant Manage business via KPI-triggered actions Mine historical data Devise new KPIs Monitor KPIs Record raw data (history) Virtuous Cycle: Monitor, Manage, Mine CEP advantage Process data incrementally, i.e., while it is in flight Avoid loading while still doing the processing you want Seamless querying for monitoring, managing and mining

13. Process Improvement: Genetic Mining http://prom.win.tue.nl/research/wiki/_media/presentations/gartner-keynote-bpm2009-wvda.pdfProcess Mininghttp://prom.win.tue.nl/research/wiki/_media/presentations/gartner-keynote-bpm2009-wvda.pdfProcess Mining Professor Wil van der Aalst, University of Eindhoven

14. STREAMINSIGHT - DEMO Market Monitor

15. Silverlight App Excel StreamInsight Output Adapter

16. Snapshot Windows defined according to time stamps in the event stream Start of next event, triggers end of snapshot windows 3 events Snapshot windows

17. Snapshot query Sum the events in the window Aggregate events for each quote type

18. “SPAGHETTI” OF PROPRIETARY INTERFACES INTEGRATION BROKER (EAI/B2B) ENTERPRISE SERVICE BUS The Changing Landscape Application Integration Domain CRM

19. Itinerary – Transform & Route Transform using rules resolver Runtime lookup of map Runtime lookup of endpoint Route using static resolver

20. BizTalk ESB – Example Scenario Declarative, Meta-data, Policy and Configuration –Driven. Transform Service RoutingRouting Process Orchestration ProtocolAdaptationProtocolAdaptation End Point Resolution Pub/Sub Service Service Consumers Service Providers 1.Message arrives on-ramp 2.Itinerary resolution 3.Transformation determined at runtime 4.Routing determined at runtime 5.Process the message 1.Message arrives on-ramp 2.Itinerary resolution 3.Transformation determined at runtime 4.Routing determined at runtime 5.Process the message On Ramp Off Ramp External Services: Transform my message External Services: Resolve a service end point address Itinerary injection Resolve endpoint Resolve mapping Key concepts: Itineraries, Resolvers

21. ESB - DEMO BizTalk Server 2009 ESB

22. Core Web Services Resolver Web Service Transformation Web Service UDDI Web Service Exception Web Service Operations Web Service BizTalk Send Ports Off-Ramps BizTalk Receive Ports On-Ramps BizTalk ESB Architecture Exception Management Store Exception ESB Management Portal Provisioning Framework Reports Alerts ESB Toolkit Core Itinerary Services Exception Management Framework Exception Logger Exception Handler Fault Processor Resolver-Adapter Provider Framework Resolvers (…) Adapter Providers(…) UDDI 3.0 BAM Generic SOAP Send Generic WCF Send Generic JMS Send Custom Send Pipeline Custom Pipeline ItineraryStoreItineraryStore Generic SOAP Receive Generic WCF Receive Generic JMS Receive Custom Receive Pipeline Custom Pipeline Route Service Custom Service Transform Service Transformation Engine Business Rules Engine Orchestration Engine BizTalk Pub/Sub Engine

23. Itinerary – Transform & Route Transform using rules resolver Runtime lookup of map Runtime lookup of endpoint Route using static resolver

24. On-Ramp – Selects itinerary

25. Cloud - Azure

26. AppFabric Service Bus Challenges in connectivity −Integrate on-premise ESB, apps via cloud −Bidirectional communication at Internet scope not trivial −Security Service Bus −Federated Identity and access control −Federated Naming −Dynamic Service Registry −Robust Messaging Fabric

27. BizTalk Cloud - Demo Architecture App Fabric Service Bus Receiver (EchoService) Echo Client (Sender) Echo Client (Sender) Backend Naming, Routing Fabric Outbound connect one-way net-tcp TCP/SSL 808/828 Msg TCP/SSL 828 Route Subscribe Frontend Nodes BizTalk ESB Itinerary Routing, dynamically transforms and resolves endpoint sb://biztalk-uk-conf.servicebus.windows.net/EchoService Outbound connect bi-directional socket

28. Start the service, visible in registry Start the service Outbound bidi connection established Service visible in Azure Service Registry

29. Connect via Client Application Connect via BizTalk Transform lookup using Business rules engine Resolve on-ramp and route Send message to cloud

30. BizTalk Service Bus Send Port BizTalk send port WCF Custom Binding

31. Evolving landscape Capture the invisible “Sense and respond” Diverse devices connected Process Mining and adaption

32. © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. Solutions with StreamInsight CEP 33 Data Sources, Operations, Assets, Feeds, Sensors, Devices Monitor & Record Monitor & Record Operational Data Store & Archive CEP Engine f(x) g(y) CEP Engine f(x) f'(x) g(y) h(x,y) History Deploy Results f'(x) h(x,y) Manage & Benefit Manage & Benefit Mine & Design Mine & Design Input Data Streams Output Data Streams

34. Deployment Scenarios CEP Engine Reference data Custom CEP Application Scenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application ISV Application with CEP Engine CEP Engine Reference data Scenario 4: Operational Intelligence w/ CEP Madison ETL Pipeline with CEP engine CEP Engine CEP KPIs KPI mining Scenario 3: CEP Enabled Device Device with Embedded CEP Engine CEP Engine.NET, C#, LINQ

35. BPM Maturity Explosion of BPM suites in the last decade Automation of structure processes Ad-hoc and semi-structure areas remain a challenge Other barriers to adoption: −Models out-of-date −Constant change of business −Events are often closer to reality than models Process Mining – links events to models −http://prom.win.tue.nl/research/wiki/_media/presentations/gartner- keynote-bpm2009-wvda.pdfProcess Mining Professor Wil van der Aalst, University of Eindhovenhttp://prom.win.tue.nl/research/wiki/_media/presentations/gartner- keynote-bpm2009-wvda.pdfProcess Mining

36. Growth of CEP Definition Gartner −In some business scenarios, such as business activity monitoring, the EDA substyle of SOA (potentially with CEP) enables a new class of globally scalable sense-and-respond applications for enhanced decision making and business effectiveness. − In 2012, more than 50% of advanced, new enterprise application systems will incorporate commercial CEP technologies to support near-real- time operational intelligence, intelligent flow management or automated sense-and-respond capabilities.