Complex Event Processing: Power your middleware with StreamInsight Mahesh Patel (Microsoft) Amit Bansal (PeoplewareIndia.com)

Agenda The Value of Timely Analytics The challenges / Scenarios Introduction to Complex Event Processing Introducing Microsoft StreamInsight Brief introduction to BizTalk Server Understand the key scenarios that StreamInsight can unlock for BizTalk Map out the architectural, integration requirements for BizTalk + StreamInsight.

The value of timely analytics

Current Products for analytics

Introduction to CEP Complex Event Processing (CEP) is an architecture style based on the principles of Event Driven Architecture (EDA). Continuously processing a high-volume stream of events from various event sources Events are logically grouped in sequences (typically known as streams) based on a time defined criteria such as an interval. Event streams are processed through a series of queries

Capabilities of a CEP Application Event Driven Continuous Queries Time Sensitive Low Latency High Volume

The value of timely analytics

Scenarios Stream Data Store & Archive Event Processing Engine Data Stream Asset Specs & Parameters Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Web Analytics: Click-stream data Online customer behavior Page layout 100,000 events /sec Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Threshold queries Event correlation from multiple sources Pattern queries Threshold queries Event correlation from multiple sources Pattern queries Lookup Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec Data Stream

Introducing StreamInsight

DEMO CEP using StreamInsight

Traditional messaging applications: “High signal” – every message is business relevant Rich data schemas – sets of related information Workflow centric and transactional Emerging data sources and trends: Machine born data is growing at a very rapid rate Not all of this data is business relevant (low signal/noise ratio) How do we: Identify and extract business relevant events from streaming data? Take action on these business insights (line of business, human workflow, etc) Messages and Events LOB Large Messages Low Volume High Volume Small Events Web Services B22B SWIFT RFID Stock Feeds Electrical Grids Operational Data Health Care CRM

Send Port MessageBox Orchestration Receive Port Message Maps Receive Pipeline Adapter (HTTP) Message LOB Host Send Port Adapter (File) Map Send Pipeline Component Basic BizTalk Architecture Component Host Management DBTracking DB Message Agent

Scenarios Good BizTalk + StreamInsight scenarios: Streaming data/event sources – high noise to signal ratio Need to analyze and extract business relevant events Integrate and transact with line of business applications.

Scenarios Manufacturing (Oil and Gas) Monitor plant floor processes Identify business relevant events Transact with LOB apps Utilities Automated metering – handle flood of data from utility meters Identify business relevant events Transact with LOB apps The data feeds change, the question changes – the core scenario doesn’t!

OSIsoft PI & StreamInsight OSIsoft PI - Industry leading process data management and historian. Integrating and leveraging StreamInsight OSIsoft shipping StreamInsight adapter for PI. Stream data in and out of PI Extract powerful business insights from PI data Integrate with LOB applications using BizTalk Server

OPC & StreamInsight (with MatrikonOPC) OPC –Industry standard for process automation and data exchange MatrikonOPC – Industry leader in OPC interfaces Developed StreamInsight adapters for their OPC implementation Extract powerful business insights from OPC data Integrate with LOB applications using BizTalk Server

Automated Metering Contoso Utilities is rolling out smart electrical meters to their customers. Meters report power usage (wattage, voltage) at a “point in time”. Data is consolidated into a back end application used for billing and forecasting. The volume of data has forced Contoso to only read values from meters every hour They want to: Monitor the data more closely, compressing the point stream to an interval stream before loading into the back end application. i.e. if the data does not change by more than 2% over a 10 minute window, keep the average value. Capture spikes and threshold events, route to a process control application for further inspection.

Automated Metering Collect high volume streaming data from smart meters Data consists of timestamp, source ID and payload (power in Watts) Sample operations on the streaming data: Consolidate stream of points events into 5 minute “chunks”; record average, count and standard deviation. Detect anomalies (threshold 10%), collect last 5 minutes worth of data for that meter and all meters fed from that substation Need to link meter ID with reference data (location, substation) Results must be integrated into line of business application

Automated Metering

Architecture – Message Integration WCF Adapters BizTalk Host OrchestrationOrchestration Business Activity Monitoring Business Rules Engine Message Box StreamInsight XmlPayload Output Adapter QueriesQueries Input Adapters StreamInsight extracts insights from real time data feeds, feeds aggregate information to BizTalk. 22 A strongly typed XML schema is inferred from the StreamInsight query 33 Batched and correlated query results are delivered to a BizTalk receive port.

StreamInsight is an in-memory event processing engine. Do not deploy StreamInsight instance on the same hardware as your BizTalk host instances. Leverage StreamInsight-local MSMQ for message transfer. Deployment – Message Integration

Summary BizTalk Server and StreamInsight together can unlock powerful scenarios: High volume data processing Message integration Leverage StreamInsight to: Consolidation and analyze incoming data streams convert event sets into messages Deliver to BizTalk receive port Leverage BizTalk Server to: Integrate these insights into line of Business applications Other potential alignment points: Business Activity Monitoring (BAM). Tap into BAM event events stream for rich analytics and event detections Business Rules Engine (BRE). Leverage BRE to define event pattern rules

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.