INNOV-17: How to Build Event Stream Processing (ESP) and Business Activity Monitoring (BAM) into Your Application Dr John Bates Dr Gareth Smith VP Products Progress Apama Principal Architect Progress Apama

Agenda Interactive Introduction to Apama Scenario Creation Definitions What is Event Stream Processing (ESP)? What is Business Activity Monitoring (BAM)? Apama platform Architecture & components Building an example application Revenue Assurance Conclusions INNOV-17: How to Build ESP and BAM into Your Application

Event Stream Processing A New Computing Physics Static Data Processing: “How many shoes did we sell in our New York stores last week?” Real-time intelligence allows a business to think differently about its operations and IT infrastructure because it can understand the state of a business in the now, rather than just in the past. Stream computing is a new style of computing that enables instant event pattern recognition, and it’s vastly different than traditional styles of computing. Traditional computing is static computing, and static computing uses static data. Static data is like a photograph, which captures information about a moment in time. For example, static business data is a table of customer data, transactions at a retail store, or records of shipments that have occurred within a company’s supply chain. Static computing can be used to answer questions like: “How many shoes did we sell in our New York stores last week?” Over $200 billion of software is sold worldwide[1] each year, and almost all of it is designed for static data. Relational databases, for example, are designed to manage static data. Real-time computing is stream computing[2], and stream computing uses events. Events arrive in streams that resemble a movie in which streams of images and sounds flow by your senses. Patterns within the stream of images and sounds can make you laugh, cry, or scream. Much like a movie, streaming events within an enterprise allow a business to feel the pulse of operations as events travel through its arteries – the ESB. With stream computing, a business can identify patterns and make instant decisions while they still matter: “When 3 transactions against the same credit card number occur within 5 seconds, deny the next request, flag the account, and send a message to the fraud detection dashboard.” And like a movie’s impact on its audience, these patterns might well make a CFO laugh, or cry. Static computing and stream computing have fundamentally different physical characteristics – stream computing enables real-time, instant, intelligent decisions, based on the patterns of business operations. These patterns are determined by the temporal, causal, or spatial relationships among events within the stream. [1] IDC, Worldwide IT Spending 2004-2008 by Vertical Market Forecast Update: North America, Western Europe, Asia/Pacific, and Rest of World (IDC #32668). January 24, 2005. [2] Stream computing has been an active field of academic research since the 1980’s, see www.eventstreamprocessing.com for an index of academic motivation and characterization of this work. 1 2 3 4 5 6 7 8 9 time Event Stream Processing: “When 3 credit card authorizations for the same card occur in any 5 second window, deny the request and check for fraud.” INNOV-17: How to Build ESP and BAM into Your Application

ESP & BAM A New Computing Physics Business Activity Monitoring: “Show on a dashboard (a) how many credit card transactions we have processed today (b) average & peak transactions per minute (c) how many fraudulent transactions per hour.” Real-time intelligence allows a business to think differently about its operations and IT infrastructure because it can understand the state of a business in the now, rather than just in the past. Stream computing is a new style of computing that enables instant event pattern recognition, and it’s vastly different than traditional styles of computing. Traditional computing is static computing, and static computing uses static data. Static data is like a photograph, which captures information about a moment in time. For example, static business data is a table of customer data, transactions at a retail store, or records of shipments that have occurred within a company’s supply chain. Static computing can be used to answer questions like: “How many shoes did we sell in our New York stores last week?” Over $200 billion of software is sold worldwide[1] each year, and almost all of it is designed for static data. Relational databases, for example, are designed to manage static data. Real-time computing is stream computing[2], and stream computing uses events. Events arrive in streams that resemble a movie in which streams of images and sounds flow by your senses. Patterns within the stream of images and sounds can make you laugh, cry, or scream. Much like a movie, streaming events within an enterprise allow a business to feel the pulse of operations as events travel through its arteries – the ESB. With stream computing, a business can identify patterns and make instant decisions while they still matter: “When 3 transactions against the same credit card number occur within 5 seconds, deny the next request, flag the account, and send a message to the fraud detection dashboard.” And like a movie’s impact on its audience, these patterns might well make a CFO laugh, or cry. Static computing and stream computing have fundamentally different physical characteristics – stream computing enables real-time, instant, intelligent decisions, based on the patterns of business operations. These patterns are determined by the temporal, causal, or spatial relationships among events within the stream. [1] IDC, Worldwide IT Spending 2004-2008 by Vertical Market Forecast Update: North America, Western Europe, Asia/Pacific, and Rest of World (IDC #32668). January 24, 2005. [2] Stream computing has been an active field of academic research since the 1980’s, see www.eventstreamprocessing.com for an index of academic motivation and characterization of this work. 1 2 3 4 5 6 7 8 9 time Event Stream Processing: “When 3 credit card authorizations for the same card occur in any 5 second window, deny the request and check for fraud.” INNOV-17: How to Build ESP and BAM into Your Application

Event-Driven Analysis & Action Example Event Driven Applications Monitor, Analyze, and Act on Business Conditions in Real Time Application Event Examples Event-Driven Analysis & Action Customer Service Customer requests, CRM records Alert me if an open customer request with priority > 3 did not get any reply event for at least 2 hours Alert me if an open customer request was reassigned more than 3 times Supply Chain Inventory system, warehouse deliveries, warehouse restock When (Warehouse 1 capacity < 50%) and (Warehouse 2 capacity < 30%) and (Warehouse 3 capacity < 25%) Alert the head of operations with the average capacity in each of the warehouses RFID/Supply Chain RFID reader detecting EPC codes, goods delivery schedule When pallet is loaded onto the wrong truck alert warehouse manager Automated Trading Stock tick & quote data Within any 20 second window, when HP rises by more than 2%, but IBM doesn’t, buy IBM Transportation Smartcard ID read When the rider travels more than 5 times within zone 1, charge at single-ride rate for subsequent trips Events are everywhere, and increasingly, businesses today need to answer questions in real-time in reaction to the events they detect, just as the human body’s nervous system reacts to the environment around it. In financial services, an example of an event is tick on a market feed that indicates that JP Morgan is offering to buy IBM at $82; an event-driven decision would be to sell 10,000 shares of IBM to JP Morgan at $82, based on a real-time analysis of a firm’s outstanding orders for IBM and the 15-minute moving average of IBM stock to determine if they should sell IBM to JP Morgan now, and, if so, at what price. Also in financial services, firms today assess their portfolio value-at-risk (VoR) in real-time, and make decisions about their trading behavior based on an up-to-the-second assessment of the value they have at risk. For example, as trade events are emitted throughout the enterprise, the firm’s VoR system can monitor trades, analyze the portfolio risk exposure, and set pricing according to internal and regulated guidelines. In numerous industries, from the travel industry to telecommunications, customers expect to “pay on demand” for services. “Micro payments” enable a firm to charge for services on demand. For example, travelers with smart cards can fill an account with a card that allows them unlimited travel within a specific subway zone. To properly charge for these services, billing is often applied immediately, while events that provide context (where the traveller is, where she has been, the current status of the account, etc.) are available for analysis. Along with increasing automation comes increasing ways to commit fraudulent activity. Real-time billing, when combined with real-time fraud detection, is critical to stop aberrant behavior before it occurs. The classic example of this is credit card fraud detection. Events are requests for a charge; the real-time challenge is to detect when more than one charge is requested for the same credit card within a small time window – an indication that computer-based fraud is occurring, with multiple charges being sent via multiple businesses at the same time. Radio Frequency Identification systems emit tiny events that represent movement (or absence of movement) so that physical assets of all types – from pallets of Gillette razor blades to newborn babies in a hospital ward – can be tracked and traced in real time. In the supply chain, the ability to monitor events being emitted by RFID readers that indicate that a pallet of items have moved through dock door #5, containing a pallet of razor blades, is an example of a real-time event. The automation of the handling process – an instruction to move this pallet to holding bay 231 – can be automated by an event-driven application that understands the layout of the distribution center, the expected shipments, and is interfaced to a warehouse management system that understands the flow of items with the distribution center. The government is full of event-driven applications. As modern military systems become increasingly electronic, the challenge of command and control INNOV-17: How to Build ESP and BAM into Your Application

What do events look like? Events are typed messages describing a change within a system Example: NewInvoice(customer, product, amount) StockTick(symbol, price, volume) TruckLocation(truck, driver, cargo, x,y,z) Events are analogous to a new entry in a database table or a Sonic message Detecting patterns in individual events or a correlation of several events over time can indicate an opportunity or threat to the business INNOV-17: How to Build ESP and BAM into Your Application

Business Benefits of ESP Monitor, Analyze, ACT The ability to take an action immediately when a business scenario is detected Enables your application to respond instantly to opportunity or threat It’s easy to add powerful capabilities to your existing application You already have the events flowing in your system – you just need to send them into Apama Add real-time rules to your application without continually changing your code You don’t have to go on extending your application Once the events are flowing into Apama, simply add or modify a rule in Apama Rules can be defined by business or technical users Codeless and code based development INNOV-17: How to Build ESP and BAM into Your Application

Agenda Interactive Introduction to Apama Scenario Creation Definitions What is Event Stream Processing (ESP)? What is Business Activity Monitoring (BAM)? Apama Platform Architecture & components Building an example application Revenue Assurance Conclusions INNOV-17: How to Build ESP and BAM into Your Application

Integration Adapter Framework Apama Product Suite Dashboards Developer Studio Research Studio Dashboard Studio Scenario Modeler Apama IDE Event Manager Event Store Integration Adapter Framework INNOV-17: How to Build ESP and BAM into Your Application

Inside the Event Manager Feedback Event Registration Event Input Queue Multi-dimensional event matcher Composite event sequencer Event Output Queue MonitorScript Virtual Machine Events Events Java Virtual Machine Match Notification C/C++ Scenarios INNOV-17: How to Build ESP and BAM into Your Application

Monitorscript event SMS-Request { string user; string service; location loc; } monitor RingtoneServiceMonitor { SMS-Request request; action onload { on all SMS-Request (service=“Ringtone"):request { on SMS-Request (user = request.user) within(60.0) { emit UpdateBilling(request.user, “Double Discount”); } INNOV-17: How to Build ESP and BAM into Your Application

JMON: Monitors in Java import com.apama.jmon.*; public class BillRequestService implements Monitor, MatchListener { public void onLoad() { EventExpression eventexpr = new EventExpression(“SMS-Request(service=\“Billing\”)”); eventexpr.addMatchListener(this); } public void match(MatchEvent event) { System.out.println(“Billing Information Requested”); INNOV-17: How to Build ESP and BAM into Your Application

Apama IDE INNOV-17: How to Build ESP and BAM into Your Application

The Apama Scenario Modeler Enables Graphical Construction of ESP Scenarios Express time-based real-time rules with a high level development tool Each scenario, or group of rules, represents a “pattern” which can be adjusted by business users to specify conditions to monitor, analyze and act on. Intuitive visual user interface designed for business analysts “SmartBlocks” encapsulate pre-packaged modules made available to non-programmers. INNOV-17: How to Build ESP and BAM into Your Application

Apama Dashboard Studio Create Customized Dashboards for Your Real-Time Business Processes Real-time variables and analytics can be visualized using graphs, charts, tables etc. Enables event-driven Apama logic to be visualized in real-time Deployment options from standalone dashboard to thin client portal Select from a palette of graphical objects. Each object can be laid out in a graphical dashboard and bound to Apama events Users can customize the look and feel of all widgets, and specify which Apama event scenarios to visualize. INNOV-17: How to Build ESP and BAM into Your Application

Event Store - Capture Apama Dashboards Event Store captures all events in time series order Event Store is also able to capture derived events, e.g. Moving Average, from Event Manager Event Store can capture all output from Event Manager as an audit trail Real-Time Event Processing Apama EventStore Historical Event Processing INNOV-17: How to Build ESP and BAM into Your Application

Event Store – Replay & Backtesting Apama Research Studio Apama Dashboards Business Intelligence Enables “pre-flight testing” of Apama scenarios with historical data sequences Output of runs can be captured in Event Store and charted Also enables “digital forensics” – using recorded information to tune future performance Real-Time Event Processing Apama EventStore Historical Event Processing INNOV-17: How to Build ESP and BAM into Your Application

Integration Adapter Framework Purpose of an adapter Receive an event stream or create an event stream Intercept transactions Use database triggers Subscribe to Sonic messages Normalize to Apama event format for efficient analysis & response Convert Apama events back to actionable messages Event Manager Mapping Apama Events Normalization Transport Integration API Publish/ subscribe OpenEdge Sonic Bus Native Sonic Messages INNOV-17: How to Build ESP and BAM into Your Application

Agenda Interactive Introduction to Apama Scenario Creation Definitions What is Event Stream Processing (ESP)? What is Business Activity Monitoring (BAM)? Apama Platform Architecture & components Building an example application Revenue Assurance Conclusions INNOV-17: How to Build ESP and BAM into Your Application

Building an Example Application Case Study: Revenue Assurance Ensuring that all service requests are both fulfilled and billed correctly and within a timely manner Monitor via complex KPIs and present an graphical overview of the status Alert relevant staff with specific problems Undertake Autonomous Actions: modify the Service QoS settings to resolve problems Same underlying principles are valid for tracking & billing in Supply Chain, ERP, Logistics etc. INNOV-17: How to Build ESP and BAM into Your Application

A (very) simplified revenue chain 1 2 User’s mobile Switch (SMSC, MSC, SIP server) Application Server(s) Billing 3 5 4 Process User requests a service (1 + 2) Service is fulfilled (4 + 5) Request is Billed (3) Problem Any message (1-5) may be lost, corrupt or delayed Depends on the throughput and the QoS Parameters (Complex-KPIs) that we calculate: Revenue leakage: requests that are not billed Overcharging: requests that are billed but not fulfilled INNOV-17: How to Build ESP and BAM into Your Application

Our revenue assurance demo package Dashboard - Display what’s going on - Monitor the alarms - Notify leakages that can’t be fixed Scenario - Calculate parameters - Apply treshold and raise alarms - Take corrective actions (e.g. adjust QoS) Smart blocks Reconciliation Value Added Service Billing 1 2 User’s mobile Switch (SMSC, MSC, SIP server) Application Server(s) Billing 3 5 4 INNOV-17: How to Build ESP and BAM into Your Application

Demonstration INNOV-17: How to Build ESP and BAM into Your Application

Agenda Interactive Introduction to Apama Scenario Creation Definitions What is Event Stream Processing (ESP)? What is Business Activity Monitoring (BAM)? Apama Platform Architecture & components Building an example application Revenue Assurance Conclusions INNOV-17: How to Build ESP and BAM into Your Application

Conclusions ESP & BAM enables Rules to be defined external to the application to monitor, alert & respond to key business opportunities & threats Vizualization of key business events on real-time dashboards In many cases events are already flowing through existing applications Apama enables Easy connection to existing applications Graphical ESP rules definition for rapid development Graphical dashboard creation INNOV-17: How to Build ESP and BAM into Your Application

Questions? INNOV-17: How to Build ESP and BAM into Your Application

Thank you for your time INNOV-17: How to Build ESP and BAM into Your Application

INNOV-17: How to Build ESP and BAM into Your Application