1. INNOV-17: How to Build Event Stream Processing (ESP) and Business Activity Monitoring (BAM) into Your Application
Dr John Bates
Dr Gareth Smith
VP Products
Progress Apama
Principal Architect
Progress Apama

2. Agenda Interactive Introduction to Apama Scenario Creation Definitions
What is Event Stream Processing (ESP)?
What is Business Activity Monitoring (BAM)?
Apama platform
Architecture & components
Building an example application
Revenue Assurance
Conclusions
INNOV-17: How to Build ESP and BAM into Your Application

3. Event Stream Processing A New Computing Physics
Static Data Processing: “How many shoes did we sell in our New York stores last week?”
Real-time intelligence allows a business to think differently about its operations and IT infrastructure because it can understand the state of a business in the now, rather than just in the past. Stream computing is a new style of computing that enables instant event pattern recognition, and it’s vastly different than traditional styles of computing.
Traditional computing is static computing, and static computing uses static data. Static data is like a photograph, which captures information about a moment in time. For example, static business data is a table of customer data, transactions at a retail store, or records of shipments that have occurred within a company’s supply chain. Static computing can be used to answer questions like: “How many shoes did we sell in our New York stores last week?” Over $200 billion of software is sold worldwide[1] each year, and almost all of it is designed for static data. Relational databases, for example, are designed to manage static data.
Real-time computing is stream computing[2], and stream computing uses events. Events arrive in streams that resemble a movie in which streams of images and sounds flow by your senses. Patterns within the stream of images and sounds can make you laugh, cry, or scream. Much like a movie, streaming events within an enterprise allow a business to feel the pulse of operations as events travel through its arteries – the ESB. With stream computing, a business can identify patterns and make instant decisions while they still matter: “When 3 transactions against the same credit card number occur within 5 seconds, deny the next request, flag the account, and send a message to the fraud detection dashboard.” And like a movie’s impact on its audience, these patterns might well make a CFO laugh, or cry.
Static computing and stream computing have fundamentally different physical characteristics – stream computing enables real-time, instant, intelligent decisions, based on the patterns of business operations. These patterns are determined by the temporal, causal, or spatial relationships among events within the stream.
[1] IDC, Worldwide IT Spending by Vertical Market Forecast Update: North America, Western Europe, Asia/Pacific, and Rest of World (IDC #32668). January 24, 2005.
[2] Stream computing has been an active field of academic research since the 1980’s, see for an index of academic motivation and characterization of this work. 1 2 3 4 5 6 7 8 9
time
Event Stream Processing: “When 3 credit card authorizations for the same card occur in any 5 second window, deny the request and check for fraud.”
INNOV-17: How to Build ESP and BAM into Your Application

4. ESP & BAM A New Computing Physics
Business Activity Monitoring: “Show on a dashboard (a) how many credit card transactions we have processed today (b) average & peak transactions per minute (c) how many fraudulent transactions per hour.”
Real-time intelligence allows a business to think differently about its operations and IT infrastructure because it can understand the state of a business in the now, rather than just in the past. Stream computing is a new style of computing that enables instant event pattern recognition, and it’s vastly different than traditional styles of computing.
Traditional computing is static computing, and static computing uses static data. Static data is like a photograph, which captures information about a moment in time. For example, static business data is a table of customer data, transactions at a retail store, or records of shipments that have occurred within a company’s supply chain. Static computing can be used to answer questions like: “How many shoes did we sell in our New York stores last week?” Over $200 billion of software is sold worldwide[1] each year, and almost all of it is designed for static data. Relational databases, for example, are designed to manage static data.
Real-time computing is stream computing[2], and stream computing uses events. Events arrive in streams that resemble a movie in which streams of images and sounds flow by your senses. Patterns within the stream of images and sounds can make you laugh, cry, or scream. Much like a movie, streaming events within an enterprise allow a business to feel the pulse of operations as events travel through its arteries – the ESB. With stream computing, a business can identify patterns and make instant decisions while they still matter: “When 3 transactions against the same credit card number occur within 5 seconds, deny the next request, flag the account, and send a message to the fraud detection dashboard.” And like a movie’s impact on its audience, these patterns might well make a CFO laugh, or cry.
Static computing and stream computing have fundamentally different physical characteristics – stream computing enables real-time, instant, intelligent decisions, based on the patterns of business operations. These patterns are determined by the temporal, causal, or spatial relationships among events within the stream.
[1] IDC, Worldwide IT Spending by Vertical Market Forecast Update: North America, Western Europe, Asia/Pacific, and Rest of World (IDC #32668). January 24, 2005.
[2] Stream computing has been an active field of academic research since the 1980’s, see for an index of academic motivation and characterization of this work. 1 2 3 4 5 6 7 8 9
time
Event Stream Processing: “When 3 credit card authorizations for the same card occur in any 5 second window, deny the request and check for fraud.”
INNOV-17: How to Build ESP and BAM into Your Application

5. Event-Driven Analysis & Action
Example Event Driven Applications Monitor, Analyze, and Act on Business Conditions in Real Time
Application
Event Examples
Event-Driven Analysis & Action
Customer Service
Customer requests, CRM records
Alert me if an open customer request with priority > 3 did not get any reply event for at least 2 hours
Alert me if an open customer request was reassigned more than 3 times
Supply Chain
Inventory system, warehouse deliveries, warehouse restock
When (Warehouse 1 capacity < 50%) and (Warehouse 2 capacity < 30%) and
(Warehouse 3 capacity < 25%) Alert the head of operations with the average capacity in each of the warehouses
RFID/Supply Chain
RFID reader detecting EPC codes, goods delivery schedule
When pallet is loaded onto the wrong truck alert warehouse manager
Automated Trading
Stock tick & quote data
Within any 20 second window, when HP rises by more than 2%, but IBM doesn’t, buy IBM
Transportation
Smartcard ID read
When the rider travels more than 5 times within zone 1, charge at single-ride rate for subsequent trips
Events are everywhere, and increasingly, businesses today need to answer questions in real-time in reaction to the events they detect, just as the human body’s nervous system reacts to the environment around it.
In financial services, an example of an event is tick on a market feed that indicates that JP Morgan is offering to buy IBM at $82; an event-driven decision would be to sell 10,000 shares of IBM to JP Morgan at $82, based on a real-time analysis of a firm’s outstanding orders for IBM and the 15-minute moving average of IBM stock to determine if they should sell IBM to JP Morgan now, and, if so, at what price.
Also in financial services, firms today assess their portfolio value-at-risk (VoR) in real-time, and make decisions about their trading behavior based on an up-to-the-second assessment of the value they have at risk. For example, as trade events are emitted throughout the enterprise, the firm’s VoR system can monitor trades, analyze the portfolio risk exposure, and set pricing according to internal and regulated guidelines.
In numerous industries, from the travel industry to telecommunications, customers expect to “pay on demand” for services. “Micro payments” enable a firm to charge for services on demand. For example, travelers with smart cards can fill an account with a card that allows them unlimited travel within a specific subway zone. To properly charge for these services, billing is often applied immediately, while events that provide context (where the traveller is, where she has been, the current status of the account, etc.) are available for analysis.
Along with increasing automation comes increasing ways to commit fraudulent activity. Real-time billing, when combined with real-time fraud detection, is critical to stop aberrant behavior before it occurs. The classic example of this is credit card fraud detection. Events are requests for a charge; the real-time challenge is to detect when more than one charge is requested for the same credit card within a small time window – an indication that computer-based fraud is occurring, with multiple charges being sent via multiple businesses at the same time.
Radio Frequency Identification systems emit tiny events that represent movement (or absence of movement) so that physical assets of all types – from pallets of Gillette razor blades to newborn babies in a hospital ward – can be tracked and traced in real time. In the supply chain, the ability to monitor events being emitted by RFID readers that indicate that a pallet of items have moved through dock door #5, containing a pallet of razor blades, is an example of a real-time event. The automation of the handling process – an instruction to move this pallet to holding bay 231 – can be automated by an event-driven application that understands the layout of the distribution center, the expected shipments, and is interfaced to a warehouse management system that understands the flow of items with the distribution center.
The government is full of event-driven applications. As modern military systems become increasingly electronic, the challenge of command and control
INNOV-17: How to Build ESP and BAM into Your Application

6. What do events look like?
Events are typed messages describing a change within a system
Example:
NewInvoice(customer, product, amount)
StockTick(symbol, price, volume)
TruckLocation(truck, driver, cargo, x,y,z)
Events are analogous to a new entry in a database table or a Sonic message
Detecting patterns in individual events or a correlation of several events over time can indicate an opportunity or threat to the business
INNOV-17: How to Build ESP and BAM into Your Application

7. Business Benefits of ESP
Monitor, Analyze, ACT
The ability to take an action immediately when a business scenario is detected
Enables your application to respond instantly to opportunity or threat
It’s easy to add powerful capabilities to your existing application
You already have the events flowing in your system – you just need to send them into Apama
Add real-time rules to your application without continually changing your code
You don’t have to go on extending your application
Once the events are flowing into Apama, simply add or modify a rule in Apama
Rules can be defined by business or technical users
Codeless and code based development
INNOV-17: How to Build ESP and BAM into Your Application

8. Agenda Interactive Introduction to Apama Scenario Creation Definitions
What is Event Stream Processing (ESP)?
What is Business Activity Monitoring (BAM)?
Apama Platform
Architecture & components
Building an example application
Revenue Assurance
Conclusions
INNOV-17: How to Build ESP and BAM into Your Application

9. Integration Adapter Framework
Apama Product Suite
Dashboards
Developer Studio
Research Studio
Dashboard Studio
Scenario Modeler
Apama IDE
Event Manager
Event Store
Integration Adapter Framework
INNOV-17: How to Build ESP and BAM into Your Application

10. Inside the Event Manager
Feedback
Event Registration
Event Input Queue
Multi-dimensional event matcher
Composite event sequencer
Event Output Queue
MonitorScript
Virtual Machine
Events
Events
Java
Virtual Machine
Match Notification
C/C++
Scenarios
INNOV-17: How to Build ESP and BAM into Your Application

11. Monitorscript event SMS-Request { string user; string service;
location loc; }
monitor RingtoneServiceMonitor {
SMS-Request request;
action onload {
on all SMS-Request (service=“Ringtone"):request {
on SMS-Request (user = request.user) within(60.0) {
emit UpdateBilling(request.user, “Double Discount”); }
INNOV-17: How to Build ESP and BAM into Your Application

12. JMON: Monitors in Java import com.apama.jmon.*;
public class BillRequestService implements Monitor, MatchListener {
public void onLoad() {
EventExpression eventexpr =
new EventExpression(“SMS-Request(service=\“Billing\”)”);
eventexpr.addMatchListener(this); }
public void match(MatchEvent event) {
System.out.println(“Billing Information Requested”);
INNOV-17: How to Build ESP and BAM into Your Application

13. Apama IDE
INNOV-17: How to Build ESP and BAM into Your Application

14. The Apama Scenario Modeler Enables Graphical Construction of ESP Scenarios
Express time-based real-time rules with a high level development tool
Each scenario, or group of rules, represents a “pattern” which can be adjusted by business users to specify conditions to monitor, analyze and act on.
Intuitive visual user interface designed for business analysts
“SmartBlocks” encapsulate pre-packaged modules made available to non-programmers.
INNOV-17: How to Build ESP and BAM into Your Application

15. Apama Dashboard Studio Create Customized Dashboards for Your Real-Time Business Processes
Real-time variables and analytics can be visualized using graphs, charts, tables etc.
Enables event-driven Apama logic to be visualized in real-time
Deployment options from standalone dashboard to thin client portal
Select from a palette of graphical objects. Each object can be laid out in a graphical dashboard and bound to Apama events
Users can customize the look and feel of all widgets, and specify which Apama event scenarios to visualize.
INNOV-17: How to Build ESP and BAM into Your Application

16. Event Store - Capture
Apama Dashboards
Event Store captures all events in time series order
Event Store is also able to capture derived events, e.g. Moving Average, from Event Manager
Event Store can capture all output from Event Manager as an audit trail
Real-Time Event Processing
Apama
EventStore
Historical Event Processing
INNOV-17: How to Build ESP and BAM into Your Application

17. Event Store – Replay & Backtesting
Apama Research Studio
Apama Dashboards
Business Intelligence
Enables “pre-flight testing” of Apama scenarios with historical data sequences
Output of runs can be captured in Event Store and charted
Also enables “digital forensics” – using recorded information to tune future performance
Real-Time Event Processing
Apama
EventStore
Historical Event Processing
INNOV-17: How to Build ESP and BAM into Your Application

18. Integration Adapter Framework
Purpose of an adapter
Receive an event stream or create an event stream
Intercept transactions
Use database triggers
Subscribe to Sonic messages
Normalize to Apama event format for efficient analysis & response
Convert Apama events back to actionable messages
Event Manager
Mapping
Apama Events
Normalization
Transport Integration
API
Publish/ subscribe
OpenEdge
Sonic Bus
Native Sonic Messages
INNOV-17: How to Build ESP and BAM into Your Application

19. Agenda Interactive Introduction to Apama Scenario Creation Definitions
What is Event Stream Processing (ESP)?
What is Business Activity Monitoring (BAM)?
Apama Platform
Architecture & components
Building an example application
Revenue Assurance
Conclusions
INNOV-17: How to Build ESP and BAM into Your Application

20. Building an Example Application
Case Study: Revenue Assurance
Ensuring that all service requests are both fulfilled and billed correctly and within a timely manner
Monitor via complex KPIs and present an graphical overview of the status
Alert relevant staff with specific problems
Undertake Autonomous Actions: modify the Service QoS settings to resolve problems
Same underlying principles are valid for tracking & billing in Supply Chain, ERP, Logistics etc.
INNOV-17: How to Build ESP and BAM into Your Application

21. A (very) simplified revenue chain1 2
User’s mobile
Switch
(SMSC, MSC,
SIP server)
Application
Server(s)
Billing 3 5 4
Process
User requests a service (1 + 2)
Service is fulfilled (4 + 5)
Request is Billed (3)
Problem
Any message (1-5) may be lost, corrupt or delayed
Depends on the throughput and the QoS
Parameters (Complex-KPIs) that we calculate:
Revenue leakage: requests that are not billed
Overcharging: requests that are billed but not fulfilled
INNOV-17: How to Build ESP and BAM into Your Application

22. Our revenue assurance demo package
Dashboard
- Display what’s going on
- Monitor the alarms
- Notify leakages that can’t be fixed
Scenario
- Calculate parameters
- Apply treshold and raise alarms
- Take corrective actions (e.g. adjust QoS)
Smart blocks
Reconciliation
Value Added
Service
Billing 1 2
User’s mobile
Switch
(SMSC, MSC,
SIP server)
Application
Server(s)
Billing 3 5 4
INNOV-17: How to Build ESP and BAM into Your Application

23. Demonstration
INNOV-17: How to Build ESP and BAM into Your Application

24. Agenda Interactive Introduction to Apama Scenario Creation Definitions
What is Event Stream Processing (ESP)?
What is Business Activity Monitoring (BAM)?
Apama Platform
Architecture & components
Building an example application
Revenue Assurance
Conclusions
INNOV-17: How to Build ESP and BAM into Your Application

25. Conclusions ESP & BAM enables
Rules to be defined external to the application to monitor, alert & respond to key business opportunities & threats
Vizualization of key business events on real-time dashboards
In many cases events are already flowing through existing applications
Apama enables
Easy connection to existing applications
Graphical ESP rules definition for rapid development
Graphical dashboard creation
INNOV-17: How to Build ESP and BAM into Your Application

26. Questions?
INNOV-17: How to Build ESP and BAM into Your Application

27. Thank you for your time
INNOV-17: How to Build ESP and BAM into Your Application

28. INNOV-17: How to Build ESP and BAM into Your Application