Marcelo R.N. Mendes

What is FINCoS? A set of tools for data generation, load submission, and performance measurement of CEP systems; Main Characteristics: Flexible; Independent of particular workloads; Neutral (not bound to any specific CEP product); Correctness check; Scalable. 2CISUC, University of Coimbra

FINCoS: Architecture 3CISUC, University of Coimbra Scalability: It is easy to increase the load: simply add more Drivers! Validation and Performance Measuring: Plug-and-Play Validators Independence of CEP Products and workloads Flexibility: Several Test Options

FINCoS: Components Controller Main Interface with the user. Allows to: Create reusable test setups; Control Drivers and Sinks (e.g., load, start, stop); Alter workload parameters on-the- fly (e.g., event submission rates, switch phases) 4CISUC, University of Coimbra

FINCoS: Components Driver Simulates external sources of events; Submits load to the SUT; Includes data generation mechanism; Sink Receives query results from CEP engine; Stores output events in log files or send them via network for subsequent validation; 5CISUC, University of Coimbra

FINCoS: Components Performance Monitor Collect Metrics of Performance Runs Two Modes: Online (network); Offline (log files). 6CISUC, University of Coimbra

FINCoS: Components January, 2012CISUC, University of Coimbra - Adapter On Input: converts from the framework neutral representation (CSV) to CEP system’s native representation On Ouput: converts output events from CEP system’s native format to neutral representation (CSV)

Lesson 1: Creating a Test Setup 1. Configure a Driver 1. Decomposable Workload 2. Synthetic vs. Thirdy-Party Dataset 3. Data Generation Options 4. Event Arrival Process 2. Configure a Sink January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - unique identifier number of threads used for load generation save generated data to disk? real time performance monitoring? to where Driver send the events machine that hosts the Driver

January, 2012CISUC, University of Coimbra - Workload can be split in independent phases

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Chooses the type of workload for this phase Sets events arrival process Event Mix Events are generated in a ordered way When data generation happens To ensure repeatability…

January, 2012CISUC, University of Coimbra - Configure event types…

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Configure Event Type Schema

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Column Name

January, 2012CISUC, University of Coimbra - Column Type

January, 2012CISUC, University of Coimbra - Data Generation Options

January, 2012CISUC, University of Coimbra - Add Values

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - unique identifier List of event types this Sink subscribes to from where Sink receive events machine that hosts the Sink Port used by Sink to receive events save generated data to disk? real time performance monitoring?

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Done!

Lesson 2: Connecting with a CEP engine 1. Configure Connection Properties 2. Load FINCoS Adapter application January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - List of input streamsList of output streams No Information Available Yet!

January, 2012CISUC, University of Coimbra - Retrieve list of streams…

January, 2012CISUC, University of Coimbra - … from the setup file created earlier

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Load vendor-specific connection properties …

January, 2012CISUC, University of Coimbra - Engine=Esper Configuration_file_path=C:\\FINCoS\\queries\\esper\\Q1\\Stream_Set.xml Queries_path=C:\\FINCoS\\queries\\esper\\Q1\\Query_Set.xml Event_format=Map Engine=Esper Configuration_file_path=C:\\FINCoS\\queries\\esper\\Q1\\Stream_Set.xml Queries_path=C:\\FINCoS\\queries\\esper\\Q1\\Query_Set.xml Event_format=Map

January, 2012CISUC, University of Coimbra - Connect to CEP engine…

January, 2012CISUC, University of Coimbra - The list of streams in the CEP engine is retrieved and matched against the configured ones in the test setup

January, 2012CISUC, University of Coimbra - Loads the Adapter: 1.Open Server Socket for Drivers (Input) 2.Subscribes to streams at the CEP engine (Output)

January, 2012CISUC, University of Coimbra - Done!

Lesson 3: Running Tests 1. Load Drivers and Sinks 2. Start Load Submission 3. Control Load Submission NOTE: At this point the following steps must have been completed: 1. Creation of a test setup; 2. Connection with a CEP engine; 3. Initialization of the FINCoS Daemon Service at the machines where Drivers and Sinks are intended to run; January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Load components

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Start Load Submission

January, 2012CISUC, University of Coimbra - Sent/Received counts

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

Lesson 4: Collecting Performance Metrics 1. Offline Performance Measurement 1. After test completion, use FINCoS Performance Monitor to process log files 2. Online Performance Measurement 1. Enable realtime monitoring at Drivers and Sinks; 2. Initialize FINCoS Performance Monitor in the Realtime Mode before test starts. January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Offline Performance Measurement

January, 2012CISUC, University of Coimbra - Add log files

January, 2012CISUC, University of Coimbra - Log file created after test completion

January, 2012CISUC, University of Coimbra - Configure M.I. here: Optionally, save collected metrics to a file Process log files

January, 2012CISUC, University of Coimbra - View Results

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Select the stream

January, 2012CISUC, University of Coimbra - Select the metric

January, 2012CISUC, University of Coimbra - Select exhibition scale

January, 2012CISUC, University of Coimbra - Add Metrics to the Chart

January, 2012CISUC, University of Coimbra - Change Visualization Options

Lesson 4: Collecting Performance Metrics (...cont) 1. Offline Performance Measurement 1. After test completion, use FINCoS Performance Monitor to process log files 2. Online Performance Measurement 1. Enable realtime monitoring at Drivers and Sinks; 2. Initialize FINCoS Performance Monitor in the Realtime Mode before test starts. January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Edit Driver Configuration

January, 2012CISUC, University of Coimbra - Enable Runtime Monitoring

January, 2012CISUC, University of Coimbra - To where send events Send only a fraction of events

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Edit Sink Configuration

January, 2012CISUC, University of Coimbra - Enable Runtime Monitoring

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Select a test setup file…

January, 2012CISUC, University of Coimbra - … the one we have just modified

January, 2012CISUC, University of Coimbra - Performance Monitor is now listening to connections of Drivers and Sinks

January, 2012CISUC, University of Coimbra - Add counters…

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra - Drivers and Sinks are now connected with Performance Monitor

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

Lesson 5: Setting Test Options 1. Response Time Measurement Mode 2. Communication Mode 3. Log Flushing January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

January, 2012CISUC, University of Coimbra -

Response Time Measurement January, 2012CISUC, University of Coimbra - DRIVER CEP ENGINE CEP ENGINE SINK ADAPTER send convert input convert output process input forward generate result forward send 1 1

Response Time Measurement January, 2012CISUC, University of Coimbra - DRIVER CEP ENGINE CEP ENGINE SINK ADAPTER send convert input convert output process input forward generate result forward send Δt = t4 – t1 End-to-end, in milliseconds: From Driver up to Sink

Response Time Measurement January, 2012CISUC, University of Coimbra - DRIVER CEP ENGINE CEP ENGINE SINK ADAPTER send convert input convert output process input forward generate result forward send Δt = t3 – t2 Inside Adapter, in nanoseconds: From immediately before the input event is forwarded to CEP engine to immediately after the result is received

When to use End-to-End RT Measurement Mode When communication mode is set to DIRECT (mandatory); When it is acceptable to have a less accurate response time measurement in order to reduce the overhead in the server machine; When you want to consider communication and conversion time as part of response time; When Drivers and Sinks run in the same machine (no problems with clock synchronization). January, 2012CISUC, University of Coimbra -

When to use “Inside Adapter” RT Measurement Mode When accurate response time measurement is required; When you want exclude communication and conversion times from response time; When Drivers and Sinks run in different machines (requires synchronization between machines: inherently inaccurate ≈10ms); When the server machine can afford with the additional overhead of response time measurement. January, 2012CISUC, University of Coimbra -