1. Marcelo R.N. Mendes

2. What is FINCoS? A set of tools for data generation, load submission, and performance measurement of CEP systems; Main Characteristics: Flexible; Independent of particular workloads; Neutral (not bound to any specific CEP product); Correctness check; Scalable. 2CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

3. FINCoS: Architecture 3CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012 Scalability: It is easy to increase the load: simply add more Drivers! Validation and Performance Measuring: Plug-and-Play Validators Independence of CEP Products and workloads Flexibility: Several Test Options

4. FINCoS: Components Controller Main Interface with the user. Allows to: Create reusable test setups; Control Drivers and Sinks (e.g., load, start, stop); Alter workload parameters on-the- fly (e.g., event submission rates, switch phases) 4CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

5. FINCoS: Components Driver Simulates external sources of events; Submits load to the SUT; Includes data generation mechanism; Sink Receives query results from CEP engine; Stores output events in log files or send them via network for subsequent validation; 5CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

6. FINCoS: Components Performance Monitor Collect Metrics of Performance Runs Two Modes: Online (network); Offline (log files). 6CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

7. FINCoS: Components January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt7 Adapter On Input: converts from the framework neutral representation (CSV) to CEP system’s native representation On Ouput: converts output events from CEP system’s native format to neutral representation (CSV)

8. Lesson 1: Creating a Test Setup 1. Configure a Driver 1. Decomposable Workload 2. Synthetic vs. Thirdy-Party Dataset 3. Data Generation Options 4. Event Arrival Process 2. Configure a Sink January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt8

9. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt9

10. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt10

11. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt11 unique identifier number of threads used for load generation save generated data to disk? real time performance monitoring? to where Driver send the events machine that hosts the Driver

12. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt12 Workload can be split in independent phases

13. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt13

14. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt14 Chooses the type of workload for this phase Sets events arrival process Event Mix Events are generated in a ordered way When data generation happens To ensure repeatability…

15. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt15 Configure event types…

16. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt16

17. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt17

18. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt18 Configure Event Type Schema

19. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt19

20. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt20

21. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt21 Column Name

22. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt22 Column Type

23. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt23 Data Generation Options

24. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt24 Add Values

25. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt25

26. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt26

27. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt27

28. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt28

29. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt29

30. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt30

31. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt31

32. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt32

33. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt33

34. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt34

35. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt35

36. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt36

37. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt37

38. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt38 unique identifier List of event types this Sink subscribes to from where Sink receive events machine that hosts the Sink Port used by Sink to receive events save generated data to disk? real time performance monitoring?

39. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt39

40. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt40

41. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt41

42. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt42 Done!

43. Lesson 2: Connecting with a CEP engine 1. Configure Connection Properties 2. Load FINCoS Adapter application January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt43

44. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt44 List of input streamsList of output streams No Information Available Yet!

45. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt45 Retrieve list of streams…

46. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt46 … from the setup file created earlier

47. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt47

48. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt48 Load vendor-specific connection properties …

49. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt49 Engine=Esper Configuration_file_path=C:\\FINCoS\\queries\\esper\\Q1\\Stream_Set.xml Queries_path=C:\\FINCoS\\queries\\esper\\Q1\\Query_Set.xml Event_format=Map Engine=Esper Configuration_file_path=C:\\FINCoS\\queries\\esper\\Q1\\Stream_Set.xml Queries_path=C:\\FINCoS\\queries\\esper\\Q1\\Query_Set.xml Event_format=Map

50. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt50 Connect to CEP engine…

51. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt51 The list of streams in the CEP engine is retrieved and matched against the configured ones in the test setup

52. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt52 Loads the Adapter: 1.Open Server Socket for Drivers (Input) 2.Subscribes to streams at the CEP engine (Output)

53. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt53 Done!

54. Lesson 3: Running Tests 1. Load Drivers and Sinks 2. Start Load Submission 3. Control Load Submission NOTE: At this point the following steps must have been completed: 1. Creation of a test setup; 2. Connection with a CEP engine; 3. Initialization of the FINCoS Daemon Service at the machines where Drivers and Sinks are intended to run; January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt54

55. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt55 Load components

56. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt56

57. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt57 Start Load Submission

58. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt58 Sent/Received counts

59. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt59

60. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt60

61. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt61

62. Lesson 4: Collecting Performance Metrics 1. Offline Performance Measurement 1. After test completion, use FINCoS Performance Monitor to process log files 2. Online Performance Measurement 1. Enable realtime monitoring at Drivers and Sinks; 2. Initialize FINCoS Performance Monitor in the Realtime Mode before test starts. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt62

63. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt63 Offline Performance Measurement

64. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt64 Add log files

65. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt65 Log file created after test completion

66. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt66 Configure M.I. here: Optionally, save collected metrics to a file Process log files

67. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt67 View Results

68. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt68

69. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt69 Select the stream

70. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt70 Select the metric

71. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt71 Select exhibition scale

72. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt72 Add Metrics to the Chart

73. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt73 Change Visualization Options

74. Lesson 4: Collecting Performance Metrics (...cont) 1. Offline Performance Measurement 1. After test completion, use FINCoS Performance Monitor to process log files 2. Online Performance Measurement 1. Enable realtime monitoring at Drivers and Sinks; 2. Initialize FINCoS Performance Monitor in the Realtime Mode before test starts. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt74

75. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt75 Edit Driver Configuration

76. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt76 Enable Runtime Monitoring

77. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt77 To where send events Send only a fraction of events

78. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt78

79. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt79 Edit Sink Configuration

80. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt80 Enable Runtime Monitoring

81. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt81

82. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt82 Select a test setup file…

83. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt83 … the one we have just modified

84. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt84 Performance Monitor is now listening to connections of Drivers and Sinks

85. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt85 Add counters…

86. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt86

87. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt87

88. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt88 Drivers and Sinks are now connected with Performance Monitor

89. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt89

90. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt90

91. Lesson 5: Setting Test Options 1. Response Time Measurement Mode 2. Communication Mode 3. Log Flushing January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt91

92. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt92

93. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt93

94. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt94

95. Response Time Measurement January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt95 DRIVER CEP ENGINE CEP ENGINE SINK ADAPTER 2 2 3 3 4 4 send convert input convert output process input forward generate result forward send 1 1

96. Response Time Measurement January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt96 DRIVER CEP ENGINE CEP ENGINE SINK ADAPTER send convert input convert output process input forward generate result forward send Δt = t4 – t1 End-to-end, in milliseconds: From Driver up to Sink 2 2 3 3 4 4 1 1

97. Response Time Measurement January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt97 DRIVER CEP ENGINE CEP ENGINE SINK ADAPTER send convert input convert output process input forward generate result forward send Δt = t3 – t2 Inside Adapter, in nanoseconds: From immediately before the input event is forwarded to CEP engine to immediately after the result is received. 2 2 3 3 4 4 1 1

98. When to use End-to-End RT Measurement Mode When communication mode is set to DIRECT (mandatory); When it is acceptable to have a less accurate response time measurement in order to reduce the overhead in the server machine; When you want to consider communication and conversion time as part of response time; When Drivers and Sinks run in the same machine (no problems with clock synchronization). January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt98

99. When to use “Inside Adapter” RT Measurement Mode When accurate response time measurement is required; When you want exclude communication and conversion times from response time; When Drivers and Sinks run in different machines (requires synchronization between machines: inherently inaccurate ≈10ms); When the server machine can afford with the additional overhead of response time measurement. January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt99