Performance Monitor for Complex, Distributed ORB Instances Chen Chen, Kaijian Liu, Hui Long, Gabe Plank Aug. 29, 2010 Problem: Assuming massive data flow and complex ORB systems, how can problems be identified and performance analyzed? Orrington United

2 Scenarios: Consider: Millions of Strong Motion measurements along a pipeline. One 3C accelerometer every 10m for 1000km at 40 SPS, equals 12Million samples per second! Consider: Millions of data points per second from mobile GPS devices. OR

Currently tools exist to monitor performance of one ORB at a time. For a complex organization, this would be useless: Who Cares ?

4 Tools also exist to monitor large networks but these are complicated and time consuming to implement. As such, a lightweight tool for basic performance analysis would be useful. Who Cares ?

5 Implementation Sketch User interface (Matlab/GUI) Implement ORB performance algorithm using Matlab Antelope toolkit Develop algorithms to detect potential problems to system. Manage Antelope RT Start and Stop.

6 User Interface A graphical tool to design synthetic ORB hierarchies and read/write configurations Option to load a pre-specified configuration file, for existing real systems (implies no data added). Two slider bars to control size/# of data packets. View adjustments for looking at subsets of ORBs. Per-ORB dynamic performance graphs. Whole-system statistics such as average performance.

7 Performance Testing Algorithm Stream data to the system according to user inputs. For each ORB, determine CPU usage, Packet Latency, Read/Write time, Disk Access. Multi-thread each performance query so that all data is returned in real time. Data returned from threads added to a common data structure or a list of structures accessible by graphing and analysis routines.

8 Manage Antelope 1. Format Antelope rtexec files from Matlab, to configure run time parameters such as ORB port. 2. Launch Antelope real time instances Using Matlab ‘system’ call. 3. Provide process ID storage mechanism for shutting down the processes when Matlab application exits. Antelope Matlab GUI Start/Stop ORB ports

9 Detect potential problems Instance 1 Instance 2 Instance 3Instance 4 1. Infinite data loops. 2. Exceeding the buffer capacity and potential data loss. 3. Hardware problems.

10 Discussion and Questions 1. How to determine & implement meaningful whole- system statistics? 2. Possible automated runs with notification problems - do Matlab users run Matlab-based applications as services? 3. Are ORB type buffers scalable to handle massive, distributed data loads?

11 Summary 1. Real time data sources are growing, and so is telemetry bandwidth. 2. Datacenters should consider the potential need for distributed processing nodes and the means to monitor them. 3. This project attempts a first pass specification for a Matlab-based, performance analysis and monitoring system for high volume, distributed real time systems.