Concept demo System dashboard

Overview Dashboard use case General implementation ideas Use of MULE integration platform Collection Aggregation/Factorization model Plotting Service Usage Examples

Concept demo of the “troubleshooting dashboard” Understand compound state of the system in past by cross analyzing independent (and possibly many) data sources. The goal of the analysis is to spot trends and pattern in the change of the system state to develop some sort of systematic response procedure. Challenge variety of resources available to supply needed data. No common format or place where that data can be fetched Otherwise, grep and awk through log files Factor, scale,compare and display data on the fly

Time series Time series element describes state of the system at time T. Series show the evolution of system state (or its aspects) Time series are generated by the data source We want dashboard to index as many data sources as possible without enforcing strict content schema Use “dimensions” as the language to describe system aspects Time series element contains list of dimensions which collectively define an aspect. To optimize time series search queries, the dashboard will need to know the data source “schema” ( list of supportable dimensions)

Concept components Brian’s Bockelman gratia graphing and reporting tool Excellent resources which renders and makes available accounting data in XML. dCache memory plots by D. Litvintsev RRD toolkit (for visual reporting) Common graphing package for time series data Time series factorization tool. Developed by me. Implements rules for finding commonalities between different time series elements SMTP, Mime message, graphical dashboard A way to deliver results to user

Where Mule fits in ? Defines common data format to describe various system activities We’ve used time series of numerical values Defines work flow of services interacting with each other in accordance with user’s initial request Implements aggregation of data Integrates results into common reporting tools

Data source(s) Our scenario Data source Web frontend Log files Gratia dCache i.providers Fuzzy troubleshooter request Client Request results Render WareHouse Warehouse Service MapReduce [MULE] Data factorization and aggregation Request splitter Time. S factorization Aggregator

General design considerations Break up work flow into components that must access and maintain smallest possible, transient context Standalone service, pass through transformations Understand means of accepting, transforming and dispatching requests Map components into UMO models Map acceptance, transformation and dispatch to Mule endpoints, transformers and routers.

UI: Accepting requests Model We need a model that describes how we want to accept and dispatch out user requests to out “troublshooting/dashboard system”. In Mule, model may perform complex business related tasks or be simply pass though component linking input and output transformation logic. While both the model and Mule transformers “transform” data, the principal trait of the model is ability to maintain and use object state

UI: Accepting requests Model: input endpoints Define independent endpoints within the model " transformers=« MimeToString XMLToObjectt« /> Web Std in Each endpoint accepts data in a different format and uses specific transformer to converge to common type used down the chain.

Dispatching requests Our use case is to accept the request and multiplex that request to a set of data source providers for lookup. Use correlation to assemble results later Translate user query into the context specific to the data source

Data source(s) model Data source model adapts particular type of the data source. Gratia, dCache billing data, log mining, cache of the previously retrieved data Example: Gratia access though Brian B. web interface Use Mule RestServiceWrapper to accept messages and proxy them to a REST WEB service. For simplicity we don’t do any parsing or filtering here.

Data source input and output endpoints Input Output Content of the Brian’s web page (which is XML) is transformed to XML representing common time series data. That data is transformed to a java specific serialized XML representation and the translated to a vector of internal time series objects.

Aggregation/Factorization model The Mule pass though model is used to implement aggregation and factorization functional pieces. Data is aggregated at the input of the model Data is factorized at the output of the model No internal processing is needed.

Aggregation Time data aggregator waits and collects pieces from all data sources selected at the dispatch stage. TimeDataAggregator is a very simple class the defines a rule of how vectors of time series data should be joined into a new data type of higher level which instructs further steps of factorization and rendering.

Factorization We want to produce collection of plots from the input data array such that each plot has the same plotting instructions yet built from a subset of supplied data. Identify similarities across data array and generate set of independent reporting instances.

Mule mapping for factorization GroupBySplitterRouter implements router splitter interface. It uses message context to dissect factorization instructions and splits the message into pieces relevant only for each independent report (graph) Uses correlation to enable optional assembly of independent reports into a top level summary

PlottingService A stub for time series renderer May be replaced with any other render. JasperReport for ex. Model accepts time series messages with opaque instruction on how that time series must be rendered. In our case – plain RRD command. Model outputs UserPlotResponse that contains initial request along with the URL of the report file

UserPlotResponse aggregation Each UserPlotResponse is aggregated by the component responsible for notifying user with results of his request Uses correlation as set by factorization router-splitter

Example Your request to Mule stdin endpoint (for simplicity) Random id factor(Vos,$VO) testDimension=Running Jobs; RRD( AREA:testDimension#FF0000 ) Factor instructions define grouping by value of the VO field of the dataset (note : uses all data source data - no initial filtering of the data source, for simplicity ) RRD command follows the syntax of the RRD tools cmd line

Example output on your mailbox ……

Example 2 Your request sent to the SAME Mule stdin endpoint Random id factor(runningJobs,Running Jobs) testDimension=VO(dzero); testDimension1=VO(cms); RRD( LINE2:testDimension#FF0000 :D0 LINE2:testDimension1#0000FF:CMS) Factor defines class of time series that have “Running Jobs” dimension (which are all in the example) RRD expression plots CMS Vs D0 (true story)

Output in your mail box

capasityDim=Capacity; utilizationDim=Utilization; RRD( CDEF:diff=capasityDim,utilizationDim,- AREA:capasityDim#FF0000:capasity(Mb) AREA:utilizationDim#0000FF:utilization(Mb) AREA:diff#00FF00:difference(Mb))