Cavisson Monitoring System (CavMon)

Goals Features Architecture of Cavisson Monitoring System NetStorm Modules CavMonAgent Monitors/Tools/API Types monitors and their purpose Data Monitors Check Monitors How to use monitors in a test Monitor Profile Scenario Designing Running Test Special Features Event logging File FTP Server Admin and Remote Server Admin Monitor Testing and Troubleshooting Tutorial

Features

Features Monitoring of different server stats while test is running Get Server signatures before test starts Special checks while test is running Check free memory, disk and other server health parameters Take process dump and store in file Trigger specific tasks such as initiate data refresh of application at start of phase of a scenario Special tasks after test is over Ftp logs to NetStorm machine for offline analysis Very powerful monitoring system Highly flexible Support all platforms Linux/Unix Windows Integrated with execution GUI for easy analysis and correlation with different metrics

Architecture

Components of CavMonSystem NetStorm Monitoring Modules Server Configuration Monitor profiles/Monitor Configuration GDF files Execution GUI/Analysis GUI CavMonAgent Agent for running monitors on the server Sending data from monitors to NS Stop all monitors after test is over Monitors bundled with NetStorm Standard monitors like Java GC monitor Custom monitors like Log monitors Check monitors like server health check monitors Monitors created by customer

Architecture TCP/IP Connection CavMonAgent NetStorm Server GDF

CavMonAgent Architecture: NSParentProcess <--------> CavMonAgent <--------> Monitor(s) NS parent process make TCP/IP connection to the CavMonAgent running on the server. Send the message with details about monitor parameters CavMonAgent starts monitors with the parameters and open a pipe for reading data Monitor sends data over the pipe CavMonAgent forwards the data to NS over TCP/IP connection At the end of test, monitors stops. Also CavMonAgent stops all monitors after test is over

Setup Required

NetStorm machine to run the test. Access to the one or more servers which need to be monitored. CavMon Package need to be installed and running on the server. Servers.dat should have entry for the targeted servers.

What are Monitors

Used for following purposes: Special programs. Used for following purposes: Get Server data Get Logs Check server status Do some special tasks.

Server Data Data are of two types Stream data – Used for showing in graphs. Server Signature – Used for capturing server configuration/settings and comparison.

What all monitor can do

What all monitor can do? Generate data Generate event FTP files from server to NetStorm machine

Why do we need Monitors

Why do we need Monitors? To Monitoring the Server health. To get the important data from the Server.

What is required to Use Monitors Add the Targeted Server in Servers.dat. Check the CavMonAgent is running on the Targeted Server. Add Required Monitors in the mprof(s). Add the mprof(s) in the scenario. Run the scenario. Get in execution GUI and check the graphs for running monitors with given Vector Names.

Types of Monitors

There are two types of Monitors: Data Stream Monitors- Monitors which generate data for showing in graphs as per GDF Check Monitors- Monitors whose job is to do miscellanies tasks such as checking health of the system, ftp of logs files, running some command and capture output in file

Types of Data Stream Monitors Custom Monitor Special Monitor Standard Monitor Dynamic Monitor

Types of Check Monitor Check Monitor Server Signature

Servers.dat

Servers.dat Servers.dat file contains the details about the machine which has to be monitored.

To add a server definition from GUI, Login to the NS as Admin and click on the Admin button.

Select Server Administration link

Click on the Add button to add the details:

Provide the details in the fields, and click on the OK button.

Server will get added.

Adding server definition from CLI: [root@netstorm scenarios]# nsu_add_server Enter Server Name/IP: QANS1 Enter ssh (Y/N): Y Enter User Name: root Enter Password: xxxxxx Enter Java Home: /apps/java/jdk1.6.0_06/bin/java Enter cmon packege Installation Dir: /opt/cavisson/monitors Enter Server Agentless (Y/N): N Enter Machine type: Linux Server ‘QANS1' added successfully. Path:- /home/netstorm/work/server Details:- QANS1|Y|root|password|/apps/java/j2sdk1.4.0/bin/java|/opt/cavisson/monitors/|N| Linux|NA|NA|NA

How to Start/Stop CavMonAgent For Linux & Windows For Linux Servers:- To Start CavMonAgent. Run /etc/init.d/cmon stop|start|restart [root@netstorm work]# /etc/init.d/cmon restart Restarting cmon: Stopping cmon with pid(2312). cmon is stopped. Removing running_tests directory Starting cmon: cmon is started with pid(4337). UID PID PPID C STIME TTY TIME CMD root 4337 4306 12 02:38 pts/0 00:00:00 /apps/java/jdk1.6.0_06/bin/java [root@netstorm work]#

How to check CavMonAgent is running 1- /opt/cavisson/monitors/bin/cmon show cmon is running with pid(9402). UID PID PPID C STIME TTY TIME CMD root 9402 1 1 11:19 ? 00:02:24 /apps/java/jdk1.6.0_06/bin/java   2- [netstorm@netstorm3 monitors]$ ps -lef | grep CavMonAgent 0 S root 2451 1 0 78 0 - 55536 202055 10:02 ? 00:00:00 /apps/java/jdk1.6.0_06/bin/java - DPKG=cmon -DCAV_MON_HOME=/opt/cavisson/monitors - DCLASSPATH=/opt/cavisson/monitors/lib/weblogic.jar:/opt/cavisson/monitors/lib/java-getopt- 1.0.9.jar:/opt/cavisson/monitors/samples:/opt/cavisson/monitors/bin:/opt/cavisson/monitors/common/monitor_co mmon.jar:/opt/cavisson/monitors/lib/gcviewer-1.29.jar - DPATH=/opt/cavisson/monitors/bin:/opt/cavisson/monitors/bin:/apps/java/jdk1.6.0_06/bin:/home/netstorm/work /bin:/bin:/apps/java/jdk1.6.0_06/bin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin:/u sr/X11R6/bin CavMonAgent 3- CavMonAgent listen on port 7891 [root@netstorm3 work]# netstat -natp | grep 7891 tcp 0 0 :::7891 :::* LISTEN 2451/java

For Windows Servers- 1- Starting CavMonAgent from Admin cmd For Windows Servers- 1- Starting CavMonAgent from Admin cmd. Open admin cmd | Navigate to c:\opt\cavisson\monitors\bin To start the CavMonAgent – net start CavMonAgent To stop the CavMonAgent – net stop CavMonAgent

2- Starting CavMonAgent from services Click on Start |run| type services.msc |enter

Monitor Profile

Monitor Profile Monitor Profile is collection of different monitors. User can create different types of monitors in a Monitor Profile (e.g. Standard Monitors, Custom Monitors, Check Monitors etc.)

How to create Monitor Profile Click on the Monitor button in NS toolbar:

Click on Add button in Monitor Profile Management

Give Monitor Profile name and Description and then click on the Save button

How to use mprof in a Scenario

How to use mprof in a Scenario From CLI:- Open the appropriate scenario. Add the mprof in scenario. Example- MONITOR_PROFILE QANS1_Std_Uptime_Monitor From GUI:- In GUI Login with :NSServerIP(followed by :port) / netstorm

Netstorm Launch Screen will appear after logging to the Netstorm Netstorm Launch Screen will appear after logging to the Netstorm. Click on Scenarios Button.

Select a Scenario from Scenario Management GUI.

Select Global Settings and Select Monitor Tab.

Select a profile from the list and click on the Add then save it.

Custom Monitors

Custom Monitors Framework to run any monitor It needs following: GDF for the monitor Program for the monitor In Custom monitors user has to provide server name, gdf name, vector name, option and program name.

Custom Monitors Keyword:- CUSTOM_MONITOR <Server name> <GDF File Name> <vector name> <run-option> program_name Example:- CUSTOM_MONITOR QANS1 cm_uptime.gdf QANS1_UPTime 2 cm_uptime

Standard Monitors

Standard Monitor Standard Monitor is Simplified version of Custom Monitor. Used for all monitors which are bundled with NetStorm release.

Types of Standard Monitors

Standard Monitors for Unix JavaGC Monitors JavaGCExtended JavaJStatGC ProcessData TCPStatesCount TCPStats SystemStats UPTime WeblogicJDBCStats WeblogicQStats WeblogicSVCTime ActimizeSVCTime

Standard Monitors for Windows WinSystemStats ProcessorStats Memory Stats LogicalDiskStats ProcessStats .NETCLRMemoryStats .NETDataProviderForSqlServerStats .NET CLRSecurityStats .NETCLRRemotingStats .NETCLRLockAndThreadsStats .NETCLRLoadingStats .NETCLRJitStats .NETCLRInterropStats .NETCLRExceptionsStats W3SVCWebServiceStats TCPIPTCP4Stats ASPActiveServerPagesStats

Standard Monitor Keyword: STANDARD_MONITOR <ServerName> <VectorName> <MonitorName> Example:- STANDARD_MONITOR QANS1 QANS1_uptime UpTime

Dynamic Vector Monitor

Dynamic Vector Monitor Dynamic Vector Monitors are those monitors whose vectors are not configured in the keyword in scenario file or monitor profile. Instead vector names are queried from the monitor using Dynamic Vector interface.

Dynamic Monitor Keyword:- Example:- DYNAMIC_VECTOR_MONITOR <ServerName> <Vector_Name><Gdf> <run options><Program Name For Getting Data> EOC <Program Name For Getting Vector List> Example:- DYNAMIC_VECTOR_MONITOR QANO1 NSDiskfree cm_df.gdf 2 cm_df -i 10000 EOC cm_df -v

Check Monitors

Check Monitors Check Monitors used for check the status of servers which Netstorm is hitting to make sure servers are healthy during test run. Keyword:- <ServerName><MonitorName><ExecuteFirstTime><RepeatThereafter><Continueuntil>< Check Monitor Program Name> Example:- CHECK_MONITOR QANS1 QANS1_Aftertest 90 NA 2 NA NA NA /home/netstorm/test/CheckMonitor/monitors/ftp_pass_cases/chkmon_ftp_posFileSize_html .sh

Server Signature

Server Signature Server Signature is required to compare report. To get the information about the machine. We can create Server Signature with two types: Command Type/File Type

Server Signature Keyword:- <ServerName><SignatureName><Signature Type><Command/File> Example:- QANS1 Server_Sig_QANS1 Command netstat

How to test monitor Match the number of fields printed in progress report, number of graphs in GUI with total number of fields provided in GDF. Testing from CLI Run a Monitor. Run the command from the below specified path, on the server for which monitors have been used- /opt/cavisson/monitors/bin

Match each and every field of all the monitors deployed with Progress Report. For windows we can test the monitor in two ways either by following above steps or using the windows in-build tool named as “Performance Monitor”.

Testing from GUI Refer “How to use mprof in Scenarios” slide. Run the scenario. Execution GUI will open Locate monitors by name on the left panel and click on that. All the graphs related to the monitor will be shown on the right panel. Now compare the graph values with the values got by running the command from bin. In case of windows monitor we can verify the data with performance Monitor too.

Graphs related to Monitors (Uptime, Processor stats)

File FTP - From Netstorm to Server NS has an API, by which we can transfer a file from NS machine to Server. Command:- nsu_server_admin -s QA_Win1 -F tx_am_ipv6.conf -D C:\\opt\\cavisson\\monitors.

File Ftp - Server to Netstorm From Check monitors/Server signatures to ftp files/logs from server to NS machine 

Remote Server Admin- Man page of Remote server admin Usage: nsu_server_admin <option> <arguments> Where options are: -a or -s: All Servers or Specific Server Name/I.P. -c Command. -D: ftp file destination. -d: Enable debug -F: ftp file name. -j: Show Java Core dumps on specified server.(Not For Windows) -p: Package tar file --monitors: Show monitors running on specified servers (Not For Windows). --restart: Restart cmon on specified server. --show: Show cmon (Not For Windows). --start: start cmon (Not For Windows). --stop: stop cmon. --version: Show version of cmon on specified server.

Example of Remote Admin

Select Server Administration link

Click on the Cmon Version

Monitors Debugging All types of events get logged here. Monitors.log and event.log Paths- /home/netstorm/work/logs/TRxxxx/ Check event.log All types of events get logged here. Check monitor.log Keep information about start of test, start/end of monitors etc.

Check CavMonAgentError.log Check CavMonAgentDebug.log CavMonAgent Logs:- Check CavMonAgentError.log Error logs are kept here. Check CavMonAgentDebug.log Debug logs are kept here. Check CavMonAgentTrace.log Run the monitor from command line.

Keep information about start of test, start/end of monitors CavMonAgentTrace.log Keep information about start of test, start/end of monitors Sample Log file:– 10/5 7:15:57|35247|NA|-1|New test run starts 10/5 7:15:57|35247|DevNOTomcatWork|1|Monitor Starts. Monitor parameters = Frequency = 10000, Option = 2, Access = 1, Timeout = -1, Check Start delay = 0, Check count = -1, Counter = 1, Program path = java, Type = cm_init_monitor, Remort Ip = NA, Remort User name = NA, Remot pwd = NA, Argscm_java_gc_ex -o 2 -f /apps/apache-tomcat- 6.0.16/logs/gc_tomcat_work.log -D, Cmd with args = java cm_java_gc_ex -o 2 -f /apps/apache-tomcat-6.0.16/logs/gc_tomcat_work.log -D 10/5 7:15:58|35247|DevNOTomcatWork|1|Monitor ends 10/5 7:15:58|35247|NA|-1|Test run ends

Run the Monitor from Command Line. For Linux- Navigate to bin directory and run the below specified command. cm_uptime [netstorm@netocean bin]$ ./cm_uptime 0.14 0.04 0.02 0.001 2

Navigate to bin folder and run the below specified command— For Windows- Navigate to bin folder and run the below specified command— CavMonCscript.exe /nologo wmi_processor.vbs _Total

Questions What are the different run options available for Custom Monitors? What is the difference between Custom and Standard Monitors and which fields are not being used in the syntax of Standard Monitors? Where we can add server definition? How can we add server definition ? Where event.log and monitor.log get created? Why we have mprof and what is the path? Can we do the same task without mprof? if yes then how, if no then why? Can we have same Vector Name for two monitors targeting two different servers? What package need to be installed on the server whose data we are going to monitor. Currently which operating systems are supported by Cavisson Monitoring System? What is the path of CavMonAgent logs. Continued…..

Questions What are the options available for check monitors. What is the difference between Server Signatures & Check Monitors. Where result/output of Server Signatures & Check Monitors get saved? How can we start and stop CavMonAgent? Default port number, used by CavMonAgent? Which feature of netstorm can be used to FTP a file from NS machine to Server? Where the program files for monitors are located? Which feature of netstorm provide option to run a command on the server from NS (not by monitors)?

Answers What are the different available run options, and how they make difference in implementation? Run Once (2) and Run Every time (1), in case of one monitor starts and end in every specific interval (default is 10 Secs) and in case of 2 monitor starts and continues till the test run is over. What is the difference between Custom and Standard Monitors and which fields are not being used in the syntax of Standard Monitor? Custom monitors are framework of all monitors, have complex structure of syntax. while standard monitors are simplified version of custom monitor, easy to use. And user need not to use run option and GDF. Can we have same Vector Name for two monitors targeting two different servers? Because in Progress report we have only Vector name to identify which monitor is used for which machine. How we can update servers.dat and the path of the file? Server.dat can be updated either by GUI or by nsu_add_server. File is located on /home/netstorm/work/server/servers.dat

THANK YOU