1. Cavisson Monitoring System (CavMon)

2. Goals Features Architecture of Cavisson Monitoring System
NetStorm Modules
CavMonAgent
Monitors/Tools/API
Types monitors and their purpose
Data Monitors
Check Monitors
How to use monitors in a test
Monitor Profile
Scenario Designing
Running Test
Special Features
Event logging
File FTP
Server Admin and Remote Server Admin
Monitor Testing and Troubleshooting
Tutorial

3. Features

4. Features
Monitoring of different server stats while test is running
Get Server signatures before test starts
Special checks while test is running
Check free memory, disk and other server health parameters
Take process dump and store in file
Trigger specific tasks such as initiate data refresh of application at start of phase of a scenario
Special tasks after test is over
Ftp logs to NetStorm machine for offline analysis
Very powerful monitoring system
Highly flexible
Support all platforms
Linux/Unix
Windows
Integrated with execution GUI for easy analysis and correlation with different metrics

5. Architecture

6. Components of CavMonSystem
NetStorm Monitoring Modules
Server Configuration
Monitor profiles/Monitor Configuration
GDF files
Execution GUI/Analysis GUI
CavMonAgent
Agent for running monitors on the server
Sending data from monitors to NS
Stop all monitors after test is over
Monitors bundled with NetStorm
Standard monitors like Java GC monitor
Custom monitors like Log monitors
Check monitors like server health check monitors
Monitors created by customer

7. Architecture
TCP/IP Connection
CavMonAgent
NetStorm
Server
GDF

8. CavMonAgent Architecture:
NSParentProcess < > CavMonAgent < > Monitor(s)
NS parent process make TCP/IP connection to the CavMonAgent running on the server.
Send the message with details about monitor parameters
CavMonAgent starts monitors with the parameters and open a pipe for reading data
Monitor sends data over the pipe
CavMonAgent forwards the data to NS over TCP/IP connection
At the end of test, monitors stops.
Also CavMonAgent stops all monitors after test is over

9. Setup Required

10. NetStorm machine to run the test.
Access to the one or more servers which need to be monitored.
CavMon Package need to be installed and running on the server.
Servers.dat should have entry for the targeted servers.

11. What are Monitors

12. Used for following purposes:
Special programs.
Used for following purposes:
Get Server data
Get Logs
Check server status
Do some special tasks.

13. Server Data Data are of two types
Stream data – Used for showing in graphs.
Server Signature – Used for capturing server configuration/settings and comparison.

14. What all monitor can do

15. What all monitor can do? Generate data Generate event
FTP files from server to NetStorm machine

16. Why do we need Monitors

17. Why do we need Monitors? To Monitoring the Server health.
To get the important data from the Server.

18. What is required to Use Monitors
Add the Targeted Server in Servers.dat.
Check the CavMonAgent is running on the Targeted Server.
Add Required Monitors in the mprof(s).
Add the mprof(s) in the scenario.
Run the scenario.
Get in execution GUI and check the graphs for running monitors with given Vector Names.

19. Types of Monitors

20. There are two types of Monitors:
Data Stream Monitors- Monitors which generate data for showing in graphs as per GDF
Check Monitors- Monitors whose job is to do miscellanies tasks such as checking health of the system, ftp of logs files, running some command and capture output in file

21. Types of Data Stream Monitors
Custom Monitor
Special Monitor
Standard Monitor
Dynamic Monitor

22. Types of Check Monitor
Check Monitor
Server Signature

23. Servers.dat

24. Servers.dat
Servers.dat file contains the details about the machine which has to be monitored.

25. To add a server definition from GUI, Login to the NS as Admin and click on the Admin button.

26. Select Server Administration link

27. Click on the Add button to add the details:

28. Provide the details in the fields, and click on the OK button.

29. Server will get added.

30. Adding server definition from CLI:
scenarios]# nsu_add_server
Enter Server Name/IP: QANS1
Enter ssh (Y/N): Y
Enter User Name: root
Enter Password: xxxxxx
Enter Java Home: /apps/java/jdk1.6.0_06/bin/java
Enter cmon packege Installation Dir: /opt/cavisson/monitors
Enter Server Agentless (Y/N): N
Enter Machine type: Linux
Server ‘QANS1' added successfully.
Path:- /home/netstorm/work/server
Details:-
QANS1|Y|root|password|/apps/java/j2sdk1.4.0/bin/java|/opt/cavisson/monitors/|N| Linux|NA|NA|NA

31. How to Start/Stop CavMonAgent For Linux & Windows
For Linux Servers:-
To Start CavMonAgent.
Run /etc/init.d/cmon stop|start|restart
work]# /etc/init.d/cmon restart
Restarting cmon:
Stopping cmon with pid(2312).
cmon is stopped.
Removing running_tests directory
Starting cmon:
cmon is started with pid(4337).
UID PID PPID C STIME TTY TIME CMD
root :38 pts/ :00:00 /apps/java/jdk1.6.0_06/bin/java
work]#

32. How to check CavMonAgent is running
1- /opt/cavisson/monitors/bin/cmon show
cmon is running with pid(9402).
UID PID PPID C STIME TTY TIME CMD
root :19 ? :02:24 /apps/java/jdk1.6.0_06/bin/java
2- monitors]$ ps -lef | grep CavMonAgent
0 S root :02 ? :00:00 /apps/java/jdk1.6.0_06/bin/java - DPKG=cmon -DCAV_MON_HOME=/opt/cavisson/monitors - DCLASSPATH=/opt/cavisson/monitors/lib/weblogic.jar:/opt/cavisson/monitors/lib/java-getopt jar:/opt/cavisson/monitors/samples:/opt/cavisson/monitors/bin:/opt/cavisson/monitors/common/monitor_co mmon.jar:/opt/cavisson/monitors/lib/gcviewer-1.29.jar - DPATH=/opt/cavisson/monitors/bin:/opt/cavisson/monitors/bin:/apps/java/jdk1.6.0_06/bin:/home/netstorm/work /bin:/bin:/apps/java/jdk1.6.0_06/bin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin:/u sr/X11R6/bin CavMonAgent
3- CavMonAgent listen on port 7891
work]# netstat -natp | grep 7891
tcp ::: :::* LISTEN /java

33. For Windows Servers- 1- Starting CavMonAgent from Admin cmd
For Windows Servers- 1- Starting CavMonAgent from Admin cmd. Open admin cmd | Navigate to c:\opt\cavisson\monitors\bin To start the CavMonAgent – net start CavMonAgent To stop the CavMonAgent – net stop CavMonAgent

34. 2- Starting CavMonAgent from services
Click on Start |run| type services.msc |enter

35. Monitor Profile

36. Monitor Profile Monitor Profile is collection of different monitors.
User can create different types of monitors in a Monitor Profile
(e.g. Standard Monitors, Custom Monitors, Check Monitors etc.)

37. How to create Monitor Profile
Click on the Monitor button in NS toolbar:

38. Click on Add button in Monitor Profile Management

39. Give Monitor Profile name and Description and then click on the Save button

40. How to use mprof in a Scenario

41. How to use mprof in a Scenario
From CLI:-
Open the appropriate scenario.
Add the mprof in scenario.
Example-
MONITOR_PROFILE QANS1_Std_Uptime_Monitor
From GUI:-
In GUI Login with :NSServerIP(followed by :port) / netstorm

42. Netstorm Launch Screen will appear after logging to the Netstorm
Netstorm Launch Screen will appear after logging to the Netstorm. Click on Scenarios Button.

43. Select a Scenario from Scenario Management GUI.

44. Select Global Settings and Select Monitor Tab.

45. Select a profile from the list and click on the Add then save it.

46. Custom Monitors

47. Custom Monitors
Framework to run any monitor
It needs following:
GDF for the monitor
Program for the monitor
In Custom monitors user has to provide server name, gdf name, vector name, option and program name.

48. Custom Monitors
Keyword:-
CUSTOM_MONITOR <Server name> <GDF File Name> <vector name> <run-option> program_name
Example:-
CUSTOM_MONITOR QANS1 cm_uptime.gdf QANS1_UPTime 2 cm_uptime

49. Standard Monitors

50. Standard Monitor
Standard Monitor is Simplified version of Custom Monitor.
Used for all monitors which are bundled with NetStorm release.

51. Types of Standard Monitors

52. Standard Monitors for Unix
JavaGC Monitors
JavaGCExtended
JavaJStatGC
ProcessData
TCPStatesCount
TCPStats
SystemStats
UPTime
WeblogicJDBCStats
WeblogicQStats
WeblogicSVCTime
ActimizeSVCTime

53. Standard Monitors for Windows
WinSystemStats
ProcessorStats
Memory Stats
LogicalDiskStats
ProcessStats
.NETCLRMemoryStats
.NETDataProviderForSqlServerStats
.NET CLRSecurityStats
.NETCLRRemotingStats
.NETCLRLockAndThreadsStats
.NETCLRLoadingStats
.NETCLRJitStats
.NETCLRInterropStats
.NETCLRExceptionsStats
W3SVCWebServiceStats
TCPIPTCP4Stats
ASPActiveServerPagesStats

54. Standard Monitor Keyword:
STANDARD_MONITOR <ServerName> <VectorName> <MonitorName>
Example:-
STANDARD_MONITOR QANS1 QANS1_uptime UpTime

55. Dynamic Vector Monitor

56. Dynamic Vector Monitor
Dynamic Vector Monitors are those monitors whose vectors are not configured in the keyword in scenario file or monitor profile. Instead vector names are queried from the monitor using Dynamic Vector interface.

57. Dynamic Monitor Keyword:- Example:-
DYNAMIC_VECTOR_MONITOR <ServerName> <Vector_Name><Gdf> <run options><Program Name For Getting Data> EOC <Program Name For Getting Vector List>
Example:-
DYNAMIC_VECTOR_MONITOR QANO1 NSDiskfree cm_df.gdf 2 cm_df -i EOC cm_df -v

58. Check Monitors

59. Check Monitors
Check Monitors used for check the status of servers which Netstorm is hitting to make sure servers are healthy during test run.
Keyword:- <ServerName><MonitorName><ExecuteFirstTime><RepeatThereafter><Continueuntil>< Check Monitor Program Name>
Example:-
CHECK_MONITOR QANS1 QANS1_Aftertest 90 NA 2 NA NA NA /home/netstorm/test/CheckMonitor/monitors/ftp_pass_cases/chkmon_ftp_posFileSize_html .sh

60. Server Signature

61. Server Signature
Server Signature is required to compare report.
To get the information about the machine.
We can create Server Signature with two types:
Command Type/File Type

62. Server Signature
Keyword:-
<ServerName><SignatureName><Signature Type><Command/File>
Example:-
QANS1 Server_Sig_QANS1 Command netstat

63. How to test monitor
Match the number of fields printed in progress report, number of graphs in GUI with total number of fields provided in GDF.
Testing from CLI
Run a Monitor.
Run the command from the below specified path, on the server for which monitors have been used-
/opt/cavisson/monitors/bin

64. Match each and every field of all the monitors deployed with Progress Report. For windows we can test the monitor in two ways either by following above steps or using the windows in-build tool named as “Performance Monitor”.

65. Testing from GUI Refer “How to use mprof in Scenarios” slide.
Run the scenario. Execution GUI will open
Locate monitors by name on the left panel and click on that.
All the graphs related to the monitor will be shown on the right panel.
Now compare the graph values with the values got by running the command from bin.
In case of windows monitor we can verify the data with performance Monitor too.

66. Graphs related to Monitors (Uptime, Processor stats)

67. File FTP - From Netstorm to Server
NS has an API, by which we can transfer a file from NS machine to Server.
Command:- nsu_server_admin -s QA_Win1 -F tx_am_ipv6.conf -D C:\\opt\\cavisson\\monitors.

68. File Ftp - Server to Netstorm
From Check monitors/Server signatures to ftp files/logs from server to NS machine 

69. Remote Server Admin- Man page of Remote server admin Usage:
nsu_server_admin <option> <arguments>
Where options are:
-a or -s: All Servers or Specific Server Name/I.P.
-c Command.
-D: ftp file destination.
-d: Enable debug
-F: ftp file name.
-j: Show Java Core dumps on specified server.(Not For Windows)
-p: Package tar file
--monitors: Show monitors running on specified servers (Not For Windows).
--restart: Restart cmon on specified server.
--show: Show cmon (Not For Windows).
--start: start cmon (Not For Windows).
--stop: stop cmon.
--version: Show version of cmon on specified server.

70. Example of Remote Admin

71. Select Server Administration link

72. Click on the Cmon Version

74. Monitors Debugging All types of events get logged here.
Monitors.log and event.log
Paths-
/home/netstorm/work/logs/TRxxxx/
Check event.log
All types of events get logged here.
Check monitor.log
Keep information about start of test, start/end of monitors etc.

75. Check CavMonAgentError.log Check CavMonAgentDebug.log
CavMonAgent Logs:-
Check CavMonAgentError.log
Error logs are kept here.
Check CavMonAgentDebug.log
Debug logs are kept here.
Check CavMonAgentTrace.log
Run the monitor from command line.

76. Keep information about start of test, start/end of monitors
CavMonAgentTrace.log
Keep information about start of test, start/end of monitors
Sample Log file:–
10/5 7:15:57|35247|NA|-1|New test run starts
10/5 7:15:57|35247|DevNOTomcatWork|1|Monitor Starts. Monitor parameters = Frequency = 10000, Option = 2, Access = 1, Timeout = -1, Check Start delay = 0, Check count = -1, Counter = 1, Program path = java, Type = cm_init_monitor, Remort Ip = NA, Remort User name = NA, Remot pwd = NA, Argscm_java_gc_ex -o 2 -f /apps/apache-tomcat /logs/gc_tomcat_work.log -D, Cmd with args = java cm_java_gc_ex -o 2 -f /apps/apache-tomcat /logs/gc_tomcat_work.log -D
10/5 7:15:58|35247|DevNOTomcatWork|1|Monitor ends
10/5 7:15:58|35247|NA|-1|Test run ends

77. Run the Monitor from Command Line.
For Linux-
Navigate to bin directory and run the below specified command.
cm_uptime
bin]$ ./cm_uptime

78. Navigate to bin folder and run the below specified command—
For Windows-
Navigate to bin folder and run the below specified command—
CavMonCscript.exe /nologo wmi_processor.vbs _Total

79. Questions
What are the different run options available for Custom Monitors?
What is the difference between Custom and Standard Monitors and which fields are not being used in the syntax of Standard Monitors?
Where we can add server definition?
How can we add server definition ?
Where event.log and monitor.log get created?
Why we have mprof and what is the path? Can we do the same task without mprof? if yes then how, if no then why?
Can we have same Vector Name for two monitors targeting two different servers?
What package need to be installed on the server whose data we are going to monitor.
Currently which operating systems are supported by Cavisson Monitoring System?
What is the path of CavMonAgent logs.
Continued…..

80. Questions What are the options available for check monitors.
What is the difference between Server Signatures & Check Monitors.
Where result/output of Server Signatures & Check Monitors get saved?
How can we start and stop CavMonAgent?
Default port number, used by CavMonAgent?
Which feature of netstorm can be used to FTP a file from NS machine to Server?
Where the program files for monitors are located?
Which feature of netstorm provide option to run a command on the server from NS (not by monitors)?

81. Answers
What are the different available run options, and how they make difference in implementation?
Run Once (2) and Run Every time (1), in case of one monitor starts and end in every specific interval (default is 10 Secs) and in case of 2 monitor starts and continues till the test run is over.
What is the difference between Custom and Standard Monitors and which fields are not being used in the syntax of Standard Monitor?
Custom monitors are framework of all monitors, have complex structure of syntax. while standard monitors are simplified version of custom monitor, easy to use. And user need not to use run option and GDF.
Can we have same Vector Name for two monitors targeting two different servers?
Because in Progress report we have only Vector name to identify which monitor is used for which machine.
How we can update servers.dat and the path of the file?
Server.dat can be updated either by GUI or by nsu_add_server.
File is located on /home/netstorm/work/server/servers.dat

82. THANK YOU