1 © 1999 BMC SOFTWARE, INC. 3/17/99 Integrating PATROL with SNMP (Simple Network Management Protocol) Eric Anderson BMC Software Developer Connection

2 © 1999 BMC SOFTWARE, INC. 3/17/99 SNMP - Overview  Network protocols Transport layer (UDP/IP) Message protocol (ASN.1)  Versions SNMPv1 SNMPv2 SNMPv3  OSI network management implementation CMIP - CMOT

3 © 1999 BMC SOFTWARE, INC. 3/17/99 SNMP : Underlying Transport UDP/IP implications (unsafe) unreliable no confirmation or guarantee order uncertain may duplicate unsecure No authentication on transport layer No verification of origin (message spoofing) No integrity (message can be tampered with)

4 © 1999 BMC SOFTWARE, INC. 3/17/99 SNMP message protocol ASN.1 (iso “abstract notation one”) Does information exchange with well defined syntax comparable in functionality with XML BER (iso 8825) Basic Encoding Rules TLV (Type, Length, Value) encoding

5 © 1999 BMC SOFTWARE, INC. 3/17/99 Management Operations  Get, Get Next, Set for SNMPv1  Get Bulk (for SNMPv2)  Unsolicited Traps from Agent (v1)  Informs (v3 and v3)

6 © 1999 BMC SOFTWARE, INC. 3/17/99 Get Request handling  Agent gets request from mgr listens on specific port : typically 161 receives PDU : request ID + OID  Lookup of OID in memory  Agent sends get response request ID OID error status + index value  Similar for get-next operation returns next OID in tree

7 © 1999 BMC SOFTWARE, INC. 3/17/99 Trap sending SNMP Agent determines when  7 predefined Generic trap types (0-6) 6=enterpriseSpecfic  Send via to port 162 on trap receiving machine  Trap destinations has to be configured Patrol : /snmp/piV1m_list

8 © 1999 BMC SOFTWARE, INC. 3/17/99 Listening for traps Only one process can listen on a port That means only one trap listener per system ! Solution :  PATROL : trap demultiplexer (Dietmar Hildebrand)  No PATROL : (DIY= write your own)

9 © 1999 BMC SOFTWARE, INC. 3/17/99 MIB (Management Information Base)  The MIB file textual description of MIB layout Written in SMI : Structure of Managment Information  SMI (the MIB definition language) Support for multiple datatypes Support for data definitions : indexed tables, structures, values,...

10 © 1999 BMC SOFTWARE, INC. 3/17/99 NAMING in MIB Naming hierarchy unique identifier, down to the instance of any object is the OID of the BMC Software subtree iso 1 3 org 6dod 1 internet 2 mgmt 4 private 1enterprises enterpriseID peer BMC Company X Whatever you like

11 © 1999 BMC SOFTWARE, INC. 3/17/99 The SNMP Manager What is an SNMP manager ? Components : present MIB info (display MIB files) send requests listen for traps

12 © 1999 BMC SOFTWARE, INC. 3/17/99 The SNMP Agent What is an SNMP agent ?  Components : MIB (the MIB in memory) request handling trap sending

13 © 1999 BMC SOFTWARE, INC. 3/17/99 Multiple SNMP Agents on 1 system Conflict : only 1 process can listen on a port Solutions : SMUX (SNMP Multiplexer) AgentX - Agent Extensibility (rfc 2741) Emanate (proprietary SNMP Research) 161 Master Agent Sub agent

14 © 1999 BMC SOFTWARE, INC. 3/17/99 SNMPv3  New standardization effort for v2 with enhanced Security User Based security model (rfc 2574)

15 © 1999 BMC SOFTWARE, INC. 3/17/99 SNMP References SNMP v1 RFCs  1155SMI  1157protocol  1212Concise MIB  1213MIB2  1215traps SNMP v2 RFCs  historical See also : The Simple Web SNMP v2C RFCs  SNMP v3 RFCs   

16 © 1999 BMC SOFTWARE, INC. 3/17/99 PATROL & SNMP

17 © 1999 BMC SOFTWARE, INC. 3/17/99 PATROL SNMP – Windows NT PATROL Agent With SNMP Sub Agent Management Station SNMP Agent (Master) Windows Service PATROL SNMP Master Agent Port 161 Port 8161 Applications Uses Windows SNMP API Applications Uses Windows SNMP API Applications Uses Windows SNMP API Applications Uses Windows SNMP API SMUX Port 191 Application SNMP Runtime Access Method Encapsulator process Snmpget() PATROL Console

18 © 1999 BMC SOFTWARE, INC. 3/17/99 PATROL SNMP – UNIX PATROL Agent With SNMP Sub Agent Management Station SNMP Agent (Master) PATROL SNMP Master Agent Port 161 Port 8161 Applications Uses Windows SNMP API Applications Uses Windows SNMP API Applications Uses Windows SNMP API Applications Dynamically loaded into Agent space SMUX Port 191 Application SNMP Runtime Access Method Encapsulator process Snmpget() PATROL Console

19 © 1999 BMC SOFTWARE, INC. 3/17/99 Master agent confusion Only needed when accessing PATROL MIB data !! Not needed for :  receiving traps  sending traps  get/set/walk/... operations

20 © 1999 BMC SOFTWARE, INC. 3/17/99 PATROL Agent as SNMP Manager  Receiving traps  Getting information from other SNMP agents  No Master Agent is needed for this !

21 © 1999 BMC SOFTWARE, INC. 3/17/99 Configuring PATROL for SNMP Set the port number and community name for the PATROL SNMP Master Agent  The PATROL SNMP Master Agent/Sub-Agent model is based on an industry standard known as SMUX that allows one or more SNMP Sub- Agents to connect to a single SNMP Master Agent using a TCP SMUX port (TCP port 199 by default). Turn on the SNMP support variable.  agent configuration variable /snmp/agent_auto_start is set to yes, the PATROL Agent starts the SNMP Sub-Agent when the PATROL Agent is started Add the SNMP manager to the list of interested SNMPV1 managers.  The SNMP management console needs to know how to recognize PATROL traps, and what to do about them. On some consoles it involves configuration of internal rules and tables. In others it may involve configuring the "trapd.conf" configuration file. Configure events to send SNMP traps.

22 © 1999 BMC SOFTWARE, INC. 3/17/99 SNMP support variable(s) The SNMPStart parameter is defined within each.km. The “ out of box ” default setting for this parameter is the active state. If active, SNMPStart launches the PATROL SNMP Master Agent (snmpmagt) if it is not started already. SNMPStart then starts the SNMP Sub-Agent. The configuration of the PATROL SNMP Sub Agent is controlled by the values contained in the PATROL configuration file. On UNIX, it is $PATROL_HOME/lib/config.default On Windows NT, it is %PATROL_HOME%\lib\ config.default The important variables are: "/snmp/support" = { REPLACE="yes" }, "/snmp/agent_auto_start" = { REPLACE="yes" }, "/snmp/default_port" = { REPLACE="161" }, "/snmp/master_agent_port" = { REPLACE="1161" }, "/snmp/trap_port" = { REPLACE="162" }, "/snmp/sysName" = { REPLACE = "unknown" }, "/snmp/sysContact" = { REPLACE = " }, "/snmp/sysLocation" = { REPLACE = "BMC Software Inc." }, "/snmp/piV1m_list" = { REPLACE="" }, "/snmp/support" = { REPLACE="yes" }, "/snmp/agent_auto_start" = { REPLACE="yes" }, "/snmp/default_port" = { REPLACE="161" }, "/snmp/master_agent_port" = { REPLACE="1161" }, "/snmp/trap_port" = { REPLACE="162" }, "/snmp/sysName" = { REPLACE = "unknown" }, "/snmp/sysContact" = { REPLACE = " }, "/snmp/sysLocation" = { REPLACE = "BMC Software Inc." }, "/snmp/piV1m_list" = { REPLACE="" },

23 © 1999 BMC SOFTWARE, INC. 3/17/99 Master agent configuration  See file./lib/snmpmagt.cfg UNIX : default port 1161, community public NT : default port 1161, community public  No need for BMC master agent if SMUX master agent available (eg. AIX)  What about HP ? They use emanate master agent = proprietary Run 2 master agents, one on 161 (default) and one on 1161 (Patrol)

24 © 1999 BMC SOFTWARE, INC. 3/17/99 PATROL SNMP Master Agent The configuration of the PATROL SNMP Master Agent is controlled by the values contained in the PATROL SNMP Master Agent configuration file. On UNIX, it is $PATROL_HOME/lib/snmpmagt.cfg. On Windows NT, it is %PATROL_HOME%\lib\snmpmagt.cfg. # GRAMMAR: # # MANAGERmanager # [SEND [ALL | NO ] TRAPS # [TO PORT ] # [WITH COMMUNITY ]] # # COMMUNITY # ALLOW op [,op]* [OPERATIONS] # [USE encrypt ENCRYPTION] # [MEMBERS [, ]* ] # # manager ::= | ipaddr # where: hostname is defined in /etc/hosts # # ipaddr ::= # op ::= ALL | GET | SET | TRAP # encrypt ::=NO COMMUNITYpublic ALLOW ALL OPERATIONS USE NO ENCRYPTION TRANSPORTordinarySNMP OVER UDP SOCKET AT PORT 1161 COMMUNITYpublic ALLOW ALL OPERATIONS USE NO ENCRYPTION TRANSPORTordinarySNMP OVER UDP SOCKET AT PORT 1161

25 © 1999 BMC SOFTWARE, INC. 3/17/99 PATROL Agent as SNMP sub agent  Ability to send traps  Has an accessible MIB  To access MIB, the master agent must be set up correctly

26 © 1999 BMC SOFTWARE, INC. 3/17/99 Sending traps  Automatically : From event catalogs Trap destinations : piV1m_List ( + /snmp/trapMibTable=yes) Filter possibility (see config.default for filter options)  From PSL snmp_trap_send : full control snmp_trap_raise_std_trap : uses piV1m_List

27 © 1999 BMC SOFTWARE, INC. 3/17/99 The PATROL MIB

28 © 1999 BMC SOFTWARE, INC. 3/17/99 MIB Object Groups iso (1) org (3) dod (6) internet (1) mgmt (2) Mib-2 (1) system (1) private (4) enterprises (1) bmc (1031)

29 © 1999 BMC SOFTWARE, INC. 3/17/99 MIB Tables – Indexing a Table Table Column Index OS>%PSL print(snmp_walk("sess47", " ")."\n"); Row Index stringHP stringPATROLAGENT stringSNMP_Test stringCOLLECTORS stringCPU stringDISK stringFILESYSTEM stringKERNEL stringMEMORY stringNETWORK stringNFS stringPATROL stringPRINTER stringPROCESS stringSECURITY stringSMP stringSWAP stringUSERS stringLOG OIDStrLenTypeapplicationName

30 © 1999 BMC SOFTWARE, INC. 3/17/99 bmc (1031) objectsTable (3) na variablesTable (5) applicationsTable (6) patrolMIB (1) patrolAgent (1) patrolConsole (2) patrolObjects (1) patrolTraps (2) agentExecuteCommand (3) objectsMask (1) r/w objectsCwd (2) r/w objectsEntry (1) na objectName (1) r objectDescr (3) r objectRowStatus (4) r/w Objects Table applInstTable (7) parametersTable (8)

31 © 1999 BMC SOFTWARE, INC. 3/17/99 Browsing PATROL namespace - 1  Set objectCwd to the tree you want to inspect for example “/”  Go to the desired “node” get the “objectsTable”, you see NT_CPU set objectCwd to “/CPU” … pick inst from objectsTable set objectCwd to “/CPU/CPU”… pick param from objectsTable set objectCwd to “/CPU/CPU/CPUCpuUtil”  Read the wanted values get the “variablesTable” to find all the attributes  Problem : No way to enforce a “session”, your set of objectCwd might have been overwritten !

32 © 1999 BMC SOFTWARE, INC. 3/17/99 Execute PSL through SNMP  Set variable “agentExecuteCommand”  Agent will execute immediately  Fail/Success from return value of snmp_set()

33 © 1999 BMC SOFTWARE, INC. 3/17/99 Variables Table bmc (1031) objectsTable (3) variablesTable(5) na patrolMIB (1) patrolAgent (1) patrolConsole (2) patrolObjects (1) patrolTraps (2) agentExecuteCommand(3) objectsMask (1) objectsCwd (2) VariablesEntry (1) na variableName (2) r variableType (3) r variableValue (4) r/w variableDesr (5) r/w variableRowStatus (6) r/w

34 © 1999 BMC SOFTWARE, INC. 3/17/99 Direct namespace access – Variables Table  “Hooked” on OID of :.variablesTable.variableEntry.variableValue  Need conversion of namespace variable = length of namespace variable = ascii to value conversion of namespace variable for example : /CPU =  Simply get : bmc  Result : Fastest Namespace access not useful for namespace “discovery” Not “walk-able”

35 © 1999 BMC SOFTWARE, INC. 3/17/99 Applications Table bmc (1031) variablesTable(5) applicationsTable(6) na patrolMIB (1) patrolAgent (1) patrolConsole (2) patrolObjects (1) patrolTraps (2) agentExecuteCommand(3) applicationsEntry (1) na applicationName (1) r applicationState (2) r applWorstInst (3) r applMasterVersion (4) r applMinorRevision (5) r applicationRowStatus (6) r applicationOid (7) na

36 © 1999 BMC SOFTWARE, INC. 3/17/99 Browsing PATROL namespace - 2  Look for the application in “applicationTable”  Use that index to get the instances from the application in “applInstTable”  Use both indexes to get the parameters from the “parametersTable”  Get the required attribute from your parameter  This method is “safe” but requires some processing

37 © 1999 BMC SOFTWARE, INC. 3/17/99 Applications Example OS>%PSL print(snmp_walk("sess47", " ")."\n"); stringHP stringPATROLAGENT stringSNMP_Test stringCOLLECTORS stringCPU stringDISK string FILESYSTEM stringKERNEL stringMEMORY stringNETWORK stringNFS stringPATROL string PRINTER stringPROCESS stringSECURITY stringSMP stringSWAP stringUSERS stringLOG Application ID String Length Type Application Name applicationsTable applicationsEntry (index) applicationName

38 © 1999 BMC SOFTWARE, INC. 3/17/99 Instances Table variablesTable(5) applicationsTable(6) patrolObjects (1) applInstTable(7) na applInstEntry (1) na applInstName (1) r applInstRuleState (2) r applInstStatus (3) r applInstWorstParam (4) r applInstCreateIcon (5) r applInstRowStatus (6) r applInstOid (7) na applInstPInstOid (9) r applInstPApplOid (8) r

39 © 1999 BMC SOFTWARE, INC. 3/17/99 Instances Example OS>%PSL print(snmp_walk("sess47", " ")."\n"); string lj4mv stringlp stringsweetlip stringNutria_HPLaser stringNETWORK stringNFS stringPATROL string PRINTER stringPROCESS stringSECURITY applInstTable applInstEntry (index) applInstName Application ID Application Name Application Instance ID

40 © 1999 BMC SOFTWARE, INC. 3/17/99 Parameters Table parametersTable(8) na parametersEntry (1) na parameterName (1) r parameterState (2) r parameterCurrentTime (3) r parameterCurrentValue (4) r parameterPollingInt (5) r parameterRetries (6) r parameterOutputMode (7) r parameterAutoScale (8) r parameterYaxisMin (9) r parameterYaxisMax (10) r parameterRowStatus (11) r/w parameterObjId (12) na parameterIntValue (13) r parameterActiveStat (14) r/w parameterRunningStat (15) r/w

41 © 1999 BMC SOFTWARE, INC. 3/17/99 Parameters Example OS>%PSL print(snmp_walk("sess47", " ")."\n"); stringlj4mv stringlp stringsweetlip stringNutria_HPLaser stringPATROL string PRINTER stringPROCESS parametersTable parametersEntry (index) parameterName Application ID Parameter ID Application Instance ID OS>%PSL print(snmp_walk("sess47", " ")."\n"); stringPRNQLengthOS >%PSL print(snmp_walk("sess47", " ")."\n"); integer0 applicationName applInstName Value parameterIntValue

42 © 1999 BMC SOFTWARE, INC. 3/17/99 Trap Table bmc (1031) patrolMIB (1) patrolAgent (1) patrolConsole (2) patrolObjects (1) patrolTraps (2) agentExecuteCommand(3) patrolTrapText (1) r patrolTrapOrigin (2) r patrolTrapExtra (3) r piV1mTable (10) na piV1mIp (10) na piV1mPort (10) na piV1mCommunity (10) na piV1mRowStatus (10) r/w

43 © 1999 BMC SOFTWARE, INC. 3/17/99 Enterprise Traps patrolTraps (2) patrolTrapText (1) patrolTrapOrigin (2) patrolTrapExtra (3) piV1mTable (10) piV1mIp (10) piV1mPort (10) piV1mCommunity (10) piV1mRowStatus (10) patrolTrapV1Raised (10) OBJECTS{patrolTrapText} A patrolTrapV2Raised trap is sent when one of the PSL scripts calls snmp_trap()function. patrolTrapV1StateChanged (11) OBJECTS{patrolTrapText} A patrolTrapV1StateChanged is sent when Patrol Agent changes state of one of the discovered application instances. patrolInformation (1) VARIABLES {patrolTrapText, patrolTrapOrigin, patrolTrapExtra} A patrolInformation trap is sent when a corresponding event which may cause to that trap occurred. patrolChangeStatus (2) VARIABLES {patrolTrapText,patrolTrapOrigin,patrolTrapExtra} A patrolChangeStatus is sent when a corresponding event which may cause to that trap occurred. patrolError (3) VARIABLES {patrolTrapText,patrolTrapOrigin,patrolTrapExtra} A patrolError is sent when a corresponding event which may cause to that trap occurred. patrolWarning (4) VARIABLES {patrolTrapText,patrolTrapOrigin, patrolTrapExtra} A patrolWarning is sent when a corresponding event which may cause to that trap occurred. patrolAlarm (5) VARIABLES {patrolTrapText,patrolTrapOrigin, patrolTrapExtra} A patrolAlarm is sent when a corresponding event which may cause to that trap occurred. patrolTrapWarmStart (1) Standard Traps Enterprise Traps (6) [ ] Traps are sent to SNMP V1 entity according to piV1mTable. Traps are sent to SNMP V1 entity according to piV1mTable.

44 © 1999 BMC SOFTWARE, INC. 3/17/99 Trap Example ========== SNMP Trap Received ============= From: ( ) Enterprise Specific (5) Uptime: 0 day(s) 23:34:55 ( ) stringAlarm #2 of global parameter 'NFSSNull' triggered on 'NFS.NFS'. 50 <= <= string/NFS/NFS/NFSSNull string ========== End SNMP Trap Received ============= ========== SNMP Trap Received ============= From: ( ) Enterprise Specific (2) Uptime: 0 day(s) 23:37:56 ( ) stringAlert on 'NFS.NFS.NFSSNull' from global parameter 'NFSSNull' cancelled; exception no longer exists string/NFS/NFS/NFSSNull string ========== End SNMP Trap Received ============= ========== SNMP Trap Received ============= From: ( ) Enterprise Specific (4) Uptime: 0 day(s) 23:40:56 ( ) stringAlarm #1 of global parameter 'NFSSReadDir' triggered on 'NFS.NFS'. 18 <= <= string/NFS/NFS/NFSSReadDir string ========== End SNMP Trap Received ============= patrolTrapText patrolTrapOrigin patrolTrapExtra Enterprise Traps (6) [ ] Source IP address patrolAlarm (5) patrolWarning (4) patrolChangeStatus (2)

45 © 1999 BMC SOFTWARE, INC. 3/17/99 PSL SNMP Functions

46 © 1999 BMC SOFTWARE, INC. 3/17/99 1st step : Starting SNMP subsystem PATROL SNMP functionality can be stopped  snmp_agent_start() starts SNMP support  snmp_agent_stop() stops SNMP support  snmp_agent_config() tells if SNMP support is active

47 © 1999 BMC SOFTWARE, INC. 3/17/99 PSL functions for SNMP managers “All” SNMPv1 functions snmp_(h_)get snmp_(h_)get_next snmp_(h_)set  Plus  snmp_walk (just loop around get_next)  No support for v2(c), v3 !! security, bulk get, new datatypes

48 © 1999 BMC SOFTWARE, INC. 3/17/99 Return format of snmp_functions If type is  String and first character printable OID [strlen] “string” [actual string]  String and first character non-printable OID [strlen] “string” [series of hex]  gauge, timeticks OID [type] [value] “(“[hex-value]”)”  integer, oid, ipAddress OID [type] [value]

49 © 1999 BMC SOFTWARE, INC. 3/17/99 Receiving traps  PSL functions : snmp_trap_ignore : close all listeners (close socket) snmp_trap_listen : start listening for traps (open socket) starts accumulating traps snmp_trap_receive : process incoming traps (block/poll based) only 1 per agent  Functionality is based on a “gentlemen's agreement” Only one receiver for multiple KM’s  Need for a “trap dispatcher” in PATROL ?

50 © 1999 BMC SOFTWARE, INC. 3/17/99 Getting info from other SNMP Agents What is an SNMP session in PATROL ? snmp_open() : opens an SNMP session snmp_close() : closes an SNMP session snmp_config() : lists the SNMP sessions  No real session (like TCP session) because SNMP = connectionless / stateless  just a memory structure, so... Easier for the developer … open session once, reuse the session with the session attributes ! Allows you to set timeouts/retries/port Faster