ACI – Monitoring the CLI

Agenda Syslog feature Callhome feature Techsupport Show health Show faults Show events Show audits Show stats

Syslog CLI In the ACI Fabric, one or more logging server-groups can be configured with description, console, logfile and one or more servers with application appropriate severity levels. These logging server-groups can be assigned to monitoring policy using the logging server-group cli as shown below. Logging server-group Config: Description for syslog group: Local destinations: Console and file

Syslog CLI (cont.) Syslog file name is well-known Remote destinations configuration for syslog     NX OS CLI and extended with mgmt and portnum default <facility>        local7 default <severity-level>  warnings(4) default mgmt              inb default portnum           514

Syslog CLI (cont.) Example for syslog server configurations:   server reach.cisco.com   server reach.cisco.com level local4   server reach.cisco.com level local4 3 inb 514  The syslog cli will be effective only for the common monitoring policy as given below example:

Callhome CLI In the ACI Fabric, Callhome configuration can be added for the common monitoring policy as shown below. Callhome common policy: Enable audit, fault, event and severity:

Callhome CLI (cont.) Destination-Profile Config:

Callhome CLI (cont.) One or more destinations can be configured: 

Callhome CLI (cont.) Query-Profile Config: One or more query’s can be configured: Periodic inventory can be configured using scheduler name:

Callhome CLI (cont.) Show commands for callhome are: 

Techsupport CLI In the ACI Fabric, one or more techsupport remote destinations can be configured using following command. Techsupport remote location configurations: (config)#  remote path <WORD> <host/ipaddress> <portnum> { ftp | scp | sftp } <username> <remotepath> <password> (just example) Using above remote locations, the techsupport can be triggered using following command. If remotename is not provided, the techsupport is collected in controllers itself.  Triggering techsupport controllers and switches: # trigger techsupport all [remotename <word>] # trigger techsupport controllers [remotename <word>] # trigger techsupport switch <101-999, 800> [remotename <word>]

Techsupport CLI (cont.) Once the techsupport is triggered, the status of these techsupports can be seen using following show commands.   Show commands for techsupport status: apic1# show techsupport all status apic1# show techsupport controllers status apic1# show techsupport switch <101-999, 800> status

ACI – CLI Show Commands For Monitoring

Prerequisites You are expected to be familiar with concepts like Managed Object (MO), Management Information Tree (M.I.T), Distinguished Name (DN), Relative Name (RN) Usage of CLI commands such as faults, eventlog, auditlog, mostats Basics of GUI Basics of faults, events, health, stats, audits infra REST APIs

Introduction Faults, events, health records and audit logs are essential tools for monitoring the administrative and operational state of an ACI fabric as well as troubleshooting current and past issues. They are the first thing to check when something is not behaving as expected! Integral part of the ACI model-driven architecture; the data constructs and operational models are common across all features and components.

Monitoring CLI – Show Health The following sections describe using examples the common syntax for show commands like Faults, Events, Health, Audits and Statistics. Each of these categories can be inter-changeable with the health keyword in the examples below. Each command is covered in the following sections where Entity refers to one of leaf, spine or tenant. At each point in the command the ‘?’ can be used to access the help. This returns all the possible keywords Command Syntax for show health:  show health [history] [min-change <NUMBER>] [max-hs <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] <— Entity —> 

Monitoring CLI – Show Health (cont.) Sample Command Output (brief) for show health: Since no tenant name is specified the health score of all tenants are returned

Monitoring CLI – Show Health (cont.) The other possible permuations for "show health” commands which will be applicable for all other monitoring commands like show faults, show events, show audits, show stats are:

Monitoring CLI – Show Health (cont.) For the following show commands, spine can be used in place of leaf:

Monitoring CLI – Show Health (cont.)

Monitoring CLI – Show Health (cont.) There are multiple filters provided for the user to narrow the results. The “show health” command filters are: end-time Health activity in time interval History Historical information max-hs Maximum health score min-change Minimum change in health score percentage start-time Health activity in time interval These filters can be used in conjugation, such as: The above command will show all the historical health records from the 12th of January that have a maximum health score of 80 and have had a minimum change of 15% for the tenant Nubecentro.

Monitoring CLI – Show Faults The faults command follows a similar syntax to the health command. Filters are again provided to make the task of querying faults easier for the user. Entity in this case could be a tenant, leaf, spine or a controller. Command Syntax for show faults: show faults [history] [code <fault-code>] [id <fault-ID>] [ack <yes/no>] [lc <lc-state>] [severity <severity-value>] [min-severity <severity-value>] [type <fault-type>] [cause <fault-value>] [last- minutes <NUMBER>] [last-hours <NUMBER>] [last-days <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] [detail] <— Entity —>

Monitoring CLI – Show Faults (cont.) These are the filters available to the user for show faults: ack Acknowledgment status cause Cause code Fault code detail Detailed faults information history Historical information id Fault ID last-days Fault activity in the last ‘x’ number of days last-hours Fault activity in last ‘x’ number of hours last-minutes Fault activity in last ‘x’ number of minutes lc Lifecycle state min-severity Minimum severity severity Severity start-time Fault activity starting from the time chosen here end-time Fault activity up to the time chosen here type Fault Type

Monitoring CLI – Show Faults (cont.) Sample Command Output (brief) for show faults:

Monitoring CLI – Show Faults (cont.) If the user needs to see the full content of the fault records they would use:

Monitoring CLI – Show Faults (cont.) Another example: The above command shows all faults that occurred in the past 10 days with code “F0765”, severity “minor”, lifecycle “raised” and acknowledgement status “no” for the tenant Nubecentro.

Monitoring CLI – Show Faults (cont.) Accessing fault records:

Monitoring CLI – Show Events The "show events” command is very similar to the "show faults” command with its syntax being: Command Syntax for show events: show events [code <event-code>] [id <event-ID>] [cause <event-value>] [last-minutes <NUMBER>] [last-hours <NUMBER>] [last-days <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] [detail] <— Entity —>   Entity once again could be a spine, leaf, tenant or controller.

Monitoring CLI – Show Events (cont.) Sample Command Output (brief) for show events:

Monitoring CLI – Show Audits Command Syntax for show audits: show audits [id <log-id>] [action <action-type>] [user <user-name>] [last-minutes <NUMBER>] [last-hours <NUMBER>] [last-days <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] [detail] <- Entity -> Show audits can be used to view the audit-logs (aaaModLR) as well as the session logs (aaaSessionLR) for the respective entity which can be a leaf, spine, tenant or a controller. Handy mechanism to track user-initiated configuration changes

Monitoring CLI – Show Audits (cont.) Sample Command Output (brief) for show audits:

Monitoring CLI – Show Audits (cont.) An example where session logs are returned for “show audits” command:

Monitoring CLI – With Great Power… A word of caution – a few commands can be used without an entity specified For example, simply typing the following commands is valid: show audits show events show faults history These are expensive queries that retrieve all audit-logs/ event-logs/fault-records that are on the entire system. An important use-case of such a command is retrieving logs of deleted entities The following would result in quicker and more filtered response: apic1# show audits last-minutes 30

Monitoring CLI – Show Stats The “show stats” command will show the relevant statistics collected for the entity which can be a leaf, spine or a tenant for a specific granularity. Command Syntax for show statistics: show stats granularity <granularity-value> [history] [cumulative] <- Entity -> The granularity value refers to the sampling interval size which can be one of : 5 minutes, 15 minutes, 1 hour, 1 day, 1 week, 1 month, 1 quarter or 1 year Cumulative is used to specify that the user wishes to view statistical information that is cumulative in nature. These values are never reset to zero. Choosing a granularity value is mandatory.

Monitoring CLI – Show Stats For Gauge counters (bytes-per-second, temperature etc.) the Average value for that sampling interval is shown. For Cumulative counters (bytes, packets etc.) the Periodic value for that sampling interval is shown by default. The “cumulative” keyword has to be explicitly specified to return cumulative stats. “history” can be used to view historical data for a given interval.

Monitoring CLI – Show Stats apic1# show stats granularity 15min tenant Nubecentro Start Time Counter Value Unit ------------------------------------------------------------------------- 2016-03-17 11:35:25 Egress multicast bytes 54,658,486,264 bytes 2016-03-17 11:35:25 Egress multicast packets 5,548,335,587 packets 2016-03-17 11:35:25 Egress unicast bytes 57,453,658,816 bytes 2016-03-17 11:35:25 Egress unicast packets 4,258,863,248 packets 2016-03-17 11:35:25 Ingress drop bytes 658,186,364 bytes 2016-03-17 11:35:25 Ingress drop packets 486,264 packets 2016-03-17 11:35:25 Ingress flood bytes 62,548,354,264 bytes 2016-03-17 11:35:25 Ingress flood packets 5,458,756,144 packets 2016-03-17 11:35:25 Ingress multicast bytes 55,652,589,724 bytes 2016-03-17 11:35:25 Ingress multicast packets 4,682,589,952 packets 2016-03-17 11:35:25 Ingress unicast bytes 52,471,852,951 bytes 2016-03-17 11:35:25 Ingress unicast packets 5,951,753,642 packets 2016-03-17 11:35:25 Health score 95 score

Monitoring CLI – Show Stats

Monitoring CLI – Show Stats (cont.)