1. ACI – Monitoring the CLI

2. Agenda Syslog feature Callhome feature Techsupport Show health
Show faults
Show events
Show audits
Show stats

3. Syslog CLI
In the ACI Fabric, one or more logging server-groups can be configured with description, console, logfile and one or more servers with application appropriate severity levels.
These logging server-groups can be assigned to monitoring policy using the logging server-group cli as shown below.
Logging server-group Config:
Description for syslog group:
Local destinations: Console and file

4. Syslog CLI (cont.) Syslog file name is well-known
Remote destinations configuration for syslog
   
NX OS CLI and extended with mgmt and portnum
default <facility>        local7
default <severity-level>  warnings(4)
default mgmt              inb
default portnum           514

5. Syslog CLI (cont.) Example for syslog server configurations:
  server reach.cisco.com
  server reach.cisco.com level local4
  server reach.cisco.com level local4 3 inb 514 
The syslog cli will be effective only for the common monitoring policy as given below example:

6. Callhome CLI
In the ACI Fabric, Callhome configuration can be added for the common monitoring policy as shown below.
Callhome common policy:
Enable audit, fault, event and severity:

7. Callhome CLI (cont.)
Destination-Profile Config:

8. Callhome CLI (cont.)
One or more destinations can be configured: 

9. Callhome CLI (cont.) Query-Profile Config:
One or more query’s can be configured:
Periodic inventory can be configured using scheduler name:

10. Callhome CLI (cont.)
Show commands for callhome are: 

11. Techsupport CLI
In the ACI Fabric, one or more techsupport remote destinations can be configured using following command.
Techsupport remote location configurations:
(config)#  remote path <WORD> <host/ipaddress> <portnum> { ftp | scp | sftp } <username> <remotepath> <password> (just example)
Using above remote locations, the techsupport can be triggered using following command. If remotename is not provided, the techsupport is collected in controllers itself. 
Triggering techsupport controllers and switches:
# trigger techsupport all [remotename <word>]
# trigger techsupport controllers [remotename <word>]
# trigger techsupport switch < , 800> [remotename <word>]

12. Techsupport CLI (cont.)
Once the techsupport is triggered, the status of these techsupports can be seen using following show commands.  
Show commands for techsupport status:
apic1# show techsupport all status
apic1# show techsupport controllers status
apic1# show techsupport switch < , 800> status

13. ACI – CLI Show Commands For Monitoring

14. Prerequisites
You are expected to be familiar with concepts like Managed Object (MO), Management Information Tree (M.I.T), Distinguished Name (DN), Relative Name (RN)
Usage of CLI commands such as faults, eventlog, auditlog, mostats
Basics of GUI
Basics of faults, events, health, stats, audits infra
REST APIs

15. Introduction
Faults, events, health records and audit logs are essential tools for monitoring the administrative and operational state of an ACI fabric as well as troubleshooting current and past issues.
They are the first thing to check when something is not behaving as expected!
Integral part of the ACI model-driven architecture; the data constructs and operational models are common across all features and components.

16. Monitoring CLI – Show Health
The following sections describe using examples the common syntax for show commands like Faults, Events, Health, Audits and Statistics.
Each of these categories can be inter-changeable with the health keyword in the examples below.
Each command is covered in the following sections where Entity refers to one of leaf, spine or tenant.
At each point in the command the ‘?’ can be used to access the help. This returns all the possible keywords
Command Syntax for show health: 
show health [history] [min-change <NUMBER>] [max-hs <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] <— Entity —> 

17. Monitoring CLI – Show Health (cont.)
Sample Command Output (brief) for show health:
Since no tenant name is specified the health score of all tenants are returned

18. Monitoring CLI – Show Health (cont.)
The other possible permuations for "show health” commands which will be applicable for all other monitoring commands like show faults, show events, show audits, show stats are:

19. Monitoring CLI – Show Health (cont.)
For the following show commands, spine can be used in place of leaf:

20. Monitoring CLI – Show Health (cont.)

21. Monitoring CLI – Show Health (cont.)
There are multiple filters provided for the user to narrow the results.
The “show health” command filters are:
end-time Health activity in time interval
History Historical information
max-hs Maximum health score
min-change Minimum change in health score percentage
start-time Health activity in time interval
These filters can be used in conjugation, such as:
The above command will show all the historical health records from the 12th of January that have a maximum health score of 80 and have had a minimum change of 15% for the tenant Nubecentro.

22. Monitoring CLI – Show Faults
The faults command follows a similar syntax to the health command.
Filters are again provided to make the task of querying faults easier for the user.
Entity in this case could be a tenant, leaf, spine or a controller.
Command Syntax for show faults:
show faults [history] [code <fault-code>] [id <fault-ID>] [ack <yes/no>] [lc <lc-state>] [severity <severity-value>] [min-severity <severity-value>] [type <fault-type>] [cause <fault-value>] [last- minutes <NUMBER>] [last-hours <NUMBER>] [last-days <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] [detail] <— Entity —>

23. Monitoring CLI – Show Faults (cont.)
These are the filters available to the user for show faults:
ack Acknowledgment status
cause Cause
code Fault code
detail Detailed faults information
history Historical information
id Fault ID
last-days Fault activity in the last ‘x’ number of days
last-hours Fault activity in last ‘x’ number of hours
last-minutes Fault activity in last ‘x’ number of minutes
lc Lifecycle state
min-severity Minimum severity
severity Severity
start-time Fault activity starting from the time chosen here
end-time Fault activity up to the time chosen here
type Fault Type

24. Monitoring CLI – Show Faults (cont.)
Sample Command Output (brief) for show faults:

25. Monitoring CLI – Show Faults (cont.)
If the user needs to see the full content of the fault records they would use:

26. Monitoring CLI – Show Faults (cont.)
Another example:
The above command shows all faults that occurred in the past 10 days with code “F0765”, severity “minor”, lifecycle “raised” and acknowledgement status “no” for the tenant Nubecentro.

27. Monitoring CLI – Show Faults (cont.)
Accessing fault records:

28. Monitoring CLI – Show Events
The "show events” command is very similar to the "show faults” command with its syntax being:
Command Syntax for show events:
show events [code <event-code>] [id <event-ID>] [cause <event-value>] [last-minutes <NUMBER>] [last-hours <NUMBER>] [last-days <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] [detail] <— Entity —>
Entity once again could be a spine, leaf, tenant or controller.

29. Monitoring CLI – Show Events (cont.)
Sample Command Output (brief) for show events:

30. Monitoring CLI – Show Audits
Command Syntax for show audits:
show audits [id <log-id>] [action <action-type>] [user <user-name>] [last-minutes <NUMBER>] [last-hours <NUMBER>] [last-days <NUMBER>] [start-time <YYYY-MM- DDTHR:MIN:SEC>] [end-time <YYYY-MM-DDTHR:MIN:SEC>] [detail] <- Entity ->
Show audits can be used to view the audit-logs (aaaModLR) as well as the session logs (aaaSessionLR) for the respective entity which can be a leaf, spine, tenant or a controller.
Handy mechanism to track user-initiated configuration changes

31. Monitoring CLI – Show Audits (cont.)
Sample Command Output (brief) for show audits:

32. Monitoring CLI – Show Audits (cont.)
An example where session logs are returned for “show audits” command:

33. Monitoring CLI – With Great Power…
A word of caution – a few commands can be used without an entity specified
For example, simply typing the following commands is valid:
show audits
show events
show faults history
These are expensive queries that retrieve all audit-logs/ event-logs/fault-records that are on the entire system.
An important use-case of such a command is retrieving logs of deleted entities
The following would result in quicker and more filtered response:
apic1# show audits last-minutes 30

34. Monitoring CLI – Show Stats
The “show stats” command will show the relevant statistics collected for the entity which can be a leaf, spine or a tenant for a specific granularity.
Command Syntax for show statistics:
show stats granularity <granularity-value> [history] [cumulative] <- Entity ->
The granularity value refers to the sampling interval size which can be one of :
5 minutes, 15 minutes, 1 hour, 1 day, 1 week, 1 month, 1 quarter or 1 year
Cumulative is used to specify that the user wishes to view statistical information that is
cumulative in nature. These values are never reset to zero.
Choosing a granularity value is mandatory.

35. Monitoring CLI – Show Stats
For Gauge counters (bytes-per-second, temperature etc.) the Average value for that sampling interval is shown.
For Cumulative counters (bytes, packets etc.) the Periodic value for that sampling interval is shown by default.
The “cumulative” keyword has to be explicitly specified to return cumulative stats.
“history” can be used to view historical data for a given interval.

36. Monitoring CLI – Show Stats
apic1# show stats granularity 15min tenant Nubecentro
Start Time Counter Value Unit
:35:25 Egress multicast bytes 54,658,486,264 bytes
:35:25 Egress multicast packets 5,548,335,587 packets
:35:25 Egress unicast bytes 57,453,658,816 bytes
:35:25 Egress unicast packets 4,258,863,248 packets
:35:25 Ingress drop bytes 658,186,364 bytes
:35:25 Ingress drop packets 486,264 packets
:35:25 Ingress flood bytes 62,548,354,264 bytes
:35:25 Ingress flood packets 5,458,756,144 packets
:35:25 Ingress multicast bytes 55,652,589,724 bytes
:35:25 Ingress multicast packets 4,682,589,952 packets
:35:25 Ingress unicast bytes 52,471,852,951 bytes
:35:25 Ingress unicast packets 5,951,753,642 packets
:35:25 Health score 95 score

37. Monitoring CLI – Show Stats

38. Monitoring CLI – Show Stats (cont.)