1. VERITAS Cluster Server for Solaris Event Notification

2. VCS_3.5_Solaris_R3.5_20020915 9-2 Objectives After completing this lesson, you will be able to: Describe the VCS notifier component. Configure the notifier to signal changes in cluster status. Describe SNMP configuration. Describe event triggers. Configure triggers to provide notification.

3. VCS_3.5_Solaris_R3.5_20020915 9-3 Notification How VCS performs notification:  The had daemon sends a message to the notifier daemon when an event occurs.  The notifier daemon formats the event message and sends an SNMP trap or e-mail message (or both) to designated recipients. had notifier SMTP SNMP had

4. VCS_3.5_Solaris_R3.5_20020915 9-4 Message Severity Levels had notifierSMTPSNMP Error SevereError Warning Information Concurrency violation Service group is online. Resource has faulted. Agent has faulted

5. VCS_3.5_Solaris_R3.5_20020915 9-5 Message Queues  The had daemon stores a message in a queue when an event is detected.  The message is sent over the private cluster network to all other had daemons to replicate the message queue.  The notifier daemon can be started on another system in case of failure without loss of messages. had notifier SMTP SNMP had   notifier SMTP SNMP Replicated Queue

6. VCS_3.5_Solaris_R3.5_20020915 9-6 Configuring Notifier The notifier daemon can be started and monitored by the NotifierMngr resource. Attributes define recipients and severity levels. For example: SmtpServer = "smtp.acme.com" SmtpRecipients = { "admin@acme.com" = Warning }  notifier NIC NotifierMngr  NIC NotifierMngr

7. VCS_3.5_Solaris_R3.5_20020915 9-7 The NotifierMngr Agent Functions: Starts, stops, and monitors the notifier daemon Required attribute: PathName Full path of the notifier daemon Required attributes for SMTP e-mail notification: SmtpServer Host name of the SMTP e-mail server SmtpRecipients E-mail address and message severity level for each recipient Required attribute for SNMP notification: SnmpConsoleName of the SNMP manager and message severity level

8. VCS_3.5_Solaris_R3.5_20020915 9-8 The NotifierMngr Resource Optional attributes: MessagesQueue Number of messages in queue; default = 30 NotifierListeningPort TCP/IP port number; default =14144 SnmpdTrapPort TCP/IP port to which SNMP traps are sent; default=162 SnmpCommunity Community ID for the SNMP manager; default = "public“ Example resource configuration: NotifierMngr Notify_Ntfr ( PathName = "/opt/VRTSvcs/bin/notifier" SnmpConsoles = { snmpserv = Information } SmtpServer = "smtp.your_company.com" SmtpRecipients = { "vcsadmin@your_company.com" = SevereError } )

9. VCS_3.5_Solaris_R3.5_20020915 9-9 SNMP Configuration Load MIB for VCS traps into SNMP console. For HP OpenView Network Node Manager, merge events: xnmevents -merge vcs_trapd VCS SNMP configuration files: –/etc/VRTSvcs/snmp/vcs.mib –/etc/VRTSvcs/snmp/vcs_trapd

10. VCS_3.5_Solaris_R3.5_20020915 9-10 The ResourceOwner Attribute VCS sends an e-mail message to the account specified in the ResourceOwner attribute when notification is configured and the attribute is defined for a resource. An entry is also created in the log file: TAG_E 2001/12/03 11:23:48 VCS:10304:Resource file1 (Owner=daniel, Group testgroup) is offline on machine1 These events cause VCS to send notification: ResourceStateUnknown, ResourceMonitorTimeout, ResourceNotGoingOffline, ResourceRestartingByAgent, ResourceWentOnlineByItself, ResourceFaulted To set the ResourceOwner attribute: hares –modify res_name ResourceOwner daniel

11. VCS_3.5_Solaris_R3.5_20020915 9-11 Event Triggers How VCS performs notification: 1. VCS determines if notification is enabled. –If disabled, no action is taken. –If enabled, VCS runs hatrigger with event-specific parameters. The hatrigger script invokes the event-specific trigger script with parameters passed by VCS. The event trigger script performs the notification tasks.

12. VCS_3.5_Solaris_R3.5_20020915 9-12 Types of Triggers postoffline Service group went offlinePostOffline resnotoff Resource not offlineResNotOff loadwarning System is overloadedLoadWarning resstatechange Resource changed stateResStateChange nofailover Service group cannot failoverNoFailover unable_to_restart_had HAD cannot be restartedUnableToRestartHad postonline Service group went onlinePostOnline preonline Group about to come onlinePreOnline violation Resource online on more than one system Violation injeopardy Cluster in jeopardyInJeopardy sysoffline System went offlineSysOffline resfault Resource faultedResFault Script NameDescriptionTrigger

13. VCS_3.5_Solaris_R3.5_20020915 9-13 Configuring Triggers Triggers enabled by presence of script file: –ResFault –ResNotOff –SysOffline –InJeopardy –Violation –NoFailover –PostOffline –PostOnline –LoadWarning –UnableToRestartHAD Triggers configured by service group attributes : –PreOnline –ResStateChange Triggers configured by default : –Violation

14. VCS_3.5_Solaris_R3.5_20020915 9-14 Sample Triggers Sample trigger scripts include example code to send an e-mail message. Mail must be configured on the system invoking trigger to use sample e-mail code. # Here is a sample code to notify a bunch of users. # @recipients=("username@servername.com"); # $msgfile="/tmp/resnotoff$2"; # `echo system = $ARGV[0], resource = $ARGV[1] > $msgfile`; # # foreach $recipient (@recipients) { # # Must have elm setup to run this. # `elm -s resnotoff $recipient < $msgfile`; # } #`rm $msgfile`;

15. VCS_3.5_Solaris_R3.5_20020915 9-15 ResFault Trigger Provides notification that a resource has faulted Arguments to resfault : –system : Name of the system where the resource faulted –resource : Name of the faulted resource

16. VCS_3.5_Solaris_R3.5_20020915 9-16 ResNotOff Trigger Provides notification that a resource has not been taken offline If a resource is not offline on one system, the service group cannot be brought online on another. VCS cannot fail over the service group in the event of a fault, because the resource will not come offline. Arguments to resnotoff : –system : Name of the system where the resource is not offline –resource : Name of the resource that is not offline

17. VCS_3.5_Solaris_R3.5_20020915 9-17 ResStateChange Trigger Provides notification that a resource has changed state Set at the service group level by the ResStateChange attribute: –hagrp serv_grp -modify TriggerResStateChange Arguments to resstatechange : –system : Name of the system where the resource faulted –resource : Name of the faulted resource –previous_state : State of the resource before change –new_state : State of the resource after change

18. VCS_3.5_Solaris_R3.5_20020915 9-18 SysOffline Trigger Provides notification that a system has gone offline Executed on another system when no heartbeat is detected Arguments to sysoffline : –system : Name of the system that went offline –systemstate : Value of the SysState attribute for the offline system

19. VCS_3.5_Solaris_R3.5_20020915 9-19 NoFailover Trigger Run when VCS determines that a service group cannot fail over Executed on the lowest numbered system in a running state when the condition is detected Arguments to nofailover : –systemlastonline : Name of the last system where the service group is online or partially online –service_group : Name of the service group that cannot fail over

20. VCS_3.5_Solaris_R3.5_20020915 9-20 Summary You should now be able to: Describe the VCS notifier component. Configure the notifier to signal changes in cluster status. Describe SNMP configuration. Describe event triggers. Configure triggers to provide notification.

21. VCS_3.5_Solaris_R3.5_20020915 9-21 Lab 8: Event Notification Triggers resfault nofailover sysoffline Student Red Student Blue resfault nofailover sysoffline RedNFSSG ClusterService BlueNFSSG webip webnic notifier