1. Click to add text IBM Software Group ® WebSphere ® Support Technical Exchange Ask the Experts HA Manager in WebSphere Application Server 12 September 2013

2. IBM Software Group 2 WebSphere ® Support Technical Exchange 2 Agenda  Introduce the panel of experts  Introduce High Availability(HA) Manager Topics  Answer questions submitted by email (8 questions)  Open telephone lines for questions  Summarize highlights

3. IBM Software Group 3 Panel of Experts PanelistRole at IBM Kumaran Nathan WebSphere Application Server Level 2 Support Krishna JaladhiWebSphere Application Server Level 2 Support Salnave (Sal) Saint- Fort WebSphere Application Server Level 2 Support Hobert Bush IIIWebSphere Application Server Level 2 Support David FollisWebSphere Application Server Development

4. IBM Software Group WebSphere ® Support Technical Exchange 4 Introduction  We will be covering a number of questions that cover various HA Manager Topics.  HA Manager service is a framework that enables components to make singleton services highly available.  HA Enabled Functions:  Messaging Engine failover  WLM routing  DRS (Data Replication Service)  Transaction log recovery

5. IBM Software Group 5 WebSphere ® Support Technical Exchange 5 Question 1  What should be the heap size of core group coordinator?

6. IBM Software Group WebSphere ® Support Technical Exchange 6 Answer to Question 1  The coordinator manages the failover of highly available singleton services and distributes live server state data to interested core group members. The coordinator uses some CPU and memory (JVM heap) resources to perform these tasks. In some configurations, the amount of resources that the coordinator uses might be large.  The coordinator election occurs whenever the view changes. Electing a new coordinator uses a lot of resources because this process causes increased network traffic and CPU consumption.  Some of the factors that affect the amount of state data include:  Number of servers in the topology.  Number of applications.  Enabled features (Web services).  Add-on products (WebSphere Virtual Enterprise).

7. IBM Software Group WebSphere ® Support Technical Exchange 7 Answer to Question 1 – Cont..  Multiple coordinators :  The default coordinator (1 coordinator by default) is sufficient for most installations and applications. Use multiple coordinators when the core group member that is selected as the coordinator uses noticeably more memory or CPU than similar core group members  Preferred coregroup coordinator :  It’s a good practice to specify coregroup coordinator  Preferred coordinator servers should be core group processes that are cycled as infrequently as possible  The preferred coordinator servers should also be hosted on machines with excess capacity.  512MB on the coordinator server is sufficient for a coregroup with less then 20 members. The size can be adjusted to 1 GB, if the coregroup size is more than 30.

8. IBM Software Group 8 WebSphere ® Support Technical Exchange 8 Question 2  At times I've seen some HA-enabled JVMs take quite a long time to start up. What is the cause of this, and how can this be remedied?

9. IBM Software Group WebSphere ® Support Technical Exchange 9 Answer to Question 2  The time required to start a HA Enabled server depends on a number of factors  The number of members in the core group  The number of core group members that are started concurrently  The existing load on the machines hosting the core group members  Hardware, Network, CPU and memory  Approximately the server startup might take 2 to 5 minutes for 100 servers in a coregroup.  Startup time can be reduced by creating preferred coregroup coordinator and by reducing the number of members in a coregroup.

10. IBM Software Group 10 WebSphere ® Support Technical Exchange 10 Question 3  CWRLS0030W message continuously logged in SystemOut. How do I resolve the issue?

11. IBM Software Group WebSphere ® Support Technical Exchange 11 Answer to Question 3  Background:  Every WebSphere process is a member of a HA Manager core group. In the case where a clustered Application Server is started, WebSphere services, including the HA Manager and Transaction Manager are initialized and started.  Before the Transaction Manager can complete its startup it must have exclusive ownership of its transaction recovery log file.  In WebSphere Application Server V6.0 and higher, the Transaction Manager relies on the HA Manager to assign it ownership of its transaction log file. This is true even if the “highly available transaction log” feature is not used.  The Transaction Manager logs a CWRLS0030W message when it is waiting for the HA Manager to assign it ownership of its transaction log.  Cause :  The HA Manager is having difficulty initializing its view and thus cannot support the recovery service.  DCSV1032I : DCS received a new connection.  Application servers report this event when another member of the core group connects to them. This is normal and occurs frequently during startup.  DCSV0008I : Failed to form an initial view.  This message reports the number of tries as well as a list of all the members in the view, which will be a number less than the configured members of the coregroup. This message needs to be examined with other messages, such as DCSV1036W, which provide information about members to which a connection could not be made.

12. IBM Software Group WebSphere ® Support Technical Exchange 12 Answer to Question 3 – Cont..  DCSV1036W: An unusual connectivity state occurred with a core group member.  This message is a warning that indicates an issue with establishing a connection between two JVM instances. The cause may be due to the target process being unresponsive (OOM or hung), network configuration issues, or problems on the underlying network.  DCSV8030I:An attempt to establish a new view with the indicated member failed.  The reason will give additional information about the root cause of the failure.  Set following core group custom properties  IBM_CS_HAM_PROTOCOL_VERSION Facilitate core group bridge scalability. This version is recommended for topologies that contain multiple core groups and core group bridges as part of their configuration.  IBM_CS_WIRE_FORMAT_VERSION Facilitate core group scalability improvements, and more support for large topologies.  IBM_CS_OOM_ACTION When this property is set to Isolate, if an OutOfMemoryException occurs when a network message is being sent or received, these exception handlers stop High Availability Manager communications to the Out of Memory process.  http://www-01.ibm.com/support/docview.wss?uid=swg21245012

13. IBM Software Group 13 WebSphere ® Support Technical Exchange 13 Question 4  I've a question in regards to HA manager recovery due to network 'instabilities'. Given a Websphere cell spread over 2 sites ( it has Websphere Process Server ), 8 clusters with 44 JVMs and a deployment Manager and 4 nodeagents and the situation where the network between the 2 sites has intermittent switch failures and also intermittent rogue network spikes what would be the best configuration for HA manager core groups and policies? At present we have one defaultCoreGroup with all 49 JVMS, default protocol providers, discovery heartbeat periods. There are 15 Policies for the various clusters, sibus and TM policy but none of them have preferred servers set? There are no core group bridge settings which I believe is correct. In the event of multiple DCS connection failures and recoveries what is the impact on the 'running' JVMs? Does the JVM halt or degrade while DCS views are being updated? Would having the preferred servers listed under preferred servers be more efficient and less of an overhead in a network spikes and dropout glitches scenario? Rest assured we are trying to fix the network as well....

14. IBM Software Group WebSphere ® Support Technical Exchange 14 Answer to Question 4  HA Manager will try to recover from any disconnects between group members due to network issues. Generally it depends on how long network recovery takes from failure. Best settings are to use latest wire protocol and HAM protocol if you are using core group bridges.  HA core group preferred coordinator doesn’t applicable to HA Policies, it is best practice to elect preferred coordinators.  DCS connection failures doesn’t halt application servers normal functionality, there may be excessive connection attempts to recover from connection Failures from HA Manager and also view change events occurs.  Having preferred coordinator server for HA Manager is always recommend but in the event of network failures it may not help much as HAM itself is trying to recover form network failure.

15. IBM Software Group 15 WebSphere ® Support Technical Exchange 15 Question 5  How do I know when the core group co-ordinator is on a CPU-constrained system?

16. IBM Software Group WebSphere ® Support Technical Exchange 16 Answer to Question 5  Both HA Manager and the underlying DCS component have code that needs to run regularly  Both components track when they get control compared to when they expect to get control  Messages are issued when they don’t get control when expected and indicate how far off things are  One message with a large time gap indicated or recurring messages with smaller gaps can cause issues with HA Manager and the core group view

17. IBM Software Group WebSphere ® Support Technical Exchange 17 Answer to Question 5 – Cont..  Two messages issued by HAMgr and DCS indicate CPU starvation issues :  DCSV0004W: DCS Stack {0} at Member {1}: Did not receive adequate CPU time slice. Last known CPU usage time at {2}. Inactivity duration was {3} seconds.  HMGR0152W: CPU Starvation detected. Current thread scheduling delay is {0} seconds.  Examples :  DCSV0004W: DCS Stack DefaultCoreGroup at Member abc\xyz\dmgr: Did not receive adequate CPU time slice. Last known CPU usage time at 20:04:16:018 CEST. Inactivity duration was 2473 seconds.  HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 5 seconds.

18. IBM Software Group 18 WebSphere ® Support Technical Exchange 18 Question 6  What are the advantages and disadvantages of disabling the HA Manager?

19. IBM Software Group WebSphere ® Support Technical Exchange 19 Answer to Question 6 - Advantages  Disabling the high availability manager consumes valuable system resources, such as CPU cycles, heap memory, and sockets. For large core groups, the amount of resources that the high availability manager consumes can become significant.  high availability manager frees these resources.  Isolation of JVM resources for problem determination.  Do not disable the high availability manager on administrative processes, such as node agents and the deployment manager, unless the high availability manager is disabled on all application server processes in that core group.

20. IBM Software Group WebSphere ® Support Technical Exchange 20 Answer to Question 6 - Disadvantages  Memory-to-memory replication  EJB stateful session bean failover is enabled for an application server.  Singleton failover  The cluster is configured to use the high availability manager to manage the recovery of transaction logs  Workload management routing  On-demand configuration routing

21. IBM Software Group 21 WebSphere ® Support Technical Exchange 21 Question 7  How do I decide which Application Server process should be a core group co-ordinator?

22. IBM Software Group WebSphere ® Support Technical Exchange 22 Answer to Question 7  By default, the coordinator functionality is assigned to a single process. The coordinator consumes extra memory and CPU cycles.  It is a best practice to create stand-alone application server processes that do not host applications to function as preferred coordinators.  Preferred coordinator server should be cycled as infrequently as possible and should be hosted on machines with excess capacity.  The default core group coordinator can be changed by going to:  Servers –> coregroups-> coregroup settings -> Default Coregroup -> preferred coordinator servers

23. IBM Software Group 23 WebSphere ® Support Technical Exchange 23 Question 8  What is the recommended way to disable HA?

24. IBM Software Group WebSphere ® Support Technical Exchange 24 Answer to Question 8  If you are running on Version 6.1.0.16 or earlier, you must use the wsadmin tool to disable a high availability manager or to enable a high availability manager that you previously disabled.  If you are running on Version 6.1.0.17 or later, you can use either the wsadmin tool or the administrative console to disable a high availability manager or to enable a high availability manager that you previously disabled.

25. IBM Software Group 25 WebSphere ® Support Technical Exchange 25 Open Lines for Questions

26. IBM Software Group 26 WebSphere ® Support Technical Exchange 26 Connect with us! 1.Get notified on upcoming webcasts Send an e-mail to wsehelp@us.ibm.com with subject line “wste subscribe” to get a list of mailing lists and to subscribewsehelp@us.ibm.com 2.Tell us what you want to learn Send us suggestions for future topics or improvements about our webcasts to wsehelp@us.ibm.comwsehelp@us.ibm.com 3.Be connected! Connect with us on Facebook Connect with us on TwitterFacebookTwitter

27. IBM Software Group 27 WebSphere ® Support Technical Exchange 27 Summary

28. IBM Software Group 28 WebSphere ® Support Technical Exchange 28 Additional WebSphere Product Resources  Learn about upcoming WebSphere Support Technical Exchange webcasts, and access previously recorded presentations at: http://www.ibm.com/software/websphere/support/supp_tech.html http://www.ibm.com/software/websphere/support/supp_tech.html  Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: http://www.ibm.com/developerworks/websphere/community/ http://www.ibm.com/developerworks/websphere/community/  Join the Global WebSphere Community: http://www.websphereusergroup.org http://www.websphereusergroup.org  Access key product show-me demos and tutorials by visiting IBM ® Education Assistant: http://www.ibm.com/software/info/education/assistant http://www.ibm.com/software/info/education/assistant  View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically: http://www.ibm.com/software/websphere/support/d2w.html http://www.ibm.com/software/websphere/support/d2w.html  Sign up to receive weekly technical My Notifications emails: http://www.ibm.com/software/support/einfo.html http://www.ibm.com/software/support/einfo.html