STP Enhancements for z10 EC, z10 BC, z9 EC, z9 BC Noshir Dhondy (retired) November 12, 2008

Server Time Protocol (STP) Enhancements summary Enhanced Accuracy to an External Time Source (ETS) Available on z10 and z9 (EC and BC) 10 microseconds accuracy vs. 100 milliseconds accuracy when using either Dial out from HMC or NTP Client support as the ETS Utilizes highly stable, accurate Pulse Per Second (PPS) output from NTP servers available from some vendors Continuous Availability of External Time Source (ETS) ETS configured for Backup Time Server (BTS) can be used for time adjustments to maintain ETS accuracy, if ETS configured for Preferred Time Server (PTS) not accessible Applies to NTP server support with or without PPS Enhanced STP recovery when Internal Battery Feature is in use Designed to enable BTS to take over as the CTS if Power outage of PTS/CTS Power outage of site where PTS/CTS is located

Server Time Protocol (STP) Enhancements summary (continued) System Management enhancement - STP configuration and time information saved across Power on Resets or power outages Available on z10 and z9 (EC and BC), z990 and z890 Does not require initializing the time or reassigning PTS/CTS roles “One server” configuration enhancement Enhanced security - NTP Server on Hardware Management Console Applicable only for z10 and z9 (EC and BC) NTP Client on Support Element (SE) can access NTP server using a secure LAN connection API to automate STP CTN Reconfiguration Allows automation to reconfigure special roles of servers (PTS, BTS, Arbiter) after a recovery action Maintains recovery capability in the event of a second failure

Enhanced Accuracy to an External Time Source (ETS)

Enhanced Accuracy to an External Time Source (ETS) Simple Network Time Protocol (SNTP) client support added to Support Element (SE) code of z9 and above servers to interface to Network Time Protocol (NTP) servers NTP Client support GA October 2007 Time reference for NTP server can be GPS, dial-out, DCF-77 etc NTP servers provide multiple NTP outputs Time information Some NTP servers also provide a highly stable, accurate “Pulse per second” (PPS) output ETR card of z10 EC, z10 BC, z9 EC and z9 BC has additional PPS input Each System z has 2 ETR cards providing capability of attaching to two NTP Servers with PPS output for redundancy STP utilizes the PPS signal to provide additional accuracy Accuracy to PPS – 10 microseconds

Enhanced Accuracy to an External Time Source (ETS) (continued) Only the Current Time Server (CTS) steers the time based on: Timing information sent by Support Element (SE) code PPS signal received by its PPS port on ETR card Recommend configuring NTP server with PPS on PTS/CTS as well as BTS Provides ETS capability when BTS becomes or is assigned the CTS role z990, z890 can only dial out from HMC for an ETS Does not have NTP client capability

Enhanced Accuracy to an External Time Source (ETS) - ETS redundancy on same server (PTS/CTS) example

Enhanced Accuracy to an External Time Source (ETS) - ETS redundancy on PTS and BTS example z10 EC PTS/CTS z990 Arbiter z9 BC (BTS)

Continuous Availability of External Time Source (ETS)

Continuous Availability of ETS - two site example BTS transmits adjustment information to PTS/CTS based on its NTP/PPS data When PTS/CTS detects failures associated with its NTP/PPS data PTS/CTS switches to using NTP/PPS data from BTS CTS role DOES NOT switch to BTS

Enhanced STP recovery when Internal Battery Feature is in use

Power Outage PTS/CTS with Internal Battery Feature With Internal Battery Feature (IBF) on CEC1 CEC1 power outage, enters IBF state CEC1 notifies CEC2 it is running on IBF CEC2 waits for 30 seconds to take action Could be a power glitch If notified within 30 seconds that CEC1 back to “normal power”, no further action If CEC1 in IBF state > 30 seconds, CEC2 takes over as the CTS CEC1 becomes S2 until IBF no longer functional and power drops CEC1 power resumes Automatic re-takeover as PTS/CTS HMC CEC1 PTS/CTS S1 P1 CEC2 BTS S2 P3 Coupling links P2 CEC power outage in same data center HMC CEC1 PTS/CTS S1 P1 CEC2 BTS S2 P3 Coupling links P2 Site power outage – 2 data centers

Power outage of data center (Site 1) with PTS and Arbiter HMC CEC1 PTS/CTS S1 P1 CEC2 (BTS) S2 Coupling links CEC3 Arbiter P3 CEC4 P4 Site 1 Site 2 P2 With Internal Battery Feature (IBF) on CEC1 and CEC3 Site 1 power outage CEC1 and CEC3 enter IBF state CEC1 and CEC3 notify CEC2 it is running on IBF CEC2 waits for 30 seconds to take action Could be a power glitch If notified within 30 seconds that CEC1 back to “normal power”, no further action If CEC1 and CEC3 in IBF state > 30 seconds, CEC2 takes over as the CTS Not dependent on Arbiter Assisted Recovery CEC1 becomes S2 until IBF no longer functional and power drops CEC1 power resumes Automatic re-takeover as PTS/CTS

IBF Recommendations Single data center Two data centers IBF only protects for server power outage CTN with 2 servers, install IBF on at least the PTS/CTS Also recommend IBF on BTS to provide recovery protection when BTS is the CTS CTN with 3 or more servers IBF not required to recover from CTS power outage, if Arbiter configured Two data centers IBF protects for both server and site power outage scenarios CTN with 2 servers (one in each data center) install IBF on at least the PTS/CTS CTN with 3 or more servers, install IBF on CTS and Arbiter (in same site as CTS)

STP configuration and time information saved across Power on Resets or power outages

Single server STP-only CTN Previously When an IML/POR is done for a single server STP-only CTN, user is required to: Initialize the Time Redefine that server as the stratum 1 after IML is complete System Management enhancement Customer selects the option to not allow any other servers to join the single server CTN when configuring the CTN STP CTN configuration (PTS/CTS) will be reinstated after IML/POR, Initialize Time not required Fulfilled customer requirement for single server CTN

NTP Server on Hardware Management Console

Using an NTP server as External Time Source (ETS) NTP client code on Support Element (SE) of z10 and z9 NTP server must be directly connected to the SE LAN SE LAN considered to be a private dedicated LAN Isolated from the corporate and public networks NTP server on HMC addresses security concerns NTP client on SE can now use the HMC as an NTP time server. Provides LAN isolation for SE NTP Client The HMC has two LAN ports, physically isolated: HMC/SE LAN used by STP’s NTP client code Second LAN used by HMC to access an NTP time server PPS cannot be obtained from ‘HMC as an NTP Server’

Internet NTP servers (NTP Project) Previous capability – Using Internet NTP Servers Internet NTP servers (NTP Project) (source for NTP server B) non-System z time synchronized servers Ethernet Switch Corporate network Internet HMC NTP server B (UNIX, Linux) Stratum 2 IBM firewall NTP server B defined on CTS/PTS CTN Remote HMC (Browser) z990 Arbiter z9 EC PTS/CTS z9 BC (BTS) IBM Support Center (RSF) NTP server B defined on BTS

Internet NTP servers (NTP Project) NTP Server on Hardware Management Console Internet NTP servers (NTP Project) (source for NTP server B) LAN isolated from HMC/SE LAN LAN non-System z time synchronized servers Ethernet Switch Corporate network Internet HMC NTP server B Stratum 2 IBM firewall NTP server B defined on CTS/PTS CTN Remote HMC (Browser) z990 Arbiter z9 EC PTS/CTS z9 BC (BTS) IBM Support Center (RSF) NTP server B defined on BTS

Redundant NTP server configuration using NTP server on HMC

API to automate STP CTN Reconfiguration

Automation to reconfigure STP-only CTN Targeted for GDPS and non-GDPS customers Fulfilled customer requirement to remove manual reconfiguration required without API. Provides automation to reconfigure STP-only CTN for the following: Unplanned outage when BTS takes over as CTS (e.g., Server failure or Power outage) BTS is single point of failure in case of second failure affecting BTS To maintain STP recovery capabilities, STP CTN reconfiguration required If original CTN has 2 servers, no further reconfiguration possible If original CTN has 3 servers, PTS, BTS reconfiguration possible If original CTN > 3 servers, PTS, BTS, Arbiter reconfiguration possible

Automation to reconfigure STP-only CTN (continued) Provides automation to reconfigure STP-only CTN for the following: Planned or Unplanned outage of BTS or Arbiter (e.g., Server failure or Power outage) PTS is single point of failure in case of second failure affecting PTS To maintain STP recovery capabilities, STP CTN reconfiguration required Planned action when disruptive action needs to be performed on CTS (e.g., POR, or activate/deactivate) CTS role needs to be moved to alternate server (most likely BTS) Capability to reconfigure CTN to maintain full recovery capability Capability to move CTS role back to original configuration after planned action completed

Summary Enhancements announced October 21, 2008 Enhancements generally available October 28, 2008 for z10 EC and z10 BC Enhancements generally available January 28, 2009 for z9 EC and z9 BC STP configuration and time information saved across Power-on-Resets or power outages on z990 and z890 generally available January 28, 2009 Additional Information: Server Time Protocol Planning Guide, SG24-7280 Server Time Protocol Implementation Guide, SG24-7281 STP Web Site: www.ibm.com/systems/z/pso/stp.html STP education module on Resource Link