Five Nines - To Dream the Impractical Dream? Presentation to the CSG Bruce Vincent

Agenda  What is driving high-availability?  How do we judge which services need HA?  How to achieve HA services?  What’s working and what’s not

What’s driving high-availability?  Frankly, [we] are…and should be.  Central IT services have gone from popular to essential  Interdependencies of services  The hassle of outages!  Choices of providers

Failure without loss of service

Failure with loss of service

How do we judge which services need high-availability?  If a service isn’t that important, why are we running it?  Turn it around…why doesn’t it need to be fault-tolerant and scalable?

How to achieve HA services?  Build in fault-tolerance and scalability by design  Monitoring and metrics  Learn from outages service  Manage Risk - Balance efforts  Service - Don’t focus on SLA legalese

Major Failure with No Outage Incident Summary: Active Directory Server rebooted after determining that it was in an impaired state, cause under investigation Incident Started: :59 Incident Stopped: :05 Systems Affected: Godzilla Server functions are replicated on other servers, no end- user outage Incident Detail: Server rebooted and has resumed operations Investigating logged error messages.

As Opposed To… Incident Summary: A failed memory board on the Production Oracle machine caused Production to be unavailable. The machine was re- booted to clear memory errors. Incident Started: :15 Incident Stopped: :50 Systems Affected: Oracle Applications Production Incident Detail: A failed memory board on the Production machine caused Production to be unavailable. The machine was re- booted to clear memory errors.

DOOH! At Tue Jan 4 10:31:30 PST 2005, the following responses were received: Incident Summary: Ofdbprd1 was down because of a failed CPU/memory board. Incident Started: :55 Incident Stopped: :10 Systems Affected: Ofdbprd1 (Oracle Financials transaction database server). Incident Detail: The failed CPU/memory board was removed from the configuration and the system was brought up on the remaining boards. The failed board has been replaced and will be returned to service on Jan 6.

Make Services Boring (sort-of)  Build it to mask failures  Control/communicate changes  Test failover regularly (A-A,A-P)  Keep service profiles for monitoring and resource trends up-to-date  Create enterprise system-wide view