1 CSSE 477 – More on Availability & Reliability Steve Chenoweth Thursday, 9/22/11 Week 3, Day 3 Right – High availability with VMWare – the major goal is to eliminate all single points of failure. When the an ESX host goes down (the red X), the virtual machines running on the failed host migrate over to the other, healthy ESX hosts that have spare capacity. After migration, the virtual machines are restarted automatically, without intervention from the administrator. From hosting/esx-high-availability.html. hosting/esx-high-availability.html

2 Today 5 Minute talks on availability project Tactics for software availability engineering… –A bit more, mostly from Musa’s book Tonight: –Project 2, final part –HW 3 (individual) John Musa ( ), the inventor of “software reliability engineering.”

3 Software failures Important to discuss with customers –Need to know what variations in system behavior are tolerable Requirements provide a positive specification –Defining failures gives you a “negative specification” What the system must not do –Adds another dimension to communication with the user Below – A windmill goes south.

4 How to classify severity of software failures Based on cost: (these are 1999 $ from Musa’s book, so you need to set your own scale) Severity classDefinition ($) 1> 100, ,000 – 100, – 10,000 4< 1000

5 How to classify severity of software failures, cntd Based on operational impact: Severity class Definition 1Unavailability to users of one or more key operations 2Unavailability to users of one or more important operations 3Unavailability to users of one or more operations but workarounds available 4Minor deficiencies in one or more operations

6 How to classify severity of software failures, cntd Need to make such a table product-specific. Suppose the product is “Fone Follower”: Failure severity class Failure definition 1 Failure that prevents calls from being forwarded 2 Failure that prevents entry of phone numbers to which calls will be forwarded 3 Failure that makes system administration more difficult, but possible through alternate means. Like GUI doesn’t work for some feature, but can use text I/F. 4 Failure that causes minor inconvenience – like screens don’t show current date.

7 Need to set “failure intensity objectives” for each release Need to define a global way of measuring the “intensity” – like failures per hour of operation that will be tolerated. Next goal is to convert these to global measures related to code, like failures per million lines of code run, for the critical subsystems. Heuristic: For a given release, the product of these factors tends to be roughly constant, related to the amount of new functionality added: 1.Failure intensity 2.Development time 3.Development cost

8 Failure intensity vs reliability Generally, = - ln R t Where R = reliability, = failure intensity, and t = number of natural time units. E.g., if reliability is for 8 hours, the failure intensity is one failure per 1000 hours. Note that Musa’s definition of “reliability” is slightly more sophisticated than usual: It’s the probability of execution without failure for a specified time interval (like hours). In contrast, usually we talk of the average time to failure, and so skip over the probability part (it’s 50%).

9 For a new system… We don’t already have a track record of failure intensities –So it’s tougher to judge how much time to spend trying to get it right, for the “next release” – that’s release 1.0 ! What to do? –Find operational data for similar systems Or, how reliable are the underlying systems you’ll use? –Consider vendor warranties – what will we promise? (Or what do others promise?) –Get experts to estimate, based on prior work

10 Availability Tactics Once you know what to prevent, you develop to counteract those problems: Try the 3 Strategies from Bass Ch 5: –Fault detection –Fault recovery –Fault prevention Musa’s variation on this list is: –Fault prevention –Fault removal –Fault tolerance Opening rounds Later

11 Musa’s development strategies Engineer the right balance among these reliability strategies –Determine where to focus them, to maximize the likelihood of meeting the objectives in an economical way –Components you buy and integrate are a problem – you have less control over those! The best you can do may be to test with them as thoroughly as possible, give feedback to their vendors

12 Musa’s strategies, cntd Testers should be in on setting the objectives and deciding the system architecture Fault prevention is done by: –Good development processes, especially: Having sound underlying methodologies Doing reviews – like to requirements & design Enforcing standards Using design tools that keep faults from being introduced

13 Musa’s strategies, cntd How’s fault removal done? –Primarily by code reviews and testing You can measure the effectiveness of the code reviews by how many were caught vs how many remained to be caught in testing Measure the effectiveness of testing by the number of faults found by that testing, vs those found after (like by the customer!)

14 Musa’s strategies, cntd How to achieve fault-tolerance? –Needs design: Anticipate what deviations are likely to occur and will lead to failures Implement “robust” software to counteract them –Like handling unexpected input from users or from other systems –How to minimize performance degradation, data corruption, or undesirable outputs Use hardware to help (see slide 1!) –Can measure effectiveness by the reduction in failure intensity that results

15 Musa’s strategies, cntd In large organizations: –Important to measure the effectiveness of each of these availability tactics –Leads to sound decisions about what to spend money on – “best strategies”

16 Software Safety A topic related to reliability Means freedom from mishaps –Mishap = loss of human life, injury, or property damage –Most software failures just cause user dissatisfaction Software safety is dependent on realistic testing –Need operational profiles to focus testing

17 Software vs hardware Hardware reliability is affected by aging and wear. Software – not so much. –Execution time affects reliability What is its “duty cycle” (in hours)? And What are its failures per execution hour? –To get ultrareliable functions, we need a test duration of several times the reciprocal of the failure intensity objective E.g., to get a failure intensity of failures per hour, you’d need to test for several times 10 9 hours.

18 Musa on fault detection A less precise idea than software failure Absolute concept – an entity you can define without reference to software failures –Like a bad disk read Operational concept – an entity that exists only with reference to the fact that failures will occur if that state is executed –The fault is the defect causing the failure

19 Absolute faults Makes it necessary to postulate a “perfect” program to which an actual program can be compared. Then a fault is incorrect code, in comparison –It’s defective, missing, or an extra instruction –The defective program can be compared with the perfect program. Of course, in reality, there could be many “perfect” programs. –You don’t know about the bad code till you are trying to fix it.

20 Operational faults The “fault” has some reality of its own, as an implementation defect. But really, a piece of bad code could be responsible for no failures, or lots of failures, etc. And “software failures” may not be in the code, per se. So this definition of “fault” has issues, too.

21 A realistic definition Musa ends up saying, like Bass, that a software fault is incorrect code that leads to actual or potential failures. Which leaves open the question of “What code should be included in the fault?” –Requires judgment –Would changing a few related instructions prevent additional failures from occurring?

22 Availability in summary Strategic thing is to work to improve it –Definitely don’t ignore it on any real product –“Hope is a city on de Nile”. Lots more to explore –Plenty of systems out there, where people will pay to have them improve on this –Like all the quality attributes, this could be a career activity A good additional course in this area – Dr. Radu’s ECE 497 – Design of fault tolerant digital systems Right - A software cruise down “de Nile”