When to Ship: A Practical Approach to Determining Application Deployment Readiness Peter Varhol Solutions Evangelist © 2011 Seapine Software, Inc. All rights reserved.

Agenda Why we don’t know when to ship How we can know when to ship Data, reports, and shipping The best laid plans Really knowing when to ship

About the Presenter Peter Varhol Evangelist at Seapine Software Former professor of computer science and mathematics Long-time speaker and writer on technology topics Founder of the Cutting Edge Computing blog – http://pvarhol.wordpress.com

When the schedule tells us to We All Know When to Ship When the schedule tells us to It’s not a bad metric, but it doesn’t take into account whether or not the application is really ready. That’s what we’re going to consider in this presentation.

Why We Don’t Know When to Ship We haven’t defined shipping criteria at the beginning We may not even agree on criteria We don’t have good measurements of completeness and quality We don’t have contingency plans when things go wrong

We All Know When to Ship Therefore, the schedule isn’t a good metric for shipping We don’t know if the software is ready In fact, What does “ready to ship” really mean? And if we don’t know what it is, we can’t measure it

Why We Don’t Know When to Ship The meaning of “shipping software” has changed In the past We did beta testing, candidate releases, manufacturing and packaging Today The process varies, but can be as short as a day Fix, test, rebuild Web application

Knowing When to Ship “The companies that know how to ship software are the ones to watch.” Mark Lucovsky Software Engineer, Google Former Distinguished Software Engineer, Micrososft

We All Know When to Ship Code completeness Developers and testers determine code completeness Quality Testers and users determine actual quality Business There are important business decisions in shipping

We All Know When to Ship We need quality standards And ultimately metrics to measure those standards We need agreement on the standards And confidence in our metrics We may have to trade off But we want a seat at that table

We All Know When to Ship Possible measurements All features seem to be working properly Failures aren’t replicable We think it’s ready There are a number of reasons that quality is a problem in many software development organizations.

We All Know When to Ship But there are more objective indicators, focusing on quality No P0 or P1 bugs New bug count is asymptotic All functional tests execute for 10 consecutive days

Requirements Play a Part All requirements are satisfied That may take a long time All requirements are not created equal Some are more important than others Requirements change

We All Know When to Ship How about the business? Balance of revenue and quality Shipping can start a new revenue stream But quality shortcomings could dent it

We All Know When to Ship Any definition is possible As long as it meets the needs of the team and the business And can be measured

We All Know When to Ship It depends on the application Safety-critical versus commercial versus internal A proposed quality definition Full feature coverage, no high-priority bugs A set code coverage percentage Find/fix defect ratio low and stabilized

We All Know When to Ship

Define What It Means to Be Ready Code complete All required features implemented or accounted for All error-handling code has been implemented Unit tests have been run

Define What It Means to Be Ready Business considerations We need the revenue this quarter We’ve promised our customers or investors We have a limited market window

Define What It Means to Be Ready Quality Base quality goals on projected operational use Quantify those goals and get agreement with stakeholders Determine how to test and measure to that level of quality

Implement a Solution A dashboard can work well Define quality measures Determine dashboard representation Enable drilldown Generate reports on status and trends

Quantifying Ready to Ship You need A platform for data collection The ability to analyze data and create ongoing reports The ability to do ad hoc queries The ability to share reports and data

Quantifying Ready to Ship Measuring feature coverage Organize test cases Track test execution Gather and analyze statistics

Quantifying Ready to Ship Code coverage percentage A developer’s metric Close to 100 percent is unrealistic

Measuring Ready to Ship

Measuring Ready to Ship

The Best Laid Plans of Mice and Men You had an agreement on quality and its measures Something goes wrong Development takes too long The project scope or direction changes What do testers do now?

The Best Laid Plans of Mice and Men Quality now has a seat at the table It’s quantified and agreed to ahead of time You may still have to compromise It still has to ship this quarter But you communicate the impact on quality

The Best Laid Plans The result could be An extended schedule If the schedule remains the same Fewer features in this release Follow-up with a service pack Explanations to customers/users

Really Knowing When to Ship A quantifiable definition is essential A number of possibilities, depending on the application Measurements are ongoing And must be analyzed in the context of the project Know what it means to ship Features, code, defects

Summary Define your criteria for shipping Determine information you need on that criteria Prepare reports and live charts Now you know when to ship

Questions?

Thank you © 2011 Seapine Software, Inc. All rights reserved.