Cleanroom Software Engineering Kris Read Paul Werbicki

Introduction Name borrowed from hardware cleanrooms Software that exhibits zero defect rate Uses mathematical and statistical models to prevent and detect defects 5/22/2019 K. Read & P. Werbicki

What is Cleanroom? Incremental development cycle Mathematically based design specifications Statistical verification and certification Software re-engineering Cleanroom is language-, environment-, and application-independent, and has been used to develop and evolve a variety of systems, including real-time, embedded, host, distributed, workstation, client-server, and microcode systems. 5/22/2019 K. Read & P. Werbicki

Why Cleanroom? Very high quality Higher productivity Reduced life cycle costs Higher return on investment Quality. Improvements of 10 to 20X and substantially more over baseline performance have been reported. Failures in field use have been greatly reduced over prior experience. For example, IBM developed an embedded, real-time, bus architecture, multiple-processor device controller product that exhibited no failures in three years use at over 300 customer locations Productivity. Significant improvements over baseline performance have been reported. For example, an Ericsson Telecom project to develop a 374 KLOC operating system reported gains of 1.7X in development productivity [Hausler 94], and an IBM project to develop a network management and outage avoidance product reported a 2X improvement in development productivity [Hausler 92]. The US Army Picatinny Arsenal STARS demonstration project reported a 4.6X productivity gain [Sherer 96]. Life cycle costs. Reductions in life cycle costs through decreases in testing, error correction, and maintenance have been reported. For example, IBM developed a COBOL structuring product that exhibited just seven minor errors in the first three years of field use, all simple fixes, with a corresponding drop in maintenance costs compared to baselines for similar products [Linger 88]. Return on investment. Reports have shown that Cleanroom adoption costs can be recovered on the first project through increased effectiveness in development and testing and reduced error correction and rework. For example, a 21 to 1 return-on-investment in Cleanroom technology introduction has been reported by the Picatinny Arsenal STARS Cleanroom project [Sherer 96]. 5/22/2019 K. Read & P. Werbicki

When is Cleanroom used? Suitable for critical applications Used when defects can cause loss of life defects can cause critical financial loss software cannot be updated at a later time The Cleanroom process has been demonstrated in software development projects in industry, as well as in NASA and the DoD STARS (Software Technology for Adaptable, Reliable Systems) program. Experience has shown substantial improvements over traditional results [Basili 94, Hausler 94, Linger 94]: 5/22/2019 K. Read & P. Werbicki

It’s all about no errors “Errorless Software Development” If no errors enter development, no errors need to be tested or debugged Elimination of testing and debugging means faster product development Introduce no errors in the first place 5/22/2019 K. Read & P. Werbicki

Figure. Cleanroom Process Flow (Linger & Trammell, 1996)

ANALYSIS & SPECIFICATION CUSTOMER INTERACTION ANALYSIS & SPECIFICATION INCREMENT PLANNING DEVELOPMENT QUALITY EVALUATION CUSTOMER INTERACTION ANALYSIS & SPECIFICATION INCREMENT PLANNING DEVELOPMENT QUALITY EVALUATION CUSTOMER INTERACTION ANALYSIS & SPECIFICATION INCREMENT PLANNING DEVELOPMENT QUALITY EVALUATION CUSTOMER INTERACTION ANALYSIS & SPECIFICATION INCREMENT PLANNING DEVELOPMENT QUALITY EVALUATION CUSTOMER INTERACTION ANALYSIS & SPECIFICATION INCREMENT PLANNING DEVELOPMENT QUALITY EVALUATION QUALITY EVALUATION STATISTICAL CERTIFICATION TESTING REENGINEERING DESIGN & VERIFICATION USAGE MODELING & TEST PLANNING ANALYSIS & SPECIFICATION REQUIREMENTS SPECIFICATION FUNCTION SPECIFICATION USAGE SPECIFICATION

Incremental Development Cycle Early and continual quality assessment Increased user feedback Facilitates process improvements Permits requirements changes Avoids integration risk 5/22/2019 K. Read & P. Werbicki

Mathematically Based Design Specifications All design and code are verified as mathematically correct Cleanroom achieves this through Increment Planning Box Structure Specification and Design 5/22/2019 K. Read & P. Werbicki

Running Example Phone Number entry component GUI based Allows users to enter phone numbers in various common formats 5/22/2019 K. Read & P. Werbicki

Increment Planning Based on Referential Transparency Pipeline of user-function software increments Early and continual quality assessment and user feedback Avoids risks inherent in late component integration Permits systematic management of requirement changes over the development cycle 5/22/2019 K. Read & P. Werbicki

Box Structure Specification and Design A way of specifying software designs Semi-formal program/component verification Three box structure stages Black box State box Clear box 5/22/2019 K. Read & P. Werbicki

Black Box Structure Program or component is modeled as a mathematical function Defines the system boundaries Defines all stimuli and responses 5/22/2019 K. Read & P. Werbicki

Figure. Example Black Box Diagram

State Box Structure Maps stimulus and response to old and new state transitions Defines state data and initial functions Defines transition functions Used to verify Black Box Structure 5/22/2019 K. Read & P. Werbicki

Figure. Example State Box Diagram

Clear Box Structure Used to design controls and operations Shows use of new and reused Black Boxes Used to verify State Box Structure 5/22/2019 K. Read & P. Werbicki

Figure. Example Clear Box Diagram

Figure. Refinement and Verification (Bohner, 2001)

Repeated Verification After each box stage the design is refined There is a review by all team members Refinement continues until there is code 5/22/2019 K. Read & P. Werbicki

Integrating Box Structure with Z Design and code are verified as mathematically correct Cleanroom uses a semi formal design specification already Focus is on correct design Code is not written, compiled or executed until final stage of testing 5/22/2019 K. Read & P. Werbicki

Mathematically Based Design Specifications Catch errors before they enter the code No unit testing Statistical quality testing forms the only real test of the software Software results can be certified 5/22/2019 K. Read & P. Werbicki

Statistical Quality Cleanroom uses statistical quality control Encompasses development and testing Focuses on detecting the statistically significant defects Cleanroom is language-, environment-, and application-independent, and has been used to develop and evolve a variety of systems, including real-time, embedded, host, distributed, workstation, client-server, and microcode systems. 5/22/2019 K. Read & P. Werbicki

Statistical Usage Testing Software has an infinite number of paths It is impossible to test all paths A large enough random sample can represent the entire system 5/22/2019 K. Read & P. Werbicki

Advantages of this Approach Mathematically sound measurement Conducted as scientific experiments Proven in hardware engineering Tests for fitness of use Highly efficient 5/22/2019 K. Read & P. Werbicki

The Testing Procedure Pre-test Correctness Verification Create a Usage Model Generate random test cases Execute statistical tests Execute additional tests if needed 5/22/2019 K. Read & P. Werbicki

Correctness Verification Function-theoretic static code analysis Verifies correctness of code compared to specifications Defined conditions code must meet Mental/verbal proofs performed during reviews Based on control structures 5/22/2019 K. Read & P. Werbicki

Correctness Example User Requirement: All characters shall be numeric digits for the phone number to be considered valid. Functional Specification: A method called checkNumeric will return true if the character is a digit, false otherwise. 5/22/2019 K. Read & P. Werbicki

Correctness Example function boolean checkNumeric (Char C) { boolean x; if ( ! C.isNumber() ) x = true; else x = false; return x; } 5/22/2019 K. Read & P. Werbicki

Correctness Example When C.isNumber() is true… Does x = true make the function true? When C.isNumber() is false… Does x = false make the function false? YES + YES = Passed Correctness 5/22/2019 K. Read & P. Werbicki

Correctness Pros and Cons Highly effective in practice Requires good specifications Can be done incrementally Code re-use may require re-engineering 5/22/2019 K. Read & P. Werbicki

Usage Models Define all possible scenarios of use Includes probabilities for each scenario Multiple models can be built Generate test cases automatically 5/22/2019 K. Read & P. Werbicki

Usage Specification Identify and classify users, scenarios, and environments Establish hierarchical usage breakdown and probability distribution for software Obtain agreement with the customer on the basis for software certification. 5/22/2019 K. Read & P. Werbicki

Usage Specification Format Not defined explicitly by Cleanroom Often represented as a Markov chain A directed graph Nodes are states of use Arcs are stimuli that cause transitions 5/22/2019 K. Read & P. Werbicki

Usage Model Example Assume only two user types: USA and Japan Assume everyone is running the program in an identical environment 5/22/2019 K. Read & P. Werbicki

USA EMPTY PART DONE STARTING STATE EMPTY PART EMPTY PART DONE 9% USA EMPTY PART DONE 30% STARTING STATE EMPTY PART 10% EMPTY PART DONE 15% FIRST INPUT EMPTY PART DONE SECOND INPUT . . . LAST INPUT

Test Execution Tests are randomly chosen Tests are generated from the models Controlled tests are executed Result validated against project goals 5/22/2019 K. Read & P. Werbicki

Conclusions Survey says: Theorists love Cleanroom A wonderful advance in software development Downright weird approach Maybe a little of both Theorists love Cleanroom Some parts are reasonable, some are not Similar resistance to formal languages 5/22/2019 K. Read & P. Werbicki

Conclusions Mathematically based software development is difficult to do Main-stream software isn’t written this way Specific to certain types of software Still fairly unproven as a software development process 5/22/2019 K. Read & P. Werbicki

References THANK YOU! 5/22/2019 K. Read & P. Werbicki Bohner, A. (2001). Software Engineering CS5704. http://www.nvc.cs.vt.edu/~bohner/cs5704/CS5704-Class14.pdf Carnige Mellon Software Engineering Institute (2000). Cleanroom Software Engineering – Suftware Technology Review. http://www.sei.cmu.edu/activities/str/descriptions/cleanroom_body.html Cleansoft – Cleanroom Software Engineering Inc. http://www.cleansoft.com/ Hendrix, S. (2000). CSCI 3308 – Spring 2001. http://www.cs.colorado.edu/~hendrixs/classes/lectures/lecture_18.pdf Linger, R. and Trammel, C. (1996). Cleanroom Software Engineering Reference Model Version 1.0. http://www.sei.cmu.edu/pub/documents/96.reports/pdf/tr022.96.pdf Linger, R. (1993). Cleanroom Software Engineering for Zero Defect Software. http://www.bsn.usf.edu/departments/isds/faculty/hevner/ism6124/Linger1.pdf Wolack, C. (2001). Taking The Art Out Of Software Development – An In-Depth Review of Cleanroom Software Engineering. http://www.scisstudyguides.addr.com/papers/cwdiss725paper1.htm Oshana, R, and Linger, R. (1999). Capability Maturity Model Software Development Using Cleanroom Software Engineering Principles - Results of an Industry Project http://www.computer.org/proceedings/hicss/0001/00017/00017042.PDF Pressmen and Associates (2000). Cleanroom Engineering Resources. http://www.rspa.com/spi/cleanroom.html Stavely, A. (2000). Integrating Z and Cleanroom. http://archive.larc.nasa.gov/shemesh/Lfm2000/Proc/cpp13.pdf THANK YOU! 5/22/2019 K. Read & P. Werbicki