1. Cleanroom Software Engineering
Kris Read Paul Werbicki

2. Introduction Name borrowed from hardware cleanrooms
Software that exhibits zero defect rate
Uses mathematical and statistical models to prevent and detect defects
5/22/2019
K. Read & P. Werbicki

3. What is Cleanroom? Incremental development cycle
Mathematically based design specifications
Statistical verification and certification
Software re-engineering
Cleanroom is language-, environment-, and application-independent, and has
been used to develop and evolve a variety of systems, including real-time,
embedded, host, distributed, workstation, client-server, and microcode systems.
5/22/2019
K. Read & P. Werbicki

4. Why Cleanroom? Very high quality Higher productivity
Reduced life cycle costs
Higher return on investment
Quality. Improvements of 10 to 20X and substantially more over baseline
performance have been reported. Failures in field use have been greatly
reduced over prior experience. For example, IBM developed an embedded,
real-time, bus architecture, multiple-processor device controller product
that exhibited no failures in three years use at over 300 customer locations
Productivity. Significant improvements over baseline performance have
been reported. For example, an Ericsson Telecom project to develop a 374
KLOC operating system reported gains of 1.7X in development
productivity [Hausler 94], and an IBM project to develop a network
management and outage avoidance product reported a 2X improvement
in development productivity [Hausler 92]. The US Army Picatinny
Arsenal STARS demonstration project reported a 4.6X productivity gain
[Sherer 96].
Life cycle costs. Reductions in life cycle costs through decreases in testing,
error correction, and maintenance have been reported. For example, IBM
developed a COBOL structuring product that exhibited just seven minor
errors in the first three years of field use, all simple fixes, with a
corresponding drop in maintenance costs compared to baselines for
similar products [Linger 88].
Return on investment. Reports have shown that Cleanroom adoption costs
can be recovered on the first project through increased effectiveness in
development and testing and reduced error correction and rework. For
example, a 21 to 1 return-on-investment in Cleanroom technology
introduction has been reported by the Picatinny Arsenal STARS
Cleanroom project [Sherer 96].
5/22/2019
K. Read & P. Werbicki

5. When is Cleanroom used? Suitable for critical applications Used when
defects can cause loss of life
defects can cause critical financial loss
software cannot be updated at a later time
The Cleanroom process has been demonstrated in software development
projects in industry, as well as in NASA and the DoD STARS (Software
Technology for Adaptable, Reliable Systems) program. Experience has shown
substantial improvements over traditional results [Basili 94, Hausler 94, Linger
94]:
5/22/2019
K. Read & P. Werbicki

6. It’s all about no errors
“Errorless Software Development”
If no errors enter development, no errors need to be tested or debugged
Elimination of testing and debugging means faster product development
Introduce no errors in the first place
5/22/2019
K. Read & P. Werbicki

7. Figure. Cleanroom Process Flow (Linger & Trammell, 1996)

8. ANALYSIS & SPECIFICATION
CUSTOMER INTERACTION
ANALYSIS & SPECIFICATION
INCREMENT PLANNING
DEVELOPMENT
QUALITY EVALUATION
CUSTOMER INTERACTION
ANALYSIS & SPECIFICATION
INCREMENT PLANNING
DEVELOPMENT
QUALITY EVALUATION
CUSTOMER INTERACTION
ANALYSIS & SPECIFICATION
INCREMENT PLANNING
DEVELOPMENT
QUALITY EVALUATION
CUSTOMER INTERACTION
ANALYSIS & SPECIFICATION
INCREMENT PLANNING
DEVELOPMENT
QUALITY EVALUATION
CUSTOMER INTERACTION
ANALYSIS & SPECIFICATION
INCREMENT PLANNING
DEVELOPMENT
QUALITY EVALUATION
QUALITY EVALUATION
STATISTICAL CERTIFICATION TESTING
REENGINEERING DESIGN & VERIFICATION
USAGE MODELING & TEST PLANNING
ANALYSIS & SPECIFICATION
REQUIREMENTS SPECIFICATION
FUNCTION SPECIFICATION
USAGE SPECIFICATION

9. Incremental Development Cycle
Early and continual quality assessment
Increased user feedback
Facilitates process improvements
Permits requirements changes
Avoids integration risk
5/22/2019
K. Read & P. Werbicki

10. Mathematically Based Design Specifications
All design and code are verified as mathematically correct
Cleanroom achieves this through
Increment Planning
Box Structure Specification and Design
5/22/2019
K. Read & P. Werbicki

11. Running Example Phone Number entry component GUI based
Allows users to enter phone numbers in various common formats
5/22/2019
K. Read & P. Werbicki

12. Increment Planning Based on Referential Transparency
Pipeline of user-function software increments
Early and continual quality assessment and user feedback
Avoids risks inherent in late component integration
Permits systematic management of requirement changes over the development cycle
5/22/2019
K. Read & P. Werbicki

13. Box Structure Specification and Design
A way of specifying software designs
Semi-formal program/component verification
Three box structure stages
Black box
State box
Clear box
5/22/2019
K. Read & P. Werbicki

14. Black Box Structure
Program or component is modeled as a mathematical function
Defines the system boundaries
Defines all stimuli and responses
5/22/2019
K. Read & P. Werbicki

15. Figure. Example Black Box Diagram

16. State Box Structure
Maps stimulus and response to old and new state transitions
Defines state data and initial functions
Defines transition functions
Used to verify Black Box Structure
5/22/2019
K. Read & P. Werbicki

17. Figure. Example State Box Diagram

18. Clear Box Structure Used to design controls and operations
Shows use of new and reused Black Boxes
Used to verify State Box Structure
5/22/2019
K. Read & P. Werbicki

19. Figure. Example Clear Box Diagram

20. Figure. Refinement and Verification (Bohner, 2001)

21. Repeated Verification
After each box stage the design is refined
There is a review by all team members
Refinement continues until there is code
5/22/2019
K. Read & P. Werbicki

22. Integrating Box Structure with Z
Design and code are verified as mathematically correct
Cleanroom uses a semi formal design specification already
Focus is on correct design
Code is not written, compiled or executed until final stage of testing
5/22/2019
K. Read & P. Werbicki

23. Mathematically Based Design Specifications
Catch errors before they enter the code
No unit testing
Statistical quality testing forms the only real test of the software
Software results can be certified
5/22/2019
K. Read & P. Werbicki

24. Statistical Quality Cleanroom uses statistical quality control
Encompasses development and testing
Focuses on detecting the statistically significant defects
Cleanroom is language-, environment-, and application-independent, and has
been used to develop and evolve a variety of systems, including real-time,
embedded, host, distributed, workstation, client-server, and microcode systems.
5/22/2019
K. Read & P. Werbicki

25. Statistical Usage Testing
Software has an infinite number of paths
It is impossible to test all paths
A large enough random sample can represent the entire system
5/22/2019
K. Read & P. Werbicki

26. Advantages of this Approach
Mathematically sound measurement
Conducted as scientific experiments
Proven in hardware engineering
Tests for fitness of use
Highly efficient
5/22/2019
K. Read & P. Werbicki

27. The Testing Procedure Pre-test Correctness Verification
Create a Usage Model
Generate random test cases
Execute statistical tests
Execute additional tests if needed
5/22/2019
K. Read & P. Werbicki

28. Correctness Verification
Function-theoretic static code analysis
Verifies correctness of code compared to specifications
Defined conditions code must meet
Mental/verbal proofs performed during reviews
Based on control structures
5/22/2019
K. Read & P. Werbicki

29. Correctness Example
User Requirement: All characters shall be numeric digits for the phone number to be considered valid.
Functional Specification: A method called checkNumeric will return true if the character is a digit, false otherwise.
5/22/2019
K. Read & P. Werbicki

30. Correctness Example function boolean checkNumeric (Char C) {
boolean x;
if ( ! C.isNumber() ) x = true;
else x = false;
return x; }
5/22/2019
K. Read & P. Werbicki

31. Correctness Example When C.isNumber() is true…
Does x = true make the function true?
When C.isNumber() is false…
Does x = false make the function false?
YES + YES = Passed Correctness
5/22/2019
K. Read & P. Werbicki

32. Correctness Pros and Cons
Highly effective in practice
Requires good specifications
Can be done incrementally
Code re-use may require re-engineering
5/22/2019
K. Read & P. Werbicki

33. Usage Models Define all possible scenarios of use
Includes probabilities for each scenario
Multiple models can be built
Generate test cases automatically
5/22/2019
K. Read & P. Werbicki

34. Usage Specification
Identify and classify users, scenarios, and environments
Establish hierarchical usage breakdown and probability distribution for software
Obtain agreement with the customer on the basis for software certification.
5/22/2019
K. Read & P. Werbicki

35. Usage Specification Format
Not defined explicitly by Cleanroom
Often represented as a Markov chain
A directed graph
Nodes are states of use
Arcs are stimuli that cause transitions
5/22/2019
K. Read & P. Werbicki

36. Usage Model Example Assume only two user types: USA and Japan
Assume everyone is running the program in an identical environment
5/22/2019
K. Read & P. Werbicki

37. USA EMPTY PART DONE STARTING STATE EMPTY PART EMPTY PART DONE9%
USA
EMPTY
PART
DONE
30%
STARTING STATE
EMPTY
PART
10%
EMPTY
PART
DONE
15%
FIRST INPUT
EMPTY
PART
DONE
SECOND INPUT
. . .
LAST INPUT

38. Test Execution Tests are randomly chosen
Tests are generated from the models
Controlled tests are executed
Result validated against project goals
5/22/2019
K. Read & P. Werbicki

39. Conclusions Survey says: Theorists love Cleanroom
A wonderful advance in software development
Downright weird approach
Maybe a little of both
Theorists love Cleanroom
Some parts are reasonable, some are not
Similar resistance to formal languages
5/22/2019
K. Read & P. Werbicki

40. Conclusions
Mathematically based software development is difficult to do
Main-stream software isn’t written this way
Specific to certain types of software
Still fairly unproven as a software development process
5/22/2019
K. Read & P. Werbicki

41. References THANK YOU! 5/22/2019 K. Read & P. Werbicki
Bohner, A. (2001). Software Engineering CS
Carnige Mellon Software Engineering Institute (2000). Cleanroom Software Engineering – Suftware Technology Review.
Cleansoft – Cleanroom Software Engineering Inc.
Hendrix, S. (2000). CSCI 3308 – Spring
Linger, R. and Trammel, C. (1996). Cleanroom Software Engineering Reference Model Version
Linger, R. (1993). Cleanroom Software Engineering for Zero Defect Software.
Wolack, C. (2001). Taking The Art Out Of Software Development – An In-Depth Review of Cleanroom Software Engineering.
Oshana, R, and Linger, R. (1999). Capability Maturity Model Software Development Using Cleanroom Software Engineering Principles - Results of an Industry Project
Pressmen and Associates (2000). Cleanroom Engineering Resources.
Stavely, A. (2000). Integrating Z and Cleanroom.
THANK YOU!
5/22/2019
K. Read & P. Werbicki