Unit Testing Workshop Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com November 23, 2018 © Robert Sabourin, 2005

Section 1 Overview and Some Philosophy Unit Testing Workshop Section 1 Overview and Some Philosophy November 23, 2018 © Robert Sabourin, 2005

Unit Testing Concepts Overview: Objective Introductions and definitions Some Philosophy Testing in Development Process Lifecycle Models Unit Testing Process Unit Testing Ideas Unit Testing Cost? Benefit? Testing Approaches Test Cases Unit Test Code Coverage Organizing Your Unit Testing Paradoxes, Problems and Politics Final Words Test Harness Example Unit Test Harness November 23, 2018 © Robert Sabourin, 2005

Objective “The main objective is to make sure that we leave the delegates with a good understanding of what Unit testing is and what they need to do to implement it. They can then build on this basic knowledge.” November 23, 2018 © Robert Sabourin, 2005

Introduction Robert Sabourin , Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada rsabourin@amibug.com November 23, 2018 © Robert Sabourin, 2005

Fundamental Question Some Philosophy How do you know when you are finished? November 23, 2018 © Robert Sabourin, 2005

Crosby on Quality Some Philosophy “Quality is defined as conformance to requirements” “Quality is not a measure of GOODNESS” Phil B. Crosby, Quality is Free November 23, 2018 © Robert Sabourin, 2005

Edsger W. Dijkstra Some Philosophy “Program testing can be used to show the presence of bugs, but never to show their absence” November 23, 2018 © Robert Sabourin, 2005

Boris Beizer Some Philosophy “Why software has bugs – the fundamental problem – Programming is a bitch.” The Frozen Keyboard - 1986 November 23, 2018 © Robert Sabourin, 2005

Watts S. Humphrey Some Philosophy “… the job of the software engineer is to deliver high-quality software products at agreed cost and schedule …” “… even the most experienced software engineer injects about one defect for ten lines of code …” November 23, 2018 © Robert Sabourin, 2005

C. Northcote Parkinson Some Philosophy Parkinson’s Law: “…work expands so as to fill the time available for its completion…” November 23, 2018 © Robert Sabourin, 2005

Yoda Some Philosophy "No! Try not, Do. Or do not. There is no try." November 23, 2018 © Robert Sabourin, 2005

“…begin with the end in mind … Some Philosophy Steve Covey “…begin with the end in mind … “…first things first …" November 23, 2018 © Robert Sabourin, 2005

Unit Testing Workshop Section 2 About Bugs November 23, 2018 © Robert Sabourin, 2005

About Bugs Bug Defined To make our job more fun, whenever we have a concern with software, we call it a “bug”. November 23, 2018 © Robert Sabourin, 2005

A Bug Vocabulary About Bugs Fault, Error mistake in implementation or design Failure, Conditions misbehavior of software due to a fault Symptom something observed due to a failure Bug anything wrong with the software Quality value to someone Defect legal term. Something is definitely wrong. Anomaly, Problem, Issue, Concern, Snag other terms to use for a bug November 23, 2018 © Robert Sabourin, 2005

Software Testing About Bugs It’s all about people! (and the occasional bug too) November 23, 2018 © Robert Sabourin, 2005

Purpose of Unit Testing Common definition: To find bugs way before our customers do! Broader definitions: Provide objective information about the status of the development task being implemented. To assess whether task at hand is finished and the work product can be used by others To confirm a developers work is ready to be integrated into a system November 23, 2018 © Robert Sabourin, 2005

Section 3 Unit Testing in Development Unit Testing Workshop Section 3 Unit Testing in Development November 23, 2018 © Robert Sabourin, 2005

Purpose of Unit Testing Unit Testing in Development Purpose of Unit Testing So what exactly did they throw over the wall? November 23, 2018 © Robert Sabourin, 2005

Testing in Development Process Unit Testing in Development Testing in Development Process Testing activities take place in all parts of software development From requirement eliciting to final shipment Testing is part of the development process Testing is part of the company business process November 23, 2018 © Robert Sabourin, 2005

Testing can take place as part of each phase of development . Rational Unified Process (RUP) Transition Construction Inception Elaboration Core Workflow Maintenance Testing Development Design Analysis Requirements Testing can take place as part of each phase of development . November 23, 2018 © Robert Sabourin, 2005

Phase Rational Unified Process (RUP) Transition Construction Inception Elaboration Core Workflow Maintenance Testing Development Design Analysis Requirements Testing can take place as part of each core workflow involved in development organization. November 23, 2018 © Robert Sabourin, 2005

Section 4 Unit Testing Process Unit Testing Workshop Section 4 Unit Testing Process November 23, 2018 © Robert Sabourin, 2005

A Traditional Approach Unit Testing Process A Traditional Approach Design & implement Code of assigned task Design tests, define Platforms & test cases Code Run test cases, identify defects, regress Plan Test Unit Test Find root cause Trace & Debug critical defects Debug Implement corrections Rework Follow Up Document unit testing Repeatable, Reviewable November 23, 2018 © Robert Sabourin, 2005

Extreme Programming Begin with the end in mind Unit Testing Process Define Test Implement Test Then “Develop the simplest code to pass the test” November 23, 2018 © Robert Sabourin, 2005

Extreme Programming Unit Testing Process “There is a rhythm to developing software unit test first. You create one test to define some small aspect of the problem at hand. Then you create the simplest code that will make that test pass. Then you create a second test. Now you add to the code you just created to make this new test pass, but no more! Not until you have yet a third test. You continue until there is nothing left to test.” www.extremeprogramming.org November 23, 2018 © Robert Sabourin, 2005

An Agile Approach Plan Tests Build Test Code Unit Test Rework Unit Testing Process An Agile Approach Organize dev work into Testable chunks Build test, code, data, stubs, drivers, harness Plan Tests Create simplest code to pass tests Build Test Code Run tests Unit Test Correct code or test Rework Follow Up Document unit testing Repeatable, Reviewable November 23, 2018 © Robert Sabourin, 2005

Getting Started Practical Tip Unit Testing Process Getting Started Practical Tip Understand testing objectives Explicitly list unit tests ideas Review list with peers Prioritize list Develop and test code until all unit test cases pass or known bugs can be deferred November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Collect all testing ideas you can find! Unit Testing Process Unit Testing Ideas Collect all testing ideas you can find! List Sort Organize Shuffle November 23, 2018 © Robert Sabourin, 2005

Testing Ideas How to find them? Unit Testing Process Does system do what it is suppose to do? Does the system do things it is not supposed to? How can the system break? How does the system react to it’s environment? What characteristics must the system conform to? Why have previous or similar projects failed? How have previous or similar systems failed? November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Collect testing ideas Unit Testing Process Unit Testing Ideas Collect testing ideas From testing ideas build a series of testing objectives (TO) Each can include all, part of, or multiple testing ideas November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Creative approaches Unit Testing Process Action verbs Mind Maps Soap Operas Lateral Thinking November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Investigative approaches Unit Testing Process We become truffle snorting pigs and try to find useful information in all evidence we discover We can even get good ideas from out of date sources or unreliable November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Bug taxonomies Unit Testing Process Collections of possible bugs Appendix A of Testing Computer Software, Kaner, Falk, Nguyen Boris Biezer Taxonomy Otto Vinter manages Shopping cart taxonomy Giri Vijayaraghavan November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Requirements Unit Testing Process Use cases Functional requirements Quality factors Constraints Written requirements Implicit requirements November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Usage Scenarios Unit Testing Process Identify classes of users Identify how users will use system Describe scenarios Use Story board or similar approaches Identify variations November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Functionality Analysis Unit Testing Process Requirements, Design or Prototypes can give insights into Domain Analysis Equivalence classes Boundary analysis CRUD November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Failure Modes Unit Testing Process What can break? Reaction to invalid input? How does software behave in constrained environment? Memory Disk Space Network Bandwidth CPU capacity Shared resources Stress, Load, Volume November 23, 2018 © Robert Sabourin, 2005

Unit Test Harness Unit Testing Process November 23, 2018 © Robert Sabourin, 2005

Quality Factors Importance For Different Web Application Types Unit Testing Process Quality Factors Importance For Different Web Application Types November 23, 2018 © Robert Sabourin, 2005

Brainstorming Unit Testing Ideas Unit Testing Process Brainstorming Unit Testing Ideas Peer review to identify and clarify testing ideas which are New Missing Redundant Incorrect Incomplete Inconsistent Vague November 23, 2018 © Robert Sabourin, 2005

Unit Testing Ideas Oracle Collection Unit Testing Process Strategies to assess correctness Similar systems Old systems Subject matter experts Gurus Standards November 23, 2018 © Robert Sabourin, 2005

Unit Testing Tools Tools Unit Testing Process Dynamic analysis Static analysis Test case generation Test engines Home brew with correct skills Danny Faught Open Testware ideas November 23, 2018 © Robert Sabourin, 2005

Section 5 Unit Testing Cost Benefit Unit Testing Workshop Section 5 Unit Testing Cost Benefit November 23, 2018 © Robert Sabourin, 2005

Unit Testing Cost and Benefit Unit Testing Effort Types of work in any development task assignment Analysis Design Implementation Unit Testing Rework Documentation related to assigned work Configuration Management Administration November 23, 2018 © Robert Sabourin, 2005

Unit Testing Cost and Benefit Unit Testing Effort Individual development tasks are often 5 to 10 person days effort Shorter projects leads to shorter tasks Involves many objects, modules, configurations are touched to implement requirement Multidisciplinary approach - polyvalent Multiple technical skills - wiz kid November 23, 2018 © Robert Sabourin, 2005

Example Effort Breakdown Unit Testing Cost and Benefit Example Effort Breakdown Task Breakdown Description Days Percent Analysis Understand problem 0.5 5% Design Map out solution 2.0 20% Implementation Implement solution 3.0 30% Unit Testing Confirm solution solves problem Rework Correct defects and other errors or omissions 1.0 10% Documentation Update project technical documentation, describe work done November 23, 2018 © Robert Sabourin, 2005

Example Effort Breakdown - SQUEEZED Unit Testing Cost and Benefit Example Effort Breakdown - SQUEEZED Task Breakdown Description Days Percent Analysis Understand problem 0.5 8% Design Map out solution 1.0 15% Implementation Implement solution 3.0 46% Unit Testing Confirm solution solves problem Rework Correct defects and other errors or omissions Documentation Update project technical documentation, describe work done November 23, 2018 © Robert Sabourin, 2005

Unit Testing Cost and Benefit Economics Cost of fixing a defect sooner (closer to when defect was injected) is significantly less than fixing the defect later Generally Find and fix problems as early as practically possible November 23, 2018 © Robert Sabourin, 2005

Relative Cost to Repair Rational Unit Testing Cost and Benefit Relative Cost to Repair Rational November 23, 2018 © Robert Sabourin, 2005

Unit Testing Cost and Benefit Example Rework Effort November 23, 2018 © Robert Sabourin, 2005

Unit Testing Workshop Section 6 Coverage November 23, 2018 © Robert Sabourin, 2005

Coverage Test Coverage Coverage is some measure of how much of the code under test is being exercised when tests are executed. Are we trying each function? Are we trying each line of code? Are we not executing some code when we test the application? Should we? November 23, 2018 © Robert Sabourin, 2005

Coverage Line Coverage Branch Coverage Condition Coverage Coverage tools November 23, 2018 © Robert Sabourin, 2005

Line Coverage Each line of code must be executed once during testing. Weakness is that even if each line is executed during testing there are many possible states for each variable and condition November 23, 2018 © Robert Sabourin, 2005

Branch Coverage All lines of code are tested All branches in code are covered November 23, 2018 © Robert Sabourin, 2005

Basis Path Coverage All conditions of all branches are covered Conditions to get branch to go one way or another November 23, 2018 © Robert Sabourin, 2005

Coverage Example Sample code 1 If (condition A) 2 { code A } 3 If (condition B) 4 { code B } 5 If (condition C) 6 { code C } Full line or branch coverage if A, B, C true Full condition coverage if A, B, C true and false Full path coverage if all combinations of A, B, C true or false November 23, 2018 © Robert Sabourin, 2005

Coverage Exercise Code sample Define test cases which will give full line coverage November 23, 2018 © Robert Sabourin, 2005

Coverage Exercise Coverage / November 23, 2018 © Robert Sabourin, 2005 begin // // check range - value must be between 1 and 99 if ( ( SideA < 1 ) or ( SideA > 99 ) ) then Msg := 'A out of Range' ; end else if ( ( SideB < 1 ) or ( SideB > 99 ) ) then Msg := 'B out of Range' ; if ( ( SideC < 1 ) or ( SideC > 99 ) ) then Msg := 'C out of Range' ; end ; / November 23, 2018 © Robert Sabourin, 2005

Coverage Exercise Coverage / November 23, 2018 © Robert Sabourin, 2005 begin // // check range - value must be between 1 and 99 if ( ( SideA < 1 ) or ( SideA > 99 ) ) then Msg := 'A out of Range' ; end else if ( ( SideB < 1 ) or ( SideB > 99 ) ) then Msg := 'B out of Range' ; if ( ( SideC < 1 ) or ( SideC > 99 ) ) then Msg := 'C out of Range' ; end ; / A = -100, B=10, C=10 A=10,B=-100,C=10 A=10,B=10,C=-100 November 23, 2018 © Robert Sabourin, 2005

Coverage Tools Execution coverage monitors Coverage often require a special build of the program to link in special coverage analysis software or hooks great for from time to time seeing how much of code is exercised by testing activities generally to heavy to use for each test cycle November 23, 2018 © Robert Sabourin, 2005

Coverage Tools Execution coverage monitors because it often forces a special build the code being tested is not the same binary as a commercial cut watch out for performance, load and system resource limit side effects tool is useful to measure performance of test plan - not performance of application! November 23, 2018 © Robert Sabourin, 2005

Coverage November 23, 2018 © Robert Sabourin, 2005

Coverage November 23, 2018 © Robert Sabourin, 2005

Coverage November 23, 2018 © Robert Sabourin, 2005

Coverage November 23, 2018 © Robert Sabourin, 2005

Coverage November 23, 2018 © Robert Sabourin, 2005

Useful coverage information Has every function been used at least once during the test pass? What % of lines of code of each module are executed during testing? If there is a lot of code not being run - do we really need it? November 23, 2018 © Robert Sabourin, 2005

Coverage About Coverage Tools Effort to get coverage analysis running is significant and should be done from day 1 Today a lot of code is put together by cut and paste reuse strategy not all code is needed it is too risky to remove unneeded code November 23, 2018 © Robert Sabourin, 2005

Unit Testing Workshop Section 7 Organization November 23, 2018 © Robert Sabourin, 2005

Unit Testing Organization Top Down Testing White box method starting with main program and working down through the software. Stubs must be created for units not yet completed. November 23, 2018 © Robert Sabourin, 2005

Unit Testing Organization Bottom Up Testing White box method starting with lower level units. Driver units must be created for units not yet completed, each time a new higher level unit is added to those already tested. November 23, 2018 © Robert Sabourin, 2005

Unit Testing Organization Sandwich Testing Combining Bottom Up and Top Down approaches. Blend of stubs and drivers depending on part of software being exercised. November 23, 2018 © Robert Sabourin, 2005

Platform / Configuration Testing Unit Testing Organization Platform / Configuration Testing Ensure functionality operates as required on different hardware and software configurations Different versions of operating systems Different locales Different versions of www browsers, plug-ins Different versions of any co-dependent software November 23, 2018 © Robert Sabourin, 2005

Unit Testing Organization Assertion Testing “Asserts” Added as in-line code Ensure that software is operating in the correct state with correct values of certain variables, tables, parameters or other data Assertions often implemented as MACROS Generally disabled in commercial builds November 23, 2018 © Robert Sabourin, 2005

Unit Testing Organization Assertion Testing Cautionary Warning About Assertions excellent practice for developers to ensure code works can cause false failures if not implemented carefully (don’t care condition handling) ensure all developers use “Asserts” consistently! November 23, 2018 © Robert Sabourin, 2005

Section 8 Paradoxes Problems and Politics Unit Testing Workshop Section 8 Paradoxes Problems and Politics November 23, 2018 © Robert Sabourin, 2005

Do we still need independent testing? Paradoxes, Problems and Politics Do we still need independent testing? Number of defects introduced at development and early phases and discovered at system testing will drop! We will focus system testing staff on issues related to how the application operates as a system and how it interacts with different software, systems and in different environments. November 23, 2018 © Robert Sabourin, 2005

Do we need a Unit Testing Bug Database? Paradoxes, Problems and Politics Do we need a Unit Testing Bug Database? No – unit testing bug databases are optional and often discouraged – if used they should be independent of system test bug list. Bugs known, but not fixed, (on purpose) at unit test stage should be entered in the system test database November 23, 2018 © Robert Sabourin, 2005

What about using automated test case generators? Paradoxes, Problems and Politics What about using automated test case generators? These tools are risky since they create their test cases exclusively based on the code as written. If code is wrong – test case will confirm code is wrong! Used with caution they can help identify important problems and enforce disciplined development habits! November 23, 2018 © Robert Sabourin, 2005

Should we develop our own test harness? Paradoxes, Problems and Politics Should we develop our own test harness? When developing API based systems it is advisable to develop a test harness which can be shared with system testing team Any code written to run unit tests should be maintained in source control systems. Third Party test harnesses, such as J-Unit, are available free of charge and may provide a simple example environment It is a software project and needs requirements, configuration management, testing and a full lifecycle. November 23, 2018 © Robert Sabourin, 2005

How do we know that Unit Testing is effective? Paradoxes, Problems and Politics How do we know that Unit Testing is effective? Measures of interest Amount of effort correcting bugs found during system test Should reduce by a margin larger than added development cost of doing unit testing November 23, 2018 © Robert Sabourin, 2005

Should we try increasing unit testing on all projects? Paradoxes, Problems and Politics Should we try increasing unit testing on all projects? Probably a good idea to instill the concept by introducing it to one pilot team Try to keep all teams within one step of process maturity Build on success Learn from mistakes Every organization has a different temperament November 23, 2018 © Robert Sabourin, 2005

Paradoxes, Problems and Politics Does Unit testing impact the relationship between developers and testers? Yes indeed! Lots of evidence. Better quality leads to better focus and communications. Focus of system testing is on bugs which could not reasonably or economically have been discovered during development! November 23, 2018 © Robert Sabourin, 2005

How do developers react to unit testing? Paradoxes, Problems and Politics How do developers react to unit testing? Software development professionals want to deliver quality code that works and meets and important business need. As long as unit testing effort is included in development estimation and commitments. … begin with the end in mind … November 23, 2018 © Robert Sabourin, 2005

Finished? How do you know you are finished? November 23, 2018 © Robert Sabourin, 2005

You know you are finished when … … all high priority unit test ideas, unit tests case have passed ... November 23, 2018 © Robert Sabourin, 2005

You know you are finished when … … and the only bugs left are the ones that you can live with … At least for now! November 23, 2018 © Robert Sabourin, 2005

Thank You Questions? November 23, 2018 © Robert Sabourin, 2005

Appendix A Example Test Harness Unit Testing Workshop Appendix A Example Test Harness November 23, 2018 © Robert Sabourin, 2005

Example Unit Test Harness Homebrew unit test harness which executes a library of test cases VC++ Code for MS-Windows security client (firewall, anti-virus, pop-up blocker, parental control) with Linux server back end Hourly builds with full unit test regression Complete build from top of source tree Complete run of all test cases Report results Updates status November 23, 2018 © Robert Sabourin, 2005

Example Unit Test Harness Workflow Development lead assigns activity to developer traced with issue tracking system based on BugZilla Developers check out modules from CVS Modify or Create code Modify or Create unit test modules Unit Testing Modules Basic or smoke test Intermediate testing Exhaustive or comprehensive test November 23, 2018 © Robert Sabourin, 2005

Example Unit Test Harness Samples November 23, 2018 © Robert Sabourin, 2005

Example Unit Test Harness Samples November 23, 2018 © Robert Sabourin, 2005

Example Unit Test Harness Samples November 23, 2018 © Robert Sabourin, 2005

Example Unit Test Harness Samples November 23, 2018 © Robert Sabourin, 2005

Example Unit Test Harness Status: Now looping over commands... Exec: CliAgentD run interactionTest clientbase/zkbaseuitest Output: [0.20 second(s)] Test clientbase/zkbaseuitest succeed - (5/5) Exec: CliAgentD runall test [18.15 second(s)] Test BusinessLogic/UpgradeMgr succeed - (12/12) [3.205 second(s)] Test PacketShim/PacketDispatcher succeed - (11/11) xxx [0.111 second(s)] Test wordscan/WSDictionaryUnitTest succeed - (13/13) [0.30 second(s)] Test wordscan/WSPreferencesUnitTest succeed - (58/58) [14.400 second(s)] Test wordscan/WSScannerUnitTest succeed - (8929/8929) [0.200 second(s)] Test zktools/FirewallRuleManagerUnitTest succeed - (293/293) [0.30 second(s)] Test zktools/FirewallUnitTest succeed - (200/200) [0.0 second(s)] Test zkui/commom/UiNavEntry succeed - (0/0) Summary of QWERTY Unit Tests: Tests failed : 0 Tests succeed : 12805 ------------------------- Tests executed : 12805 Success Rate : 100% November 23, 2018 © Robert Sabourin, 2005