SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Quality assurance of non- functional requirements Mika Mäntylä TKK/SoberIT/SPRG T Software engineering seminar

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 2 Outline  Introduction  QA of performance and usability (short)  QA of evolvability (long)  How are qualities effected by  code review,  pair-programming  exploratory testing  Practical approach

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 3 QA of … Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 4 … with TDD Module Testing Load testing Exploratory testing Bounce checker Test case based testing Code review Load tester Pair- programming GUI robot Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 5 TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair- programming Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance UsabilityScalability Security Availability Maintainability Modifiability TestabilitySafety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance QualitiesMethods

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 6 TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair- programming Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance UsabilityScalability Security Availability Maintainability Modifiability TestabilitySafety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 7 TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair- programming Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance UsabilityScalability Security Availability Maintainability Modifiability TestabilitySafety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 8 TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair- programming Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance UsabilityScalability Security Availability Maintainability Modifiability TestabilitySafety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Qualities and methods have many to many relationship

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 9 TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair- programming Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance UsabilityScalability Security Availability Maintainability Modifiability TestabilitySafety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Relationships between qualities and methods are poorly understood

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 10 TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair- programming Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance UsabilityScalability Security Availability Maintainability Modifiability TestabilitySafety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Method may not be the most important factor affecting a quality

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 11 TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair- programming Unit testing Functional testing Acceptance testing External beta- testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance UsabilityScalability Security Availability Maintainability Modifiability TestabilitySafety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Specific methods for specific qualities

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 12 Summary  Qualities and methods have many to many relationship  Often relationships between qualities and methods are poorly understood  Method may not be the key factor when assuring particular quality  Code review’s effect on security  Largely dependent on the reviewer’s instructions  Specific methods exist for some qualities  Usability testing  Load testing

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 13 Outline  Introduction  QA of performance and usability (short)  QA of evolvability (long)  How are qualities effected by  code review,  pair-programming  exploratory testing  Practical approach

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 14 QA of performance  Requirements analysis  Estimate data volumes and user volume & behavior  Peak usage vs. Regular usage  Realistic data volumes  Acceptable response time  Hardware restrictions  With clustered server you can always buy more servers with mobile phone you cannot  Design & Code  Testability of the application with load testing tools  Design solutions  Bottlenecks, key algorithms, clustering  Testing  Data gathered in RA is crucial  Automate and parameterizes scripts  Measure  Use profiler to identify bottlenecks

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 15 Nixon B. A., 2000

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 16 QA of Usability  Usability – Quality in Use  The capability of the software to be understood learned used and liked by the user under specified conditions (ISO )  Understand context and users  Design for Usability  Field observation, interviews, prototypes  Usability reviews, e.g.  Heuristic evaluation  Cognitive walkthrough with task scenarios  Usability testing, e.g.  Thinking aloud protocol in usability lab  Co-discovery (Pair usability testing)  Surveys and questionnaires

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 17 Outline  Introduction  QA of performance and usability (short)  QA of evolvability (long)  How are qualities effected by  code review,  pair-programming  exploratory testing  Practical approach

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 18 Quality assurance of evolvability  Design  Company Standards / Coding conventions  Code inspections  Assessing software evolvability with tools  Improving software evolvability  Knowledge of the system  Environment support

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 19 Method 1 for evolvability: Design  Design for maintainable software  Principles  Low and loose coupling  System level, E.g.  No database specific functionality  Subsystem level, E.g.  Clear separation of responsibilities  Interaction through Interfaces  High cohesion  Information hiding  Most probable change scenarios should be evaluated  Different software component have, e.g. GUI vs. Encryption  different lifetimes  different modification rates  Collaborative design  Design review

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 20 Method 2 for evolvability: Coding conventions  Program layout has great impact to program comprehension  Documentation  Naming of code elements  Should reflect their intent  Naming standard  Commenting  Studies have shown that too much comments are also harmful !  Use when naming does not clearly state the intent  e.g. “Check needed because of bug in Java virtual machine”  Structure – Rules for acceptable structure  Explicit metrics  “Do not write methods over 100 lines”  Or heuristics  “Make sure all methods perform single explicit task”

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 21 Method 3 for evolvability: Code review  Spreads knowledge of the system  Increases evolvability inderectly  Distribution of CR issue (performance under functionality)

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 22 Method 4 for evolvability: Tools  Code metrics  E.g. Lines of code, number of coupling, execution paths  Quantify the vague concept of evolvability  Predict future development effort  Also industrial adoption  Reverse-engineering  i.e. identify the system's components and their interrelationships, e.g. code - > UML  Analyze whether the design matches the actual source  Metrics  Pro’s  Provides objective numbers  Does not require in depth understanding of the system  Con’s  Does not say anything about the correctness (quality)  e.g. Does the class have the correct associations  Reverse-engineering  Pro’s  Provides higher level view of the source code  Can be compared with the intended design  Con’s  Requires in depth understanding of the system

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 23 Method 5 for evolvability: Improve  Terms  Refactoring - Controlled way to improve the software’s structure without changing its observable behavior  Re-engineering - Examination and alteration of a subject system to reconstitute it in a new form and the subsequent implementation of the new form  Re-writing – Recreate the product starting with clean slate  All of the above can come handy during software evolution  Refactoring - continuous activity performed by all software developers  Re-engineering - when current system structure is no longer usable  Rewriting  When re-engineering is not enough  When moving to different new technology/language

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 24 Method 6 for evolvability: Knowledge distribution  Further development is harder, if expertise is lost  Expertise = solutions and locations to domain problems, domain knowledge, software architecture, design decisions  Good HRM policy can help maintain knowledge  Regardless there is risk of losing the people with knowledge  Retirement, moving to area, dying, changing career etc.  Strategies to keep vital knowledge of the system  Documenting  Spreading knowledge to several people

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 25 Method 7 for evolvability: Environment support  Good unit test harness increases evolvability  Increases courage to  improve software structure  add new features  Reduces developers test efforts  Other environment support can have similar effect  “We don't need unit test as we have excessive number of testers”

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 26 Quality assurance of evolvability  Design  Company Standards / Coding conventions  Code inspections  Assessing software evolvability with tools  Improving software evolvability  Knowledge of the system  Environment support

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 27 Outline  Introduction  QA of performance and usability (short)  QA of evolvability (long)  How are qualities effected by  code review,  pair-programming  exploratory testing  Practical approach

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 28 Code review’s effect to qualities  Spreads knowledge of the system  Increases evolvability inderectly  Distribution of CR issue (performance under functionality)

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 29 Pair programming’s effect to qualities  Increases understandability  Makes design better (evolvability)

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 30 Exploratory testing’s effect to qualaties  Single study: Exploratory testing (ET) vs Test Case Testing (TCT)  ET found more usability and GUI issues  TCT found more performance issues TypeETTCT ET/TCT (%) Total Documentation % 12 GUI % 119 Inconsistency % 8 Missing function % 194 Performance % 80 Technical defect % 120 Usability % 24 Wrong function % 502 Total % 1059

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 31 Outline  Introduction  QA of performance and usability (short)  QA of evolvability (long)  How are qualities effected by  code review,  pair-programming  exploratory testing  Practical approach

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 32 Teams with different goals Rank effort Rank statem. Rank memory Rank clarity Rank output Team1 (effort) Team2 (statem.) Team3 (memory) Team4 (clarity) Team5 (output)

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 33 Practical approach FunctionalityReliabilityUsabilityEfficiencyMaintainabilityPortability Code Review Test sessions Load Tests Multi-platform build Company specific list of practices Company specific list of quality attributes, e.g.: Only developing Windows application get rid of Portability Developing crypto library remove Usability take in Security

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 34 Summary  Many quality attributes and many quality assurance methods exist  Lecture series instead of single lecture?  Plenty of research on assuring certain qualities  Usability, Security, Evolvability (=Maintainability)  Lack of research on qa methods effects on qualities  Effects of pair-programming, exploratory testing to qualities  Exists specific methods for specific qualities  Usability testing

SoberIT Software Business and Engineering Institute HELSINKI UNIVERSITY OF TECHNOLOGY Mika Mäntylä 35 Topics  Consulting plan for a company  How to test Unit tests effect to quality X