Progression of Test Categories Unit test Functional test Unit test . Component test System test Unit test . . Acceptance; Installation; System configuration: Performance; Consistency of UI; Security; Recovery; Regression; etc. Component test Functional test Performance; Consistency of UI; Navigation; Reliability; Security; etc. Functions; Performance; Reliability: UI; etc. Unit test Mostly Functionality 3

Component and System Test Working with more material and more people more functions (than unit test) more complexity (integrated sets of functions) more aspects (performance, reliability, consistency of UI, navigation and flow, installability, recoverability, etc.) Requiring ------- more test cases and libraries of test cases more people more steps and process activities The main Goal is still discovery of faults and fixing the errors to achieve : customer and user acceptance minimize support effort 1

Sources of Faults Component and System Tests are conducted towards the backend of the process and thus faces: undetected errors or misunderstanding or omissions from requirements phase undetected errors or misinterpretation or omissions or impreciseness from architecture and design phase undetected errors or misinterpretation or omissions from coding and implementation phase ----- and ------ errors introduced in the fixing of failures erroneously identified errors errors introduced in the combining of parts during build cycle 2

V & V Terminology Verification : Validation : Did we develop the system correctly ? Did we develop according to the way our processes and our designers dictated ? Validation : Did we develop the correct system ? Did we develop the system that the customers and users specified ? Component Test and System Test are interested in both V&V ---- but : System Test is definitely focused on the customers and users more (Validation) Lesser on the support effort (Verification) 4

Build Process In order to perform Functional, Component, and System tests there must be a way to construct these “sub-systems” so that they can be tested there must be a way to associate the test cases and testing activities with these “sub-systems” As part of the over-all Test Plan there is a portion called the Build and Integration plan Spell out the set of the parts (with pre-requisites and co-requisites, including data files ) for execution combining of the parts to form a bigger “component,” keeping track of the re-requisites and co-requisites each “spin” is a set of the components that may be executed for a component or a system test (** used in Agile a lot) 5

Configuration Manager Build and Integration may be performed manually with compilers and files but with any level of complexity, there is a need for a tool (Configuration Management tool): capability of tracking a variety of and a large number of parts speed of compilation versus accessing pre-compiled objects complexity of keeping track of pre-requisites and co-requisites keeping track of libraries of “tested and locked” parts and “untested” parts merging all the necessary parts to “build” an executable component 6

Some Definitions of Subtypes of Tests Stress Tests : testing the system to the “limits” of all the items (users, transactions, etc.) specified in the requirements ----- performance System Configuration Test: testing the overall hardware, software, and network combinations as specified in the requirements, including the compatibility and interfacing with existing systems ---- installation Recovery Test: testing the systems response to a loss of some resource and evaluating the response speed and results against the stated recovery requirement ---- back-up/recovery Security Test: testing to ensure that the security requirements are met Human Factors and Usability Test: testing to the user interface and the user performance requirements ----- UI consistency and Navigation

Reliability, Availability, Maintainability Some desirable characteristics of the system that we like our software to have : Reliability : to function consistently and correctly (according to the requirements) over a period of time (the longer the better) Availability : to be operational (every time if possible) when we need it Maintainability : to be repairable and modifiable quickly and easily (within some time frame every time, if possible). For Customer (valid) For Support (verifiable)

Measuring these Attributes A reasonable metric is needed for each of these attributes if we are to test for and understand these attributes: Mean Time To Failure (MTTF) : average time between two failures (ignoring fix time for a while) caused by some defect. For failure incidents of f1, f2, through fn, measure the time from one failure to the next failure (e.g. between fm and f (m+1) ) t1, t2 through tn. The average of t1 through tn is the the MTTF. With the MTTF, we are interested in the prediction of what the t (n+1) may be. Reliability is the estimated probability of t (n+1), based on MTTF. There are other metrics use for reliability such as defects/loc or defects /function point.

Measuring these Attributes (cont) Mean Time to Repair (MTTR): the average time required to locate and fix the problem. This is the time that the system is down and getting repaired; therefore the system is not available. Maintainability is sometimes measured with MTTR. Mean Time Between Failure (MTBF): this time between failure takes into account of the average time that the system is under repair or MTBF = MTTF + MTTR. Availability of the system may be measured with a ratio such as MTTF/MTBF. Try a numerical example to see this “availability” figure

Another Way to Use the 3 Metrics M. Shooman used the same metrics and characterized reliability, availability and maintainability with a standard scale between 0 and 1. (the closer to 1 --- the better) Reliability = MTBF/(1+MTBF) looking for big MTBF Availability = MTBF/(MTBF + MTTR) looking for big MTBF and small MTTR Maitainability = 1/(1 + MTTR) looking for small MTTR --- close to zero Repeats MTTR because MTBF contains MTTR or MTTF/MTBF (previous page)