Test Processes for Information Systems TEMA Levels of Test Technology Test Process according to IEEE-829 Testing as a Separate Subproject Requirement Solicitation Test Phases of the V-Model Test Objects of the V-Model Life Cycle Test Process by W. Perry Dual Track Process Model Test as a parallel Project by H. Sneed A Standard Test Process ANECON System Test Process Sneed System Test Process System Testing Cycle Defining Test Levels Establishing a Testing Profile Planning the System Test Defining Test Requirements Allocating Test Requirements Developing Test Scenarios Implementing the Test TPI Test Process Evaluation Model Key Areas of a System Testing Process Test Process Maturity Levels Test Process Maturity Categories Sample Test Process Maturity Matrix Conclusions for testing Information Systems 1 14 2 15 16 3 4 17 5 18 19 6 7 20 8 21 9 22 10 23 11 24 12 25 13 26

Levels of Testing Technology TEMA TPRO-1 Levels of Testing Technology From abstract Test Process Railroad Process Model Increasing Level Of Detail Choice of a Testing Strategy Test Strategy Data Driven System Test Test Selection Method Selective Test Data Generation Test Methods Assertion Language for Test Data Specification Test Scripts Test Procedures Assertion-based Test Data Generator Test Tools To Concrete

Test Process according to ANSI/IEEE-829 TEMA TPRO-2 Test- Plan Test- Design Test Case Specificat. Test Proc Coding Test Object Delivery T e s t Execution Test Log Test Incident Reporting Test- Result Reporting

Testing as a Separate Subproject TEMA TPRO-3 Testing as a Separate Subproject The Main Project Requirements Subproject Development Subproject Test Subproject

Requirement Solicitation is the Basis of Testing TPRO-4 TEMA Requirement Solicitation is the Basis of Testing System Requirements Functional / Non-Functional Requirement Collection Basin System Requirement Documents Hardware Software

Test Phases in the V-Modell TEMA TPRO-5 Requirements Acceptance Analysis System test Design Integration test Coding Unit Test Components

Test Objects in the V-Modell TEMA TPRO-6 Req. Specs System test Procedures System test cases System Design Integration test Procedures Integration test cases Module Design Unit test Procedures Unit test cases Code

Life Cycle Test Process by W. Perry SYST TPRO-7 Development Requirements System Maintenance Projektstart Systemtest Regression test Requirements Test Requirements Design Programming Test Integration Maintainenance SDLC-Phase nach ISO-12207

Dual Track Process Model TPRO-8 TEMA Requirement Specification Development Track Testing Track Ties Delivered Software System Introduced by Gelperin & Hetzel Founders of the ISQI

Test as a Parallel Project TEMA TPRO-9 Development Project Require- ments N Project Planning Analysis Design Coding Unit Test Correct? Y Test planning Test Design Test Case specification Test Data & Procedures System Test Test Project Test Evaluation Acceptance

A Standard Test Process TEMA TPRO-10 Init Test Planning Test Plan Test Conception Test Concept Test Spec. Test Cases Test Execution Test Results Test Evaluation Closure Test Docum.

ANECON System Testing Process TPRO-11 ANECON System Testing Process Req Specs Test Metrics Requirement Analysis TEXT Test Planning Test Plan Test Design TPLN TCON Preliminary Test Cases Test Case Specification Final Test Cases TSPC Test Data Generation Expected Results Test Data Schemas TDAT Test Execution Test Evaluation Test Results TEXE TVAL Test Documentation

Sneed System Test Process (1) TEMA Base Line Document German/ English Texts Requirement Analysis Req. Based Test Cases Requirement Specs. Test Plan Test Goals Test Strategy Test Method Test Procedures Test Effort Estimation ................. DESIGN UML Diagrams Information System Architecture Design Analysis Design Based Test Cases Test Case Database Test Case Id Test Case Purpose Test Case Trigger Requirement Tested Objects Tested Use Case Tested Pre Conditions Post Conditions ................. CODING Java Classes Code Components Code Analysis Code Based Test Cases

Sneed System Test Process (2) TEMA Tester Data Interfaces Test Case Specification Information System Design Test Data Generation Test Data Validation Report Test Case Database Test Design Information System Test Scenarios Test Result Evaluation Post States Pre States Test Test Case Analysis Information System Test Requirements Test Environment Construction Test Results System Under Test Environment Test Monitor Test Case Deficiencies Test Evaluation Test Coverage

System Testing Cycle Analyze Requirement Plan Test Soll Ist Extract TPRO-13 TEMA System Testing Cycle Analyze Requirement Plan Test Soll Ist Extract Test Cases Specify Test Cases Design Test Scenarios Generate Test Data Construct Test Environment Execute Test Evaluate Test

DOD Testing Levels Software Development Plan Software Test Plan TPRO-14 TEMA DOD Testing Levels Software Development Plan Software Test Plan Secondary Test Plans, Procedures, Scenarios, and Data System Testing Integration System Integration Build Subsystem functional (allocated by build) Unit Test Module Test Level 1 Level 2 Level 3 Level 4 Level 5 Level 6

Establishing a Testing Profile TPRO-15 TEMA Establishing a Testing Profile Level 6 Level 5 Configuration management Organizational strategies External simulator Regression testing Hierarchal testing Random testing Software instrumentation Level 6 5 4 3 2 1 Level 4 Level 3 Configuration management Regression testing System simulation Organizational autonomy Incremental qualification Software instrumentation Level 2 Level 1 Configuration management Data reviews Test tool definition

Planning the System Test TEMA TPRO-16 Test Profile System Test- plan Software Development plan Secondary planning documentation System Procedures Software Test plan Phase 1. Phase 2. Functional Procedures Scenario Data System Test Scenarios Test tools, Techniques and methodologies Build Test Plan 1 Build Test Folder General Unit Test Plan Procedures, Scenarios, data System Test Data

Defining the Test Requirements TEMA TPRO-17 Defining the Test Requirements System Test- plan Construction Rules Software Development plan Planned Metrics Programming Practices, Standards and conventions Quality guidelines Configuration management Software Test plan Test roadmap Test specs. Functional Procedures Build Test Plan 1 Test Scenarios and Data Software Design Specification Build Test Folders Build Procedures, Scenarios and Data

Allocating the Test Resources TPRO-18 TEMA Allocating the Test Resources System Test- plan Software Development plan Software Test- plan Schedules _____ Functional Procedures Allocated To builds Functional Specification Build Test- plan Software architecture Build Test Folder Database architecture

Developing Test Scenarios TEMA TPRO-19 Developing Test Scenarios Integration Test requirement Level 3 Test scenario Test step Software Test- plan Test step Test scenario Test Procedure Level 5 Test scenario Test step System Test- plan Test step Test scenario Test Procedure

Implementing the Test Test planning and development Test plans Build TEMA Implementing the Test TPRO-20 Test planning and development Test plans Build test folder System test folder Test procedure Test implementation and execution Development testing Test scenario Test scenario Test scenario Test data Test data Test data Test implementation and scenario Test tooling Unit development folder Test execution Test environment creation Test analysis Test execution Test analysis

The TPI Test Process Evaluation Model (Koomen & Pol) TEMA TPRO-21 Test maturity matrix Key Areas Maturity levels Acquired points Impovement Suggestions

Key Areas of a System Test Process TEMA TPRO-22.1 Key Area Level A Level B Level C Level D Test Strategy Strategy for single high-level test Combined strategy for high-level tests Combined strategy for high-level tests plus low-lvel tests or evaluation Combined strategy for all test and evaluation levels Life-cycle model Planning, Specification, Execution Planning, Preparation, Specification, Execution, and Completion Moment of involvement Completion of test basis Start of test basis Start of requirements definition Project initiation Estimating and planning Substantiated estimating and planning Statistically substantiated estimating and planning Test specification techniques Informal techniques Formal techniques Static test techniques Inspection of test basis Checklists Metrics Project metrics (product) Project metrics (process) System metrics Organization metrics (>1 system) Test tools Planning and control tools Execution and analysis tools Extensive automation of the test process Test environment Managed and controlled test environment Testing in the most suitable environment ‘Environment-on-call’ Office environment Adequate and timely office environment

Key Areas of a System Test Process TEMA TPRO-22.2 Key Area Level A Level B Level C Level D Commitment and motivation Assignment of budget and time Testing integrated in project organization Test-engineering Test functions and training Test manager and testers (Formal) Methodical, Technical and Functional Support, Management Formal internal Quality Assurance Scope of methodology Project specific Organization generic Organization optimizing, R & D activities Communication Internal communication Project communication (defects, change control) Communication within organization about the quality of the test processes Reporting Defects Progress (status of tests and products), activities (costs and time, milestones), defects with priorities Risks and recommendations, substantiated with metrics Recommendations have a Software Process Improvement character Defect management Internal defect management Extensive defect management with flexible reporting facilities Project defect management Testware management Internal testware management External management of test basis and test object Reusable testware Traceability system requirements to test cases Test process management Planning and execution Planning, execution, monitoring, and adjusting Monitoring and adjusting within organization Evaluation Evaluation techniques Evaluation strategy Low-level testing Low-level test life cycle (planning, specification and execution) White-box techniques Low-level test strategy

Test Process Maturity Levels TEMA TPRO-23 A Minimum Fulfillment, low maturity level (1) B Medium Fulfillment, average maturity level (2) C General Fulfillment, high maturity level (3) D Full Fulfillment, maximum maturity level (4)

Test Process Maturity Categories TEMA TPRO-24 1-6 Controlled Test Process is planned and controlled Test cases are specified, Test results are documented Errors are recorded. Test progress is reported 7-10 Efficient Test automation, Data generation, Result validation, Test documentation 11-13 Optimizing Control improvement Tuning, Calibrating First structure, then automate (Pol) Automating chaos leads to faster chaos (Graham)

Sample Test Process Maturity Matrix TEMA TPRO-25 Scale Key Area 1 2 3 4 5 6 7 8 9 10 11 12 13 Controlled Efficient Optimizing Test strategy A B C D Life-cycle model Moment of involvement Estimating and planning Test specification teschniques Static test techniques Metrics Test Tools Test environment Office environment Commitment and motivation Test functions and training Scope of methodology Communication Reporting Defect management Testware management Test process management Evaluation Low-level testing

Conclusions for Testing Information Systems TPRO-26 TEMA Conclusions for Testing Information Systems The development project should be accompanied by a separate test project. Both development and test should be based on the requirements specification. The budget for the test project should be equivalent to that for the development project. The test project should proceed methodologically from the test design to the test execution and evaluation. The test project results should be subject to a similar quality assurance as the development results. 1 2 3 4 5