1. Test Management Approach (TMap)
Chris Hampton, PMP
Practice Manager, Software Control & Testing
Sogeti USA
September 2007

3. Agenda: The State of, and Business Case for, Software Quality
What is testing?
Sogeti’s Test Management Approach® (Tmap®)
Sogeti’s Test Process Improvement® (TPI®) Methodology

4. Poor State of Software Delivery
“Software failure costs the American economy $59.5 billion annually.”
Source: NIST (2002)
90% of projects are delivered late
66% of projects rated unsuccessful
The CMMI has had an impact on global software development far greater than that of any other standard. However, even with the breadth of CMMI’s impact, the Standish Group still reports disturbing results for software development.
Build 1 – 30% of projects are cancelled prior to completion
Build 2 – 54% are over budget
Build 3 – 66% are not considered successful
Build 4 – and 90% are delivered late
Build 5 – Is it any wonder that the U.S. National Institute of Standards and Technology reports that the United States alone wastes almost $60 billion per year on software problems?
54% of projects over budget
30% of projects cancelled early
Source: Standish 2003 4

5. The Business Case For QA
Compliance
Basel II, Sarbanes Oxley, HIPAA, etc…
Reduced liability from defective software and defective process
Cost Savings
Quality processes now reduce support costs later
Lost revenue
Satisfaction
Higher User satisfaction
Improved business alignment
Growth
Better quality and measurements allows development optimization
Increase ability to deliver 5

6. Where Quality Problems Originate7%
10%
56%
27%
Source: James Martin

7. How Quality Provides Tremendous Savings
Source: Cutter Consortium/Forrester/Sogeti
Planning
Analysis
System
Design
$140
Production
Code
$14000
Unit
Test
$1000
COST TO REPAIR DEFECT
Unit
Test
Certification
Test
System
Test
$4500
Integration
Test
$2500
$7000
Code
Time cost and resource effort increase
exponentially later in the lifecycle
April 21, 2017

8. What is testing? Testing is a process aimed at:
Finding defects in a controlled manner
Detecting the level of quality of the test object
Demonstrating the gap between specifications and the actual product
Demonstrating that the end product functions as called for in the requirements
Ask the question first, and let trainees come up with answer. Show panel afterwards.
The panel shows the “negative” approach first (finding defects). This is an important part of testing but it’s just a part. There’s more to testing. The examples slowly evolve to a more positive and comprehensive description of the meaning of testing.
A possible definition:
Testing is a process of planning, preparing and measuring aiming at establishing characteristics of an information system. Testing identifies the gap between actual and desired status of a system.

9. Testing is not . . . . Testing is not: Implementation Acceptance
Defect repair
A phase after development – although testing includes a phase after development
Training on a new system
Ten behoeve van een juiste afbakening wordt door middel van deze slide ingegaan op de vraag "Wat is testen niet?".
Testen en implementeren zijn tegengestelde activiteiten. Testresultaten torpederen de implementatieplannen nogal eens.
Testen is niet vrijgeven of accepteren. Het testen levert advies over de kwaliteit. De vrijgave beslissing is aan anderen, veelal de opdrachtgever van de test.
Testen is ook geen foutherstel, slechts bij de programmatest is testen en herstellen in een hand te leggen. Bij alle andere testen moet het principe gehanteerd worden, dat er niet getest wordt door de persoon die heeft gebouwd en niet hersteld wordt door de persoon die heeft getest.
Testen is niet goedkoop, de testkosten schommelen, afhankelijk van het systeemtype, tussen de 20 en 40 procent van de ontwikkelingskosten. Aan de andere kant is testen misschien toch wel goedkoop. Een goed uitgevoerde test kan een kwalitatief beter systeem opleveren waardoor tijdens productie minder fouten/storingen zullen optreden.
Testen is niet een fase na ontwikkeling, het behelst een serie activiteiten die moeten worden uitgevoerd vanaf een pril stadium van ontwikkeling.
Testen is in eerste instantie niet bedoeld om het systeem te toetsen op volledigheid en juistheid van de functionaliteit. Dit wil niet zeggen dat er tijdens het testen niet ingegaan mag worden op onjuistheden in de functionele specificaties.
Het mag niet het hoofddoel worden en er moet procedureel anders mee worden omgegaan.
Implementation is a phase after testing
Acceptance is done by business people, never by the test team. The test manager advices.
The test team finds defects, developers do defect repair
Testing is expensive, but in the long run it will save you money, if done in a structured manner
Test planning, specification and preparation is done prior to and during development. Only the execution (and completion) phases take place after development
Business analysts and users define the desired functionality. The test team checks if the system meets these requirements. Testers do not doubt the desired functionality

10. Why do we test ? To mitigate business risks that have been identified:
Validate software quality
Verify integration of business processes
Reduce time-to-market
Competitive purposes
To ensure business usability of the software:
Release low-error/known-error software
Move high quality software (meets or exceeds quality expectations) into production
Demonstrate the usefulness of new technology
The industry has not yet reached a level of maturity where testing is obsolete (will probably never happen).

11. Testing of information systems
What do we test?
Application Software
Hardware
System Software
Procedures
Documentation
Functionality
Continuity
Performance
Usability
Interoperability (between different applications and systems)
An information system (test object) is more than software alone. It also includes hardware, documentation etc.
Quality characteristics are used to determine what needs to be tested. 11

12. Structured Software Testing
Testing everything is impractical
Funding limitations
Limited time and resources
Diminishing value after a certain point
Isn’t there a more effective way?
Yes! Structured software testing: a risk-based, quality-centric approach to testing

13. Test Management Approach (TMap®)
Sogeti’s methodology for structured testing of software products.
TMap® has evolved to be the standard for software testing in Europe and quickly gaining traction in the US.
It is being used by more than three hundred Dutch, Belgium and German organizations.
Industry adoption includes :
Financial Services
Insurance
Government
Consumer Electronics
Telecommunications
Medical systems
Gold left-most triangle - TMap is based on a business-driven test management (BDTM) approach.
Right side parallelogram - TMap describes a structured test process.
Gray bottom triangle - TMap contains a complete tool box.
Arrow all around - TMap is an adaptive test method.
< Details on next slides > 13 13

14. 4 Essentials of TMap®
Business-driven test management (BDTM) approach
Structured test process
Complete tool box
Adaptive test method
Gold left-most triangle - TMap is based on a business-driven test management (BDTM) approach.
Right side parallelogram - TMap describes a structured test process.
Gray bottom triangle - TMap contains a complete tool box.
Arrow all around - TMap is an adaptive test method.
< Details on next slides >
BDTM puts focus on risk-based testing—only testing what has impacts to the organization. 14

15. Business Driven Test Management
Choices must be made in what is tested and how thoroughly.
This is an economic decision based on:
The risks that an organization thinks it will incur
The availability of time and resources
The results the customer wishes to achieve.
The choices based on risks, result, time and cost constitutes the basis for the BDTM approach.
The risks of the system = product risks, the risk of defect occurring in production.
The short of BDTM is that there is a business case for what gets tested. Costs of defects vs. the costs of testing.
Steps:
Defining test of scope and gathering test goals.
Determining risk class for each combination of characteristic and object part.
Determining whether a combination of characteristic and object part must be tested thoroughly or lightly.
An overall estimate is then made for the test and a planning set up.
Allocating test techniques to the combinations of characteristic and object part.
Throughout the test process, the test manager reviews with various stakeholders 15

16. The BDTM Process Formulate the test goals.
Determine the risk class for each combination of characteristic and object part.
Determining the depth of testing required.
Based on risk/resource trade off
Develop the test estimate and test plan.
Allocate the appropriate test techniques.
Throughout the test process, the test manager provides the stakeholders with insight into and control options over test process and test object. 16

17. Risk Definition
Testing should be based on the mitigation of risk and validation of expected quality defined by the business requirements
Definition of risk
A “risk” is a chance of a failure occurring, related to expected damage to the project (should the failure occur)
Risk Categories
Risks identified in different categories such as business risks, project risks, product risks, process risks
Risk Ranking
Risk are ranked in criticality relative to one another, by instituting a method of risk ranking (or “risk rating”) 17

18. How Risk Ranking Works
Assemble a team of people representing various roles from the project
PMs, Business Leads/SMEs, Test Leads, Tech Leads
Create an initial list of risks
Team assigns a numeric value to each risk = the probability of occurrence of each risk
Team assigns a second value to each risk, representing the impact on the project/organization
Multiply the two values together (the probability of occurrence X the impact)
The result is a relative value for each risk
Order the risks by their relative risk values
“risk rating” or “risk ranking”
Ranking helps manage the most critical risks, especially those falling in the middle tier of the ranking
Periodically update the list of risks

19. Risks – quality characteristics matrix
High
Priority
Low
Quality Characteristics
Medium Priority
Risks
Medium
High Priority/
Low Risk
Low Priority/
High Risk
Medium Priority/
Medium Risk
Multi-Level Matrix

20. Ten steps for creating a test case
Identify sub-systems / sub-processes
Identify risks
Rank risks
Agree on relevant quality characteristics
Rank relevant quality characteristics
Create subsystem / quality characteristic pairings
Create test scenarios for each subsystem
Associate test scenarios with risks
Select suitable test techniques
CREATE TEST CASES
Facilitated Sessions
FUNCTIONAL AREA

21. Structured Test Process
Master Test Plan, managing the total test process
Acceptance and System Tests
Developmental Test
Supporting Processes
Lifecycle Model
Master Test Plan is developed. Manage the entire test process (planning & control of testing).
Acceptance & System Tests –Lifecycle model for these business-focused tests.
Development tests – testing against design
Supporting processes – make sure that test processes comply with overall quality initiatives

22. Master Test Plan
Developed by the test manager in partnership with the appropriate stakeholders
Jointly define:
What will be tested for each test level,
When it will be tested
To what degree of thoroughness.
This plan constitutes the basis for the test plans for the separate test levels.
Unit Testing
Systems Integration Testing
Acceptance Testing
Developmental Testing

23. Life Cycle Model The life cycle model is a generic model.
Can be applied to all test levels and test types
Used in parallel with the life cycle models for system development.
Test activities are divided across seven phases:
Planning
Control
Setting up and maintaining infrastructure
Preparation,
Specification
Execution
Completion

24. TMap - Structured testing lifecycle
P&C P S E C
PREPARATION
Testability review
Allocate techniques
Requirements review
Data cycle test
Process cycle test
Other tests
Specify test cases
Create/assess infrastructure
Test specification
Data flow test
EXECUTION
Pretest
(Re)test
Checking
Assessing
Debugger
Record and playback
Monitoring
Preserve testware
Evaluate process
PLANNING & CONTROL
Inspection and study -Test strategy
Develop test strategy -Test estimation
Risk analysis -Reports
Test Estimation (TPA) -Management tools
Setup organization -Budgeting tools
Prepare test plan -Defect management
Management and control tool
SPECIFICATION
Create test scripts
Create infrastructure
COMPLETION
Preserve testware for future use
In the Planning phase, the test manager formulates a coherent approach that is supported by the client to adequately execute the test assignment. This is laid down in the test plan.
Control phase - the activities in the test plan are executed, monitored, and adjusted if necessary.
The Setting up and maintaining infrastructure phase aims to provide the required test infrastructure that is used in the various TMap phases and activities.
The Preparation phase aims to have access to a test basis, agreed with the client of the test, of adequate quality to design the test cases.
The tests are specified in the Specification phase and executed in the Execution phase.
This provides insight into the quality of the test object.
The test assignment is concluded in the Completion phase. This phase offers the opportunity to learn lessons from experiences gained in the project. Furthermore activities are executed to guarantee reuse of products.

25. Complete Tool Kit Techniques
Many techniques can be used in the test process. A test technique is a combination of actions to produce a test product in a universal manner.
TMap provides techniques for the following:
Test estimation
Defect management
Creating metrics
Product risk analysis
Test design
Product evaluation.

26. Adaptive Test Method Complete And Flexible Works with any SDLC
Iterative
RAD
Agile
Waterfall
RUP
Works with any technology
Mainframe
Web
ERP
SOA

28. TMap® – Differentiators
Business-Driven Test Management
Risk-based test strategy
Efficient testing—coverage without overlap
Testing off of critical path as much as possible
Criteria and metrics about production readiness
Management of testing to project timelines
Compliments industry tools
Proven methodology applied in hundreds of companies internationally over the past 14 years!

29. Benefits of Structure Testing
Defects are found early, costing less time/money to reach production  delivering a higher quality product
Required quality of the various test objects is tested for and validated, by focusing on testing for quality as a risk mitigation strategy
By keeping a larger percentage of the testing process/effort off the critical path, faster time-to-market results
Structured software testing is more cost-effective and efficient than non-structured testing approaches
Sound test coverage is provided, without the need to overlap phases
Establishes a test organization that is prepared and efficient
Delivers a repeatable process
Coverage – requirements that are agreed must be tested HAVE been tested

30. Test Process Improvement (TPI)
The Sogeti TPI® Model was developed to facilitate a stepwise improvement of the testing process.
Developed from practical experience from 30+ years of Sogeti work in testing.
Offers a frame of reference to determine the strengths and weaknesses of your current testing process.
Covers 20 key areas that need specific improvement to achieve a well define testing process.
Key TPI Areas:
Test Strategy
Life-Cycle Model
Moment of Involvement
Estimating and Planning
Test Specification Techniques
Static Test Techniques
Metrics
Test Automation
Test Environment
Office Environment
Commitment and Motivation
Test Functions and Training
Scope of
Methodology Communication
Reporting
Defect Management
Testware Management
Test Process Management
Evaluation and Low-Level Testing

31. Typical TPI Process Steps
Interview key people
Describe the ‘As-Is’ process including Strengths & Weaknesses
Describe the ‘To-Be’ process
Document process improvement actions
Define expected benefits in terms of Objectives
Plan for Implementation of the Improvement Actions (short-term and long-term)

32. TPI - Test Maturity Matrix
Increasing Maturity

33. Resources Web Sites TMap® certification (Netherlands) Books
Sogeti USA Web Site:
TMap Web Site:
TMap® certification (Netherlands)
Foundation
TMap ® professional Advanced/Expert
TMap ® management Advanced/Expert
Books

34. Sogeti SCT Services Managed Testing Services (MTS)
Quality Management Services
Test Automation Services
Testing Professional Services
SAP Testing Services
3000+ people worldwide in SCT space
30+ yrs experience

35. Sogeti’s Geographical Presence
A workforce of over 18,000 staff spread over 200 locations
USA
Sweden UK
Denmark
Ireland NL
Germany
Belux
Switzerland
France
Spain
India

36. Practice Manager, Software Control & Testing
Thank you!
Chris Hampton, PMP
Practice Manager, Software Control & Testing
Sogeti USA
Cell: (214)
Sogeti Office: (972)