1. Don’t Just “Test”… Validate!!

2. Agenda Overview of Testing versus Validation
Common Activities and Responsibilities
Justification for Validation
Discussion

3. Traditional View of Testing
Production
Environment
Install
Set For
Year End
Design and Coding
Project Life Cycle

4. Project Life Cycle Time & Resource Constraints Production Environment
Hiring Freeze
Install
Set For
Year End
Design and Coding
Production
Environment
Testing
Changes to
Requirements
Project Life Cycle

5. Risk Factors Examples of Project Risk Factors
SEI identifies at least 10 project risk factors:
Over-ambitious Schedule
Under-ambitious Budget
Over-ambitious or unrealistic expectations
Undefined or misunderstood obligations
Inadequate software sizing estimate
Unsuitable or lack of development process model
Continuous requirements change
Unsuitable organizational structure
Inadequate software development plan
Lack of political support
Assess these factors in relationship to the testing effort

6. Factors Affecting Testing
Short Discussion:
What have you seen?

7. Testing Objectives Maximize System Quality Minimize Testing Effort
Did we build the system right?
Maximize System Quality
Minimize Testing Effort

8. What Is Validation?
Providing documented evidence with high degree of assurance the system will consistently produce a product or perform a process which meets the predetermined specifications and quality attributes

9. Critical Success Factors
Factors Driving Validation
Risk to Business
Critical Success Factors
Validation
Business Needs

10. What to address most thoroughly
Relationship of Risk to Validation
Risk tells you
What to address first
What to address most thoroughly
Degree of Validation
Degree of Risk

11. Did we build the right system? Maximize System Quality
Validation Objectives
Did we build the right system?
Maximize System Quality
Meet Business Needs

12. Goals of Validation Control of system development:
Requirements meaningful, specified, and approved
Effective process capability of developer
Existence of and conformance to SOPs
Effective, appropriate, documented testing
Configuration management & change control
Control of system use:
Current accuracy and reliability
Continued accuracy and reliability
Management awareness and control
Auditability
Data integrity
Reviewer independence and credentials

13. The Business Environment
What Makes Up A System?
Peripherals
Documentation
Procedures
Infrastructure
Software
Hardware
Trained Personnel
The Computer System
The Business Environment

14. The Computer System Testing (traditionally) Encompasses… Documentation
Procedures
Infrastructure
Software
Hardware
Trained Personnel
The Computer System

15. The Business Environment
Validation Encompasses…
Peripherals
Documentation
Procedures
Infrastructure
Software
Hardware
Trained Personnel
The Computer System
The Business Environment

16. CMM Quality Assurance Program Program Owner Accountabilities QA Accountabilities occur at every touch point
Product Process
Idea
Generation
General
Release
Customer
Requirement
Marketing
Design
Project
Orientation
Initiation
Planning
Executing
Closure
Transition
Construction
Elaboration
Inception
Transition
SDLC
Project Management Process
Test Strategy
Test Construction
Test Scope
Test Planning
Test Execution
Test Evaluation QC
Process
Implementation
Operations
Support
Retirement

17. Measurement & Analysis
QA & QC Coming Together
Quality Assurance
Process Audits
Process Consultation
Defect Analysis
Measurement Analysis
Continuous Improvement
Standards Development
Project
Orientation
Initiation
Planning
Executing
Closure
Transition
Risk Management
Defect Management
Test Planning
Static Testing
Dynamic Testing
Test Automation
Test & Quality Control
Defect
Resolution
Metrics
Test Execution
Metrics
Project Planning
Support
Defect Cost
Analysis
Benchmarking
Forecasting
Measurement & Analysis

18. Each Level is Composed of These Activities
Common Activities
Each Level is Composed of These Activities
Strategy
Evaluate
Execute
Design

19. Strategy Development Determine Validation Strategy
Perform Risk Assessment
Identify Critical Success Factors
Set Validation Objectives
Define Validation Activities
Estimate Time & Resources
Document Validation Strategy
Strategy
Design
Execute
Evaluate

20. Identify Critical Success Factors
Correctness
Authorization
Data Integrity
Service Levels
Access Control
Methodology
Reliability
Ease of use
Maintainability
Portability
Performance
Extent to which ...
… system must satisfy stated requirements
… processing requires management authorization
… data stored by the system must be accurate
… schedules must be met
… access must be restricted
… development plan must be followed
… system cannot fail during operation
… effort is required to learn and use system
… effort is required to fix errors in the system
… system can operate in multiple environments
… functions must perform within a specified time

21. Design Develop/obtain tools for conducting Validation Activities
Deliverables & Phase Checklists
Procedures describing reviews & inspections
Validation Matrices
Test Cases
Automated Testing Tools
Audit plans
Strategy
Execute
Evaluate
Design

22. Examples of Validation Objectives#
Validation Objective
Priority
Completion Criteria 1
Customer orders processed correctly
High
Enter customer order information as described in the validation test cases and verify the results are correct 2
System is ready to use
High
No open defects of type SEVERE or HIGH
Performance criteria are met
Users are trained, documentation is ready 3
Hold is placed on inventory when QC test fails
Med
Enter FAIL test results for product
Verify MRP II indicates status as HOLD
Verify product is segregated in warehouse 4
Increased through-put
Low
Measure increase by time study 5
Labels printed as req’d
Low
Print labels, verify against approved label copy specifications

23. Examples of Validation Activities
Activity
Performed by
Description
Deliverable
Requirements Reviews
Business Analysts,
Developers,
Users
Study & discuss system requirements to ensure they meet business needs
Reviewed statement of requirements, ready to be translated into design
Design Reviews
Developers
Study & discuss system design to ensure it will support requirements
Design, ready to be translated into software, hardware, documents, etc.
Code Inspections
Developers
Informal analysis of code to find defects
Software ready to be inspected or tested
Code Walk-throughs
Developers
Formal analysis of code to find defects, problem areas, coding techniques, design flaws
Software ready to be tested

24. Examples of Validation Activities
Activity
Performed by
Description
Deliverable
Unit Testing
Developers
Test single units of code. Validates that the software performs as designed.
Software unit ready for testing with other components
Integration Testing
Independent Test Team
Test related units, programs, modules. Validates that multiple parts of the system interact according to design.
Portions of the system ready for testing with other portions
System Testing
Conducted at end of Integration Testing. Test entire system. Validates system requirements.
Tested system, based on specification
Acceptance Testing
Users
Test system to make sure it works in the environment. Validates business needs.
Tested system, based on user needs

25. Detailed Validation Matrices
Requirement
Test Case 1 2 3 4 5 6
Validate sign-on x
Invalid term ident x
Validate password x
Password/ Signon error x
Validate Main Menu x
Re-enter O, X, M, R x
Validate Customer # x
Customer # Required x
Field entry too long x
Customer ___ not found x

26. Execution Organize Validation Tools Train Team Members
Execute Validation Plan
Track Validation Progress
Perform Regression Testing
Document Results
Strategy
Design
Evaluate
Execute

27. Evaluation Evaluate Compliance of System Development Evaluate Defects
Develop Findings and Recommendations
Formalize Report
Strategy
Design
Execute
Evaluate

28. Developer Business Unit (User) Validation Process Quality Assurance
Who Is Responsible for Validation?
Developer
Business Unit
(User)
Validation
Process
Quality Assurance
Independent Tester

29. Business Unit Responsibility
Define system requirements
Define data used in the system
Enforce Change Control
Plan & Perform Acceptance Testing
Ensure adequate backup & recovery of data is available
Work with developers to coordinate schedules
Investigate Defects
Develop User Documentation
Develop User Procedures
Enforce security
Provide audit trail
Train personnel

30. Developer Responsibility
Follow development process
Work with QA to improve the process
Manage changes to system components
Provide adequate backup & recovery
Maintain configuration management
Plan & Perform Unit Testing
Provide Technical Documentation
Maintain communication with QA , Testers & Business

31. QA Responsibility Define Development & Validation Process
Verify personnel have adequate training & resources for their job
Review documentation - adherence to standards
Periodically review process activities
Evaluate vendor capabilities & processes
Verify enhancements & maintenance are documented
Conduct Risk Assessments
Verify business unit has established procedures
Ensure system is adequately secure

32. Independent Tester Responsibility
Follow Validation Process
Working with Users &/or BA’s, perform Requirements Validation
Work with QA to improve the process
Plan & Perform Integration & System Testing
Conduct Risk Assessments
If required, Plan & Perform Stress & Performance Testing

33. Testing Levels & Validation
Business Needs
Acceptance Testing
Requirements
System Testing
Design
Integration Testing
Code
Unit Testing
Production

34. Plan for Maintenance Prepare for Maintenance during Development
Keep Deliverables and Documentation up-to-date
Requirements
Design
Test Cases, Test Data, Test Results
Maintain all related deliverables during error correction
Requirement specifications
Program specifications
Design specifications
Defect reports

35. Maintenance Deliverables
Risk Assessment
Software updates
Hardware upgrades
System documentation
Manuals, procedures
Revised/New Test Cases
Updated Validation Plan

36. Validation Issues Responsibility lies with the user
If outsourced, the responsibility is shared between the outsourcing organization and the user
Develop deliverables as system is maintained
Group maintenance changes together
Schedule maintenance in releases
Follow defined methodology

37. Purchased System Responsibilities
Validation Activity
Vendor
Customer
System Staff
Users
Feasibility Review x x
Requirements Review x x x
Design Review x x
Code Review x
Unit Testing x
Integration Testing x
System Testing x x
Acceptance Testing x
Conversion Testing x x
Maintenance Testing x x

38. Cost of Quality The Cost of Correction$ $

39. Savings Cost of Quality Real Benefits Failure Appraisal Prevention
Without
Quality Process
With
Savings

40. Validation Costs and Benefits
Incremental Benefits
Benefits
Net Benefits
Net Costs
Time
Incremental Costs
Costs
Development
Maintenance

41. Self-Assessment Y N ?
Are risk assessments being performed on projects?
Is there a methodology in place for system development and validation?
Are verification techniques, such as walk-throughs, being used?
Have all personnel been trained in validation techniques?
Are standards in place for system development and validation?
Are the users active participants in the validation process?
Are the results of the validation used to improve the development and validation processes?
Is the culture ready to accept new processes for validation?
Is there someone responsible for validation of systems?

42. Did we build the right system? Did we build the system right?
Validation:
Did we build the right system?
Testing:
Did we build the system right?
Hidden slide

43. System Validation Validates BOTH Business & Technical Requirements
Reduces Development Costs by identifying Errors early in Development Cycle
Reduces Risk of Operating Failures after Implementation
Reduces Post Implementation Development Costs by Fixing Defects during Development
Improves Overall Quality of System
Greatly improves User Satisfaction

44. Summary The Road to Success!! Projects Delivered Projects Delivered
Without Validation
Projects Delivered
With Validation
Written proposal and remove from slide show
The Road to Success!!

45. Questions / Discussion