Don’t Just “Test”… Validate!!

Agenda Overview of Testing versus Validation Common Activities and Responsibilities Justification for Validation Discussion

Traditional View of Testing Production Environment Install Set For Year End Design and Coding Project Life Cycle

Project Life Cycle Time & Resource Constraints Production Environment Hiring Freeze Install Set For Year End Design and Coding Production Environment Testing Changes to Requirements Project Life Cycle

Risk Factors Examples of Project Risk Factors SEI identifies at least 10 project risk factors: Over-ambitious Schedule Under-ambitious Budget Over-ambitious or unrealistic expectations Undefined or misunderstood obligations Inadequate software sizing estimate Unsuitable or lack of development process model Continuous requirements change Unsuitable organizational structure Inadequate software development plan Lack of political support Assess these factors in relationship to the testing effort

Factors Affecting Testing Short Discussion: What have you seen?

Testing Objectives Maximize System Quality Minimize Testing Effort Did we build the system right? Maximize System Quality Minimize Testing Effort

What Is Validation? Providing documented evidence with high degree of assurance the system will consistently produce a product or perform a process which meets the predetermined specifications and quality attributes

Critical Success Factors Factors Driving Validation Risk to Business Critical Success Factors Validation Business Needs

What to address most thoroughly Relationship of Risk to Validation Risk tells you What to address first What to address most thoroughly Degree of Validation Degree of Risk

Did we build the right system? Maximize System Quality Validation Objectives Did we build the right system? Maximize System Quality Meet Business Needs

Goals of Validation Control of system development: Requirements meaningful, specified, and approved Effective process capability of developer Existence of and conformance to SOPs Effective, appropriate, documented testing Configuration management & change control Control of system use: Current accuracy and reliability Continued accuracy and reliability Management awareness and control Auditability Data integrity Reviewer independence and credentials

The Business Environment What Makes Up A System? Peripherals Documentation Procedures Infrastructure Software Hardware Trained Personnel The Computer System The Business Environment

The Computer System Testing (traditionally) Encompasses… Documentation Procedures Infrastructure Software Hardware Trained Personnel The Computer System

The Business Environment Validation Encompasses… Peripherals Documentation Procedures Infrastructure Software Hardware Trained Personnel The Computer System The Business Environment

CMM Quality Assurance Program Program Owner Accountabilities QA Accountabilities occur at every touch point Product Process Idea Generation General Release Customer Requirement Marketing Design Project Orientation Initiation Planning Executing Closure Transition Construction Elaboration Inception Transition SDLC Project Management Process Test Strategy Test Construction Test Scope Test Planning Test Execution Test Evaluation QC Process Implementation Operations Support Retirement

Measurement & Analysis QA & QC Coming Together Quality Assurance Process Audits Process Consultation Defect Analysis Measurement Analysis Continuous Improvement Standards Development Project Orientation Initiation Planning Executing Closure Transition Risk Management Defect Management Test Planning Static Testing Dynamic Testing Test Automation Test & Quality Control Defect Resolution Metrics Test Execution Metrics Project Planning Support Defect Cost Analysis Benchmarking Forecasting Measurement & Analysis

Each Level is Composed of These Activities Common Activities Each Level is Composed of These Activities Strategy Evaluate Execute Design

Strategy Development Determine Validation Strategy Perform Risk Assessment Identify Critical Success Factors Set Validation Objectives Define Validation Activities Estimate Time & Resources Document Validation Strategy Strategy Design Execute Evaluate

Identify Critical Success Factors Correctness Authorization Data Integrity Service Levels Access Control Methodology Reliability Ease of use Maintainability Portability Performance Extent to which ... … system must satisfy stated requirements … processing requires management authorization … data stored by the system must be accurate … schedules must be met … access must be restricted … development plan must be followed … system cannot fail during operation … effort is required to learn and use system … effort is required to fix errors in the system … system can operate in multiple environments … functions must perform within a specified time

Design Develop/obtain tools for conducting Validation Activities Deliverables & Phase Checklists Procedures describing reviews & inspections Validation Matrices Test Cases Automated Testing Tools Audit plans Strategy Execute Evaluate Design

Examples of Validation Objectives # Validation Objective Priority Completion Criteria 1 Customer orders processed correctly High Enter customer order information as described in the validation test cases and verify the results are correct 2 System is ready to use High No open defects of type SEVERE or HIGH Performance criteria are met Users are trained, documentation is ready 3 Hold is placed on inventory when QC test fails Med Enter FAIL test results for product Verify MRP II indicates status as HOLD Verify product is segregated in warehouse 4 Increased through-put Low Measure increase by time study 5 Labels printed as req’d Low Print labels, verify against approved label copy specifications

Examples of Validation Activities Activity Performed by Description Deliverable Requirements Reviews Business Analysts, Developers, Users Study & discuss system requirements to ensure they meet business needs Reviewed statement of requirements, ready to be translated into design Design Reviews Developers Study & discuss system design to ensure it will support requirements Design, ready to be translated into software, hardware, documents, etc. Code Inspections Developers Informal analysis of code to find defects Software ready to be inspected or tested Code Walk-throughs Developers Formal analysis of code to find defects, problem areas, coding techniques, design flaws Software ready to be tested

Examples of Validation Activities Activity Performed by Description Deliverable Unit Testing Developers Test single units of code. Validates that the software performs as designed. Software unit ready for testing with other components Integration Testing Independent Test Team Test related units, programs, modules. Validates that multiple parts of the system interact according to design. Portions of the system ready for testing with other portions System Testing Conducted at end of Integration Testing. Test entire system. Validates system requirements. Tested system, based on specification Acceptance Testing Users Test system to make sure it works in the environment. Validates business needs. Tested system, based on user needs

Detailed Validation Matrices Requirement Test Case 1 2 3 4 5 6 Validate sign-on x Invalid term ident x Validate password x Password/ Signon error x Validate Main Menu x Re-enter O, X, M, R x Validate Customer # x Customer # Required x Field entry too long x Customer ___ not found x

Execution Organize Validation Tools Train Team Members Execute Validation Plan Track Validation Progress Perform Regression Testing Document Results Strategy Design Evaluate Execute

Evaluation Evaluate Compliance of System Development Evaluate Defects Develop Findings and Recommendations Formalize Report Strategy Design Execute Evaluate

Developer Business Unit (User) Validation Process Quality Assurance Who Is Responsible for Validation? Developer Business Unit (User) Validation Process Quality Assurance Independent Tester

Business Unit Responsibility Define system requirements Define data used in the system Enforce Change Control Plan & Perform Acceptance Testing Ensure adequate backup & recovery of data is available Work with developers to coordinate schedules Investigate Defects Develop User Documentation Develop User Procedures Enforce security Provide audit trail Train personnel

Developer Responsibility Follow development process Work with QA to improve the process Manage changes to system components Provide adequate backup & recovery Maintain configuration management Plan & Perform Unit Testing Provide Technical Documentation Maintain communication with QA , Testers & Business

QA Responsibility Define Development & Validation Process Verify personnel have adequate training & resources for their job Review documentation - adherence to standards Periodically review process activities Evaluate vendor capabilities & processes Verify enhancements & maintenance are documented Conduct Risk Assessments Verify business unit has established procedures Ensure system is adequately secure

Independent Tester Responsibility Follow Validation Process Working with Users &/or BA’s, perform Requirements Validation Work with QA to improve the process Plan & Perform Integration & System Testing Conduct Risk Assessments If required, Plan & Perform Stress & Performance Testing

Testing Levels & Validation Business Needs Acceptance Testing Requirements System Testing Design Integration Testing Code Unit Testing Production

Plan for Maintenance Prepare for Maintenance during Development Keep Deliverables and Documentation up-to-date Requirements Design Test Cases, Test Data, Test Results Maintain all related deliverables during error correction Requirement specifications Program specifications Design specifications Defect reports

Maintenance Deliverables Risk Assessment Software updates Hardware upgrades System documentation Manuals, procedures Revised/New Test Cases Updated Validation Plan

Validation Issues Responsibility lies with the user If outsourced, the responsibility is shared between the outsourcing organization and the user Develop deliverables as system is maintained Group maintenance changes together Schedule maintenance in releases Follow defined methodology

Purchased System Responsibilities Validation Activity Vendor Customer System Staff Users Feasibility Review x x Requirements Review x x x Design Review x x Code Review x Unit Testing x Integration Testing x System Testing x x Acceptance Testing x Conversion Testing x x Maintenance Testing x x

Cost of Quality The Cost of Correction $ $

Savings Cost of Quality Real Benefits Failure Appraisal Prevention Without Quality Process With Savings

Validation Costs and Benefits Incremental Benefits Benefits Net Benefits Net Costs Time Incremental Costs Costs Development Maintenance

Self-Assessment Y N ? Are risk assessments being performed on projects? Is there a methodology in place for system development and validation? Are verification techniques, such as walk-throughs, being used? Have all personnel been trained in validation techniques? Are standards in place for system development and validation? Are the users active participants in the validation process? Are the results of the validation used to improve the development and validation processes? Is the culture ready to accept new processes for validation? Is there someone responsible for validation of systems?

Did we build the right system? Did we build the system right? Validation: Did we build the right system? Testing: Did we build the system right? Hidden slide

System Validation Validates BOTH Business & Technical Requirements Reduces Development Costs by identifying Errors early in Development Cycle Reduces Risk of Operating Failures after Implementation Reduces Post Implementation Development Costs by Fixing Defects during Development Improves Overall Quality of System Greatly improves User Satisfaction

Summary The Road to Success!! Projects Delivered Projects Delivered Without Validation Projects Delivered With Validation Written proposal and remove from slide show The Road to Success!!

Questions / Discussion