1. Session 5b Dr. Dan C. Surber, ESEP
ME Fall Introduction to Systems Engineering
Session 5b
Dr. Dan C. Surber, ESEP
© Copyright 2013

2. Lecture Topics Development of
Test & Evaluation Master Plan (TEMP)
Master Compliance Matrix (MCM) or Reqirements Traceability Verification Matrix (RVTM)
Use of Technical Performance Measures (TPM)

3. Definitions VALIDATION1 VERIFICATION VALIDATION2
Are these the right requirements?
Set scope & bound the problem
Operational Suitability & Effectiveness
VERIFICATION
Does the design satisfy the requirements on the basis of the methods used & results?
I, A, D, T, QbyS
Did we build it right?
VALIDATION2
Assessed under operational conditions by trained users and maintainers with tools & tech data.
Did we build the right system?

4. Flow-down Helps Testing
Requirements -> Functions -> Architecture
System, Segment, Subsystem, Products, Configuration Items, Assemblies, Subassemblies, Components
Iterations from top to bottom help “discover” interdependencies and a possible integration pathway (critical path)
Lead times, complexity, INTRAFACES also help determine integration pathway
Models, testing tools, & simulations help identify groups of test cases to be together

5. Test Plan, Case, Procedure
Test & Evaluation Master Plan:
laid out the “big picture”
Described the LEVELS of Tests, and some ordering of Tests
Test cases
describe HOW the capabilities and requirements are proven, and under what conditions
5 “W”s
Who
What
When
Where
Why
How
Test Procedure
Setup the test environment
Ensure the system under test is in a known, stable STATE & MODE
Execute the Test in a Step by step methodology to Pass/Fail the requirements
Use the verification methods prescribed by the RVM
Record the results and annotate a record with any anomalies or failures
TP should be “repeatable”
Our goal is to isolate the object under test from its test environment so any failure can be traced back to the object itself by eliminating all other causes.

6. Test Procedure Requirements that are verified
Steps to follow, given that the “setup” conditions described in the Test CASE are done
Note the steps where PASS/FAIL criteria are assessed
Ensure data files are captured, labeled, controlled
Follow post-test data reduction & analysis
ENSURE each TP is “repeatable”

7. Integration vs. Qualification Testing
Integration: How do I put things together to reduce risks, ensure qualification and assure readiness for intermediate architecture verification?
Engineering integration testing (EIT)
“White Box” tests
Qualification
“Black Box” tests
Formal, witnessed Testing
System “sell off” (pre-Validation, or Production)
Acceptance testing
Customer acceptance test

8. Master Compliance Matrix
Maps the decomposed requirements and functions to their allocations in the physical architecture and across to their verification events (level & status).
Assumes traceability & baseline configuration control are implemented.
Maps requirements & verification methods to test cases to test procedures.

9. MCM Use Data Elements from REQUIREMENTS:
Project Unique Identifier (PUID)
Description
Rationale
Allocation (to CI in the ABD)
Method of Verification
Level of Verification
ADD the following Additional Data Elements
Order of precedence for testing
Buildup from assembly to subsystem
Tie-in any TPMs
Note any RISKS and SAFETY HAZARDS
BUILD this data EVERY Level of Hierarchy

10. MCM Example PUI Ver Level Funct Facility Insp Subsy Anal HWCI Demo
CSCI
Test
Comp
M/S
QbS

11. TPMs Track at LEVEL of the System
Track to closure (usually by design reviews)
Close but do NOT DELETE – inactive
Different TPMs are appropriate for PHASES
Preliminary design
Detailed design
Production readiness

12. TPM Candidates Key Performance Parameters (KPP)
Parameters that are allocated through BUDGETS (space, weight, cost, power/ht)
Average of 10 per system/subsystem level
RISK (technical) that must be mitigated
HAZARDS that must be mitigated
INTERFACES (critical timing, sizing, throughput, bandwidth, data rates, resolution, apertures, uncertain protocols)