Author Software Engineering Institute 4/26/2017 Model Testing: Testing Executable Requirements, Architecture, and Design Models Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 Donald G. Firesmith Principal Engineer Engineer

Author Software Engineering Institute Copyright 2015 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution except as restricted below. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. DM-0002518 Author Software Engineering Institute 4/26/2017

Verification and Validation Methods Traditionally, V&V ≈ Analysis + Demonstration + “Inspection” + Test. There are actually more methods for verification and validation. Testing is similar to other dynamic V&V methods.

Author Program 4/26/2017 Model Testing Model testing tests executable requirements, architecture, and/or design models the same way traditional testing tests software (and systems). A requirements, architecture, or design model of a system is executable if: It describes the system in sufficient detail (e.g., components, functions, relations, processes, and status changes) that it can be executed as a simulation of that system. A means exists (e.g., modeling tool, simulation engine, execution engine, or human brain) to run the model. Model testing is NOT the same thing as Model-Based Testing (MBT)! Model testing uses testing to uncover defects in the models. MBT uses models to automate testing to uncover defects in the software / systems.

Model-Based Shift Left Testing Author Program 4/26/2017 Model-Based Shift Left Testing

Shift Left Testing Model-Based Shift Left Testing ← Author Program 4/26/2017 Shift Left Testing Major trend in software testing. Move testing earlier in development cycle (left on timeline) Four types of shift left testing: Traditional Shift Left Testing Move emphasis from system-level GUI testing via record/playback to API, component integration, and unit testing Incremental Shift Left Testing Few deliveries/releases with testing of multiple large increments Agile/DevOps Shift Left Testing Many deliveries/releases with testing of multiple small increments Model-Based Shift Left Testing ← Test executable requirements, architecture, and/or design models (move testing left to models)

Model Testing

Author Program 4/26/2017 Model Testing

Model Testing

Testable Models – Requirements Models Author Program 4/26/2017 Testable Models – Requirements Models Example executable models include: ConOps storyboards (simulations) Use case path sequence diagrams Decision trees, finites state machines, and Petri nets Executable requirements languages Requirements prototypes

Testable Models – Architecture/Design Models Author Program 4/26/2017 Testable Models – Architecture/Design Models Examples executable models include: Architecture Analysis and Design Language (AADL) models Unified Modeling Language (UML) models and System Modeling Language (SysML) models: Communication diagrams, interaction diagrams, sequence diagrams, state machine diagrams, activity diagrams, and timing diagrams, Object Constraint Language (OCL) or Action Semantic Language (ASL) Unified Profile for DODAF and MODAF (UPDM 2.0) Hardware Description Languages Architecture prototypes Design models for automatic code generation