1. 1 Software Reliability Analysis Tools Joel Henry, Ph.D. University of Montana

2. 2 Outline Background Background MATLAB Automated Testing Tool MATLAB Automated Testing Tool Graphical Input Specification Tool Graphical Input Specification Tool Real-Time Analysis Testing Tool Real-Time Analysis Testing Tool Status Status Conclusion Conclusion

3. 3 System Development Example: Wind Tunnel Software Controls devices that control: Controls devices that control: Wind generator Wind generator Model support Model support Tunnel atmosphere Tunnel atmosphere Emphasizes reliability and safety Emphasizes reliability and safety Utilizes multiple development strategies Utilizes multiple development strategies Based on simple structure Based on simple structure

4. 4 System Development Simple Structure Controlling Computers External Devices ALGORITHM Sample Inputs Run Software Update Outputs

5. 5 Testing Problems Size Size Input variables – sampled over time Input variables – sampled over time Outputs variables – produced over time Outputs variables – produced over time Sample time – variable or set frequency Sample time – variable or set frequency Requirements Requirements Input file/matrix Input file/matrix Output file/matrix Output file/matrix Analysis tools Analysis tools Domain determinants Domain determinants Input variable – minimum, maximum, and accuracy Input variable – minimum, maximum, and accuracy Output variable – minimum, maximum, and accuracy Output variable – minimum, maximum, and accuracy Test requirements Test requirements Input file/matrix with all possible values for input Input file/matrix with all possible values for input Output file/matrix much more complex problem Output file/matrix much more complex problem

6. 6 Solution Approach Overview Automation to: Automation to: Generate large input matrices/files Generate large input matrices/files Perform simulation and/or test auto-generated code Perform simulation and/or test auto-generated code Analyze output matrices/files Analyze output matrices/files Methods to: Methods to: Evaluate domain coverage Evaluate domain coverage Aid debugging Aid debugging Evaluate results Evaluate results

7. 7 Solution Approach Command and Control Algorithm Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results MATLAB/Simulink Environment Source Code Executable Code Verification and Validation Methodology Suite of testing tools

8. 8 Solution Approach Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results Suite of testing tools Verification and Validation Methodology Command and Control Algorithm MATLAB/Simulink Environment Model Information Test Data Test Results Executable Code Test Data Test Results

9. 9 MATLAB Automated Testing Tool Creates test data Creates test data Executes simulation or test Executes simulation or test Captures results Captures results Detects exceptions Detects exceptions Saves test data or complete test results Saves test data or complete test results

10. 10 MATT – Test Setup Inputs Set test running time Set test running time Select and configure tests using 31 test types Select and configure tests using 31 test types Now supports non-scalar inputs* Now supports non-scalar inputs*

11. 11 MATT – Test Setup States Assign Starting values for state blocks* Assign Starting values for state blocks*

12. 12 MATT – Test Setup Outputs Set output accuracy and exception ranges Set output accuracy and exception ranges Now supports non-scalar outputs* Now supports non-scalar outputs*

13. 13 MATT – Test Setup New Exception Types Percent change* Percent change* Allows exception detection if the output value changes more than a specified percent over a specified number of steps Allows exception detection if the output value changes more than a specified percent over a specified number of steps Absolute change* Absolute change* Allows exception detection if the output value changes more than a specified amount over a specified number of steps Allows exception detection if the output value changes more than a specified amount over a specified number of steps

14. 14 MATT – Simulate Set up simulation and code generation options Simulate: Model Auto-generated Code Compare both

15. 15 MATT – View Results View input values for every time-step View output values for every time-step Steps causing exceptions are highlighted red* View/edit output exception info and settings

16. 16 MATT – Advanced Exceptions* Advanced Exceptions allow exception detection based on multiple exception criteria Advanced Exceptions allow exception detection based on multiple exception criteria A combination of output ports A combination of output ports Disjoint ranges Disjoint ranges Create separate A.E definition for each range that needs to be tested Create separate A.E definition for each range that needs to be tested Do a combinational process based on all the A.E definitions Do a combinational process based on all the A.E definitions Overall system reliability Overall system reliability Create multiple A.E definitions based on system specifications Create multiple A.E definitions based on system specifications Perform a combinational process based on all the defined Advanced Exceptions Perform a combinational process based on all the defined Advanced Exceptions

17. 17 MATT – Advanced Exceptions Combination of output ports In the example shown below, an exception is said to have occurred when: In the example shown below, an exception is said to have occurred when: Stick output is between 5 and 10 Stick output is between 5 and 10 Alpha output is greater than 20 rads Alpha output is greater than 20 rads After clicking the process button, one discovers: After clicking the process button, one discovers: 7 exceptions occurred from the given exception definition 7 exceptions occurred from the given exception definition The time steps where the 7 exceptions occurred The time steps where the 7 exceptions occurred The output values that produced the 7 exceptions The output values that produced the 7 exceptions

18. 18 MATT – Advanced Exceptions Disjoint Ranges In this example, two separate exception definitions are specified: In this example, two separate exception definitions are specified: (1) Alpha producing values between 0 - 4 rads (1) Alpha producing values between 0 - 4 rads (2) Alpha producing values between 15 – 19 rads (2) Alpha producing values between 15 – 19 rads These two exception definitions are combined, and exception results are calculated in aggregate. These two exception definitions are combined, and exception results are calculated in aggregate.

19. 19 MATT – Advanced Exceptions Overall System Reliability By combining multiple A.E. definitions, systems can be stress-tested for accuracy and reliability. By combining multiple A.E. definitions, systems can be stress-tested for accuracy and reliability.

20. 20 MATT - Storage Save complete test setup for future use Save complete test setup for future use Input, State, and Output settings Input, State, and Output settings Advanced Exception Setting Advanced Exception Setting Input Matrix Input Matrix Output Matrix Output Matrix Import and export custom input matrices Import and export custom input matrices Comma delimited files (*.csv) Comma delimited files (*.csv) MATLAB data files (*.mat) MATLAB data files (*.mat)

21. 21 GIST Reads MATT test data files Reads MATT test data files Allows freehand specification of test data Allows freehand specification of test data Saves new MATT test data files Saves new MATT test data files

22. 22 GIST – Manage Input Sets Add/Remove inputs for altered models View graphs of any input Edit any input

23. 23 GIST – Edit Inputs Use simple click and drag to create new input values View original input values as drawing

24. 24 RATT Reads MATT test result files Reads MATT test result files Calculates reliability, probabilities, and completeness measures Calculates reliability, probabilities, and completeness measures Supports analysis of multiple test files (test suite) Supports analysis of multiple test files (test suite) Exports to MS Excel and populates analysis charts and graphs Exports to MS Excel and populates analysis charts and graphs

25. 25 RATT - Analysis Load several MATT test files at once Perform analysis Input coverage Output coverage MTTF Probability of failure Etc View simple results in RATT

26. 26 RATT – Excel ® Tool Load RATT files into Excel® View graphs showing coverage View coverage on multiple inputs/exceptions at the same time

27. 27 Current Status Windows and UNIX versions of MATT ready Windows and UNIX versions of MATT ready Windows versions of GIST and RATT ready Windows versions of GIST and RATT ready Working with STEREO project Working with STEREO project Testing MATLAB/Simulink® models using MATT Testing MATLAB/Simulink® models using MATT

28. 28 Upcoming Functionality MATT MATT View intermediate state values from simulation View intermediate state values from simulation Create and use custom test-types Create and use custom test-types Load models inside MATT Load models inside MATT Allows MATT to launch outside of MATLAB® Allows MATT to launch outside of MATLAB® GIST GIST Being integrated into MATT for more seamless use Being integrated into MATT for more seamless use RATT RATT Being revised to accommodate Advanced Exceptions Being revised to accommodate Advanced Exceptions

29. 29 Conclusion – Testing Enhanced Using software tools to aid testing allows Using software tools to aid testing allows More tests to be run More tests to be run Each test to be run more quickly Each test to be run more quickly Test results to be easily stored and referenced Test results to be easily stored and referenced

30. 30 Conclusion – Tools Enhanced Improved MATT capabilities allow Improved MATT capabilities allow Systems to start at any state with ease Systems to start at any state with ease More advanced exception catching More advanced exception catching

31. 31 Questions and Contact Info Joel Henry Joel Henry henryj@cs.umt.edu henryj@cs.umt.edu henryj@cs.umt.edu MATT, RATT, and GIST MATT, RATT, and GIST http://www.cs.umt.edu/RTSL/matt/ http://www.cs.umt.edu/RTSL/matt/ MATLAB and Simulink users MATLAB and Simulink users