1. ARO Workshop Wendy Roll - May 2004 Topic 4: Effects of software certification on the current balance between software/system modeling, analysis and testing

2. ARO Workshop Wendy Roll - May 2004 2 Context  Traditional approach for certified software development has been to implement the system and then perform rigorous testing and analysis (post-creation) – Safety critical systems are designed to reduce this testing/analysis burden  Newer model-based approaches are intended to reduce errors and support analysis (pre-creation), but…  Certification processes require the same degree of confidence in all evidence used therein – E.g., when less direct methods are to be used for certification:  Any models, modeling tools, analysis tools, etc, must be certified to the same level as the underlying system, and  Fidelity of any system representations must be similarly certified for all applicable aspects (e.g. functionality, timing, concurrency) Certify?

3. ARO Workshop Wendy Roll - May 2004 3 Responses Concerned with  Applying certification criteria to larger scope of behavior – E.g. more dynamic behavior  Certification of “traditional” designs using non-system testing methods

4. ARO Workshop Wendy Roll - May 2004 4 Question 1  …What techniques are available which compellingly reduce the aggregate certification effort?  (Topic 2) – Formal Methods?  (Sylvester Fernandez) – Commoditized components may help  Issues of trust and confidence, rather than technology alone, must be addressed  (Richard Payne) – Modeling and simulation (discrete event simulation) could be used to represent the behavior of the system and this simulation could be certified in relation to requirements  (Fredrick Sheldon) – SPN/SAN Models

5. ARO Workshop Wendy Roll - May 2004 5 Question 2  …What unique approaches, if any, can more efficiently certify tools and representations than the system itself?  (Richard Payne) – Model-based testing – construct models of intended system behavior in the form of a sequence-based specification to validate system requirements are complete and correct. – Markov chain usage models- to generate a large sample of system software test cases of mission-based test scenarios – Model-based testing can be implemented at a relatively small cost compared to the overall software development and test budget as well as rework costs.

6. ARO Workshop Wendy Roll - May 2004 6 Question 3  …What aspects are most amenable to these approaches and techniques? – Can techniques address unique challenges resulting from mobile ad hoc networking or other system of system dynamics?  (Topic 2) – Concurrency – formal methods “proof of properties”  (Joe Loyall) – Increase the ability to certify dynamic systems by constraining their dynamism  Dynamically choosing from a set of certified static choices – If we can’t certify exactly correct behavior for highly dynamic systems, perhaps we can certify their limits – Can we certify the adaptive mechanisms that delimit behavior, recover, protect, or keep software operating within a “safe” subset of possibilities  (Richard Payne) – Usage modeling process aid in identifying possible combinations and configurations of the system operating in a variety of ways and documenting them explicitly in behavioral models – It can help not only to verify that the system functions correctly under dynamic conditions but also help to determine what those conditions are  (Chris Gill) – Reliable message delivery in the face of transient connectivity by exploiting motion and availability profiles

7. ARO Workshop Wendy Roll - May 2004 7 Question 4  …Are these approaches and techniques scalable to the size of FCS?  (Richard Payne) – The model-based testing approach has not been used on a program the scope of the UoA but from a modeling standpoint, it should be feasible – 10 5 - 10 6 usage states can be captured in usage models with the latest available techniques – Major challenge is providing required test capabilities for large scale, automated testing – Without a robust, semi-automated test capability, it will be very difficult to ever measure the software reliability with any level of confidence  (Chris Gill) – If we can find ways to decompose the problem