1. Hosted by: Institute for Software Integrated Systems (ISIS) Vanderbilt University Software Reliability for FCS Discussion Format May 18-19, 2004 ARO Workshop

2. Janos Sztipanovits 2 Topic Leads Topic 1: Gabor Karsai Topic 2: Edward Lee Topic 3: Doug Schmidt Topic 4: Dave Sharp Topic 5: John Goodenough

3. Janos Sztipanovits 3 Scope  Networked System of Systems  Development timeframe is between 2004-2015

4. Janos Sztipanovits 4 Format  Session Lead presents: – Objectives for the session – Format to be followed (Time and charts allowed) – Key questions to address – Sets the stage and presents an introduction – Summarizes the main outcome of the session  Subset of Participants present their viewpoints  Group questions to elicit broader discussion among all participants GOAL is to have each session (topic) be very interactive with all participants

5. Janos Sztipanovits 5 Topic 1: Emerging paradigms for improvements of software reliability  What are the novel technologies that improve software reliability? – Recovery-oriented Computing? – Autonomic Computing? – QoS in middleware?  How to use (model-based) analysis? – How to construct models that can be used in system verification, but are also abstractions of code? – How to verify (or predict) dependability? – How to build models for legacy codes?  How to build contingency management into software? – How to detect faults?…mitigate their effects? – How to engineer software “health management”?  How do paradigms scale up? – How to build a system of systems that is more reliable than the “sum” of its parts?

6. Janos Sztipanovits 6 Topic 2: Balance between formal and informal methods, engineering and artistry, evolution and rebuild  Is the objective of formal methods proof or understanding? – proof for certification? – proof for identifying overlooked behaviors? – does the approach change if the emphasis is different? – relationship with testing?  How should models relate to programs? – are models the source code or abstractions of the source? – do they represent individual traces or families of behaviors?  Are non-procedural languages viable? Will they help? – declarative? concurrent? constraints? – value of raising the level of abstraction? enabling rebuild? – can we get better formal properties? – acceptance?  Do visual notations help? Which ones help most? – Simulink? Statecharts? – UML static structure? UML use case? UML...?  Are source code annotations acceptable? useful?

7. Janos Sztipanovits 7 Topic 3. The role of open standards, open- source development, end-user programming and other radically different models  What are the good candidates for open standards? – how to identify them? – how to balance business interest and technical rational? – how to avoid the design by committee syndrome?  What is the role of open standards in software reliability? – are there standards for reliability? – do standards improve reliability?  Is there relationship between open-source and software reliability? – what are the technical arguments? – what are the business arguments? – what are the social arguments?  What program development models increase reliability? – process-oriented methods? – extreme programming? end-user programming? others?

8. Janos Sztipanovits 8 Topic 4: Effects of software certification on the current balance between software/system modeling, analysis and testing  Given that certification processes require the same degree of confidence in all evidence used therein – E.g., when less direct (e.g. non-system-testing) methods are to be used for certification:  Any models, modeling tools, analysis tools, etc, must be certified to the same level as the underlying system, and  Fidelity of any system representations must be similarly certified for all applicable aspects (e.g. functionality, timing, concurrency)  …What unique approaches, if any, can more efficiently certify tools and representations than the system itself?  …What techniques are available which compellingly reduce the aggregate certification effort?  …What aspects are most amenable to these approaches and techniques? – Can techniques address unique challenges result from mobile ad hoc networking or other system of system dynamics?  …Are these approaches and techniques scalable to the size of FCS?

9. Janos Sztipanovits 9 Topic 5: Reliability metrics for real-time, embedded software for FCS  Are reliability metrics for real-time embedded software different than for other kinds of FCS software?  How should the contribution of software to overall system reliability be defined and measured? Is it reasonable to say R sys = R HW  R SW ?  What process should be used to confirm that FCS software satisfies high requirements for reliability and availability? – what combination of testing and other analysis? – how should testing/analysis results be combined into a summary statement of reliability? – how should field experience be used to establish a software reliability estimate?