California Institute of Technology Requirements Decomposition Analysis, Model Based Testing of Sequential Code Properties Allen P. Nikora, John D. Powell Jet Propulsion Laboratory, California Institute of Technology Pasadena, CA John D. The work described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology. This work is sponsored by the National Aeronautics and Space Administration’s Office of Safety and Mission Assurance under the NASA Software Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance Technology Program Office (ATPO).

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium2 Requirements Decomposition Analysis Task Description Requirements Decomposition Analysis n Problem Statement: Requirements play a pivotal role in planning, selection, development, testing and operation of NASA's missions. Starting from mission objectives, requirements are successively decomposed. The correctness of this decomposition is critical, yet V&V of this crucial step is limited to manual inspection and pointwise testing, which are cumbersome and fallible (e.g., Mars Polar Lander). n Task: Rigorous lightweight analysis methods for requirements decomposition have been developed by the software engineering research community, and have shown promise in successful application to critical systems (e.g., rail transportation). We study their application to the V&V of spacecraft software requirements, to ascertain if, when and how they are suitable for use by NASA.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium3 Requirements Decomposition Analysis Task Description (cont’d) Model Based Testing of Sequential Code Properties n Problem Statement: A common issue with model checking is that the state space to be explored can become too large to search in a reasonable time. This may prevent the application of model checking techniques in situations where it may be desirable to do so. Reducing the number of states visited would make it possible to apply model checking to larger systems. n Task: Recent results indicate that checking a small number of states in a system model is as effective as checking a large number of states. We explore this conjecture by comparing the results of applying traditional model checkers, such as SPIN, to the results of applying the LURCH tool, which does not visit all of the system`s states.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium4 Requirements Decomposition Analysis Goals and Objectives Requirements Decomposition Analysis n Goal: study the applicability to NASA spacecraft requirements of rigorous analysis methods for requirements decomposition that have been developed by the software engineering research community. n Objectives: 1. Manually apply decomposition analysis methods applied to spacecraft requirements. 2. Based on the results of of these application studies, emerge with recommendations for the application of these methods, identify needed extensions to those methods, and indicate the opportunities for their support (e.g., via checklists, procedures and/or tool support). 3. Develop the most promising support approaches identified by the first phase to make them suitable for application to NASA's spacecraft requirements.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium5 Requirements Decomposition Analysis Goals and Objectives (cont’d) Model Based Testing of Sequential Code Properties n Goal: Determine whether reduced-state model checking can be as effective as traditional model checking n Objectives: 1. Manually translate an existing formal model of a JPL system into LURCH  MER Arbiter 2. Apply LURCH and traditional model checking techniques (i.e., SPIN) to the models 3. Compare the effectiveness of LURCH and SPIN in identifying errors in the models.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium6 Requirements Decomposition Analysis Importance and Benefits n As the complexity of spacecraft systems increases, and as increasing reliance is placed on software as an enhancing or enabling technology, the need for analysis of requirements decomposition is expected to grow further. n Improved methods for assuring correctness of requirements decomposition advance the state of the practice in software V&V. n Early detection of flaws in requirements decomposition permits early-lifecycle repair, thus avoiding costlier downstream repairs. n Reducing the state space required for model checking will enable analytical verification of larger systems with less effort.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium7 Requirements Decomposition Analysis Relevance to NASA n The study uses requirements of actual NASA spacecraft u The New Millennium Program ST6 Autonomous Rendezvous eXperiment (ARX)  Purpose: demonstrate autonomous rendezvous between the spacecraft and a passive in-orbit payload  Key technology demonstration in preparation for the Mars Sample Return mission  Mission requirements decompose into software requirements which must control the laser rangefinder, the on-board calculation of trajectory maneuvers, the commanding of the propulsion system, and the orchestration of data downlinks to report the mission's results to Earth.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium8 Requirements Decomposition Analysis Relevance to NASA (cont’d) n The study uses requirements of actual NASA spacecraft (cont’d) u Mars Reconnaissance Orbiter  Goals: Study the history of water on Mars Become first link in “interplanetary network”  Mission requirements decompose into software requirements for typical on-board functions Commanding Data handling Navigation Attitude Control u Mars Exploration Rover arbiter (for model checking investigation)

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium9 Requirements Decomposition Analysis Highlights n Examined ST-6 Autonomous Sciencecraft Experiment requirements (approx. 9 pages) u Got a feel for the potential complexity of analyzing the decomposition of resource requirements, while working with a relatively small set of requirements (approximately 9 pages of technical detail) n Focused on the Mars Reconnaissance Orbiter requirements (approx. 1,370 in all) u Developed a means to use the project-provided traceability information to extract all the requirements that are related to a requirement of interest  WHY: Convenience & comprehension – extracts just those requirements connected, directly or indirectly, assembling the results into a (web-browser-viewable) table. The result is easier to study than following individual links within the large set of requirements, and is more focused than the “graphic mode” that the requirements tool DOORS provides. In the event of the need to make a change to a requirement, this capability has potential utility, by finding and reporting all the requirements related (directly or indirectly) to that requirement.  HOW: This is in the form of an automatic script, which takes as input the user’s identification of the requirement of interest, and returns the requirements linked to that (both “parents” of that requirement, and “children” of that requirement), the requirements linked to those requirements, etc.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium10 Requirements Decomposition Analysis Highlights (cont’d) n Focused on the Mars Reconnaissance Orbiter requirements (cont’d) u Developed a means independent of the project traceability information to extract relevant requirements.  WHY: avoids reliance on the potentially incomplete or incorrect the traceability information within the existing documentation, thus giving a means in independently assure the correctness of requirements decomposition.  HOW: text-based search for keywords (e.g., search for the word “mass” and the string “kg” – for kilograms), and regular-expression textual searches for more refined patterns (e.g., a digit, then a space, then the letters “kg”)

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium11 Query 1: yields this set of requirements Query 2: yields this set of requirements As before, for easy of viewing the results are presented in HTML tables that provide hyperlinks to the requirements themselves. Simplest example, of two result sets Example: requirements that trace to a mass-related requirement Example: requirements that involve the string Kg Result: mass-related requirements that do NOT involve the string Kg Result: requirements that do involve the string Kg but are not related to mass requirements Result: requirements that do involve the string Kg AND are related to mass requirements Requirements Decomposition Analysis Highlights (cont’d) n Trace- and string- based means to query a set of requirements have been developed. The result of such a query is a set of requirements. We have also developed capabilities to compare such result sets. Given two or more queries each of which yields a set of requirements, the capabilities developed allow the calculation of the intersection, difference and union among the several returned sets. n The simplest example, of two results sets, is shown below. The results are placed into one of three categories: occurs in only the results returned by the first query; occurs in only the results returned by the second query; occurs in both sets of results (see diagram below). More generally, for N queries, results are distributed among 2N – 1 categories.

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium12 Requirements Decomposition Analysis Highlights (cont’d) n Tools Developed u TraceRequirements.pl  Traces one or more requirements through a set of documents  For each specified requirement, parents and children are found, as well as siblings and all other possible relatives u FindPatterns.pl  A specified set of documents is searched for one or more patterns  Patterns can be specified as regular expressions u CompareResults.pl  Two or more trace and/or search results are compared to identify requirements common to the results

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium13 Model Based Testing of Sequential Code Properties Highlights Simplicity of Modeling Systems with Embedded C code n Lurch Modeling Language annotates C code to randomly exercise the code u User Modeled a system with only 15 hours training u Model allows the set of legal calling sequence u Model prohibits the set of legal calling sequence n Promela (SPIN Modeling Language) exhaustively search the model’s state space u Steep Learning Curve Semester long College course required u C code faults reported as having occurred at the model lever where the C call is made u Make debugging embedded C code hard u Confusion over whether errors are  Errors Embedding C in Promela  Errors in the imbedded C code itself

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium14 Model Based Testing of Sequential Code Properties Highlights (cont’d) Accuracy n Lurch confirmed deadlock violations found earlier with SPIN n Lurch provided easier access (than SPIN) to information about the location of C code faults u After faults repaired  The new C code is used in conjunction with the old SPIN model  The number of SPIN reported verification problems was reduced by half  SPIN subsequently ran out of memory before quitting RA-RRE Model still too big for LURCH

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium15 Model Based Testing of Sequential Code Properties Highlights (cont’d) Scalability of LURCH n LURCH scales better than SPIN over larger systems n X-axis is # of entities / size n Y-axis is Time / Memory Used

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium16 Requirements Decomposition Analysis Future Work n Reduce false positives for traces, searches n Develop guidelines for consistently expressing u Static values u Temporal properties n Natural language processing? n Integrate with measurement tools u In theory, measurable changes in requirements could be related to number of faults inserted/number of failures observed u Requires traceability to implementation

California Institute of Technology 20 July, 2004OSMA Software Assurance Symposium17 Model Based Testing of Sequential Code Properties Future Work n Automatic Generation of LURCH models from u State Charts u Source Code n Extension of LURCH for Test Case Generation n Application of LURCH as a C source code debugging agent n Application of LURCH to ongoing software projects n Future work contingent on securing additional funding u Current work performed with minimal funds (12K) and achieved  Solid value added with respect to cost (ROI)  Valuable Lessons learned  LURCH established as viable and practical testing tool u Clarified and Next Steps and for Maximizing future ROI