Using Symbolic PathFinder at NASA Corina Pãsãreanu Carnegie Mellon/NASA Ames

Introduction Modern software Modern software Multi-core, cloud-based, cyber-physical Multi-core, cloud-based, cyber-physical Needs to be highly reliable and secure Needs to be highly reliable and secure Testing Testing Most widely used Most widely used May miss important errors May miss important errors Formal analysis Formal analysis theorem proving, model checking, symbolic execution, static analysis, etc. theorem proving, model checking, symbolic execution, static analysis, etc. Strong guarantees of reliability and safety Strong guarantees of reliability and safety Goal Goal Use formal techniques to make testing more effective Use formal techniques to make testing more effective

Symbolic PathFinder (SPF) Performs Performs Automated test case generation Automated test case generation Property checking Property checking Integrates Integrates Symbolic execution, model checking and constraint solving Symbolic execution, model checking and constraint solving Uses analysis engine of Ames’ JPF tool Uses analysis engine of Ames’ JPF tool Test case generation – the easiest path towards the infusion of advanced formal verification technologies JPF

Uses of SPF Analysis/test generation for Analysis/test generation for Byte-code Byte-code Models (Simulink/Stateflow, UML) Models (Simulink/Stateflow, UML) Translation into Java Translation into Java Polyglot Framework – w/ U. Vanderbilt, U. Minnesota Polyglot Framework – w/ U. Vanderbilt, U. Minnesota SPF handles SPF handles Pre-conditions, data structures, concurrency Pre-conditions, data structures, concurrency User customizes User customizes Coverage, search strategy, output format Coverage, search strategy, output format Used at NASA, Fujitsu, academia Used at NASA, Fujitsu, academia Orion orbits the moon (Image Credit: Lockheed Martin) Polyglot

Experience: Onboard Abort Executive (OAE) Prototype for CEV ascent abort handling being developed by JSC GN&C Inputs Pick Highest Ranked Abort Checks Flight Rules to see if an abort must occur Select Feasible Aborts OAE StructureResults Baseline Baseline Manual testing: time consuming Manual testing: time consuming Guided random testing could not cover all aborts Guided random testing could not cover all aborts Symbolic PathFinder Symbolic PathFinder Generates tests to cover all aborts and flight rules Generates tests to cover all aborts and flight rules Found major bug in new version of OAE Found major bug in new version of OAE Environment constraints Environment constraints Input data is constrained by environment/physical laws Input data is constrained by environment/physical laws Example: inertial velocity can not be ft/s when the geodetic altitude is 0 ft Need to encode these environment constraints explicitly Need to encode these environment constraints explicitly Solution Solution Use simulation runs and machine learning to get data correlations Use simulation runs and machine learning to get data correlations

Lessons Learned Not enough to generate test cases that achieve good coverage Not enough to generate test cases that achieve good coverage Should also make sense to developers Should also make sense to developers Environment assumptions need to be made explicit Environment assumptions need to be made explicit Using both input from the user and automated inference techniques Using both input from the user and automated inference techniques True for any analysis tool that can only analyze components (not entire systems) True for any analysis tool that can only analyze components (not entire systems)

Usability Challenges Tool Usability Tool Usability Research Challenges Research Challenges Domain Specific Domain Specific

Tool Usability Easy set-up -- Not true for JPF/SPF Easy set-up -- Not true for JPF/SPF Portability for multiple platforms Portability for multiple platforms Animation and/or graphical display of results Animation and/or graphical display of results Intermediate results Intermediate results Important for tools that do not give “instant” results like SPF Important for tools that do not give “instant” results like SPF Informative counterexamples/Explanation of errors Informative counterexamples/Explanation of errors Documentation easily accessible on-line -- JPF’s wiki Documentation easily accessible on-line -- JPF’s wiki Integration with other tools familiar with developers Integration with other tools familiar with developers JPF’s Eclipse plug-in; SPF outputs JUnit tests and Antares scripts JPF’s Eclipse plug-in; SPF outputs JUnit tests and Antares scripts Clear display of different options --Not true for JPF/SPF Clear display of different options --Not true for JPF/SPF Availability --JPF is open-source Availability --JPF is open-source …

Research Challenges Scalable and timely verification Scalable and timely verification Analysis as efficient as possible Analysis as efficient as possible Parallelization, compositional techniques, fast constraint solvers Parallelization, compositional techniques, fast constraint solvers New constraint solvers New constraint solvers Solve complex, non-linear constraints (NASA applications) Solve complex, non-linear constraints (NASA applications) Solve strings combined with numeric constraints (web applications, security) Solve strings combined with numeric constraints (web applications, security) Automatic discovery of interfaces Automatic discovery of interfaces With (minimal) inputs from user With (minimal) inputs from user Combinations of automata and machine learning techniques Combinations of automata and machine learning techniques …

Domain Specific Knowledge Add the user to the picture Add the user to the picture Integration of domain specific knowledge Integration of domain specific knowledge Adaptation of tools to users’; specific needs Adaptation of tools to users’; specific needs E.g. NASA uses models; adapt SPF from code to models; customize search strategies/output format E.g. NASA uses models; adapt SPF from code to models; customize search strategies/output format …

?