1. Evaluating a Formal Methods Technique via Student Assessed Exercises Alastair Donaldson, Alice Miller University of Glasgow

2. 13/07/2015FM-Ed2 Outline  Need for evaluation  SymmExtractor  Examples for evaluation – student solutions  Ethical approval  Documentation process  Evaluation results  Future evaluation

3. 13/07/2015FM-Ed3 Need for evaluation  Automated FM tries to solve intractable or undecidable problems Model checking – quickly becomes intractable Parameterised model checking – undecidable  Progress made by restricting application domain “Applicable to C programs without pointers” “The system must have a fixed no. of components”

4. 13/07/2015FM-Ed4 Need for evaluation  Is restricted application domain still useful?  Need evaluation with users of technique Can tool do what they want? Can they change needs easily to fit technique?  Example: symmetry reduction for model checking Automatic symmetry detection Exploiting symmetry Both computationally difficult Both easy to solve when application domain limited

5. 13/07/2015FM-Ed5 Symmetry reduction for model checking  Replication in topology of concurrent system → replication (symmetry) in state-space  State space partitioned into equivalence classes Only need to search one state per class  System comprised of n components Equivalence classes may be as large as n!  Model checking is automatic: Symmetry must be automatically detected

6. 13/07/2015FM-Ed6 SymmExtractor  Detects symmetry in Promela specifications, for verification with SPIN  Extracts static channel diagram of a specification  Computes symmetries of static channel diagram  Derives state-space symmetries from these  Specification must satisfy certain restrictions Need evaluation to see how restrictions affect applicability of SymmExtractor

7. 13/07/2015FM-Ed7 Examples for evaluation: submissions to student assessed exercise  Modelling reactive systems Final year FM course at Glasgow Main focus: model checking with SPIN  Assessed exercise 2004/2005 Specification and verification of (3 versions of) a 2-user telephone exchange  Intuitively, underlying state spaces should exhibit one non-trivial symmetry  Can SymmExtractor detect this?

8. 13/07/2015FM-Ed8 Ethical approval  Followed Glasgow Ethics Code and gained ethical approval from faculty  Obtained signed consent forms from all participating students  Ensured evaluation took place after formal assessment of submissions  17 (out of 35) students gave approval 51 Promela specifications for input to SymmExtractor

9. 13/07/2015FM-Ed9 Documentation process  For each specification, documented Size of unreduced state-space (SPIN) State-space symmetries computed explicitly (SPIN-to- GRAPE) Symmetry breaking features (experimenter) Violations of SymmExtractor’s restrictions (SymmExtractor) Modifications required to fix violations (experimenter) Symmetries computed by SymmExtractor (SymmExtractor) Size of quotient state space (TopSPIN)

10. 13/07/2015FM-Ed10 Results  Approx. half specifications had symmetry breaking features Set of modelling guidelines to avoid common pitfalls  After fixing these: 23 specifications – symmetry detected 13 specifications – violated restrictions, needed minor modification for symmetry to be detected 7 specifications – medium modifications 8 specifications – major modifications

11. 13/07/2015FM-Ed11 Results  Minor modifications – violation of restrictions which could easily be lifted  Medium modifications – problems due to use of global variables, which SymmExtractor could be modified to cope with  Major modifications – problems involving way arrays indexed by process identifiers are accessed Serious usability problem due to restrictions requires further research effort to fix

12. 13/07/2015FM-Ed12 Future evaluation  Benefit here was one-way: students’ assessments used to aid our research  Evaluation took place after completion of course  May be possible to run evaluations during the course Students apply symmetry detection/reduction to own programs and report results