The Automated Refinement of a Requirements Domain Theory Lee McCluskey co-researchers: Margaret West Beth Richardson Department of Computing and Mathematical Sciences

School of Computing and Mathematics, University of Huddersfield Talk Outline PART 1. Introduction to the - ATC application - The Conflict Prediction Specification (CPS) - CPS tools environment PART 2. Research into the Theory Revision Tool reference: Journal of Automated Software Engineering, Special Issue on Inductive Logic Programming, Spring 2001.

School of Computing and Mathematics, University of Huddersfield FAROAS - A Case Study involving Aircraft Separation Criteria Shanwick Oceanic Area segment1 segment2

School of Computing and Mathematics, University of Huddersfield Example of Separation Requirement Paragraph of the Manual of Air Traffic Services, Part 2, Section 3 - separation standards states: “For subsonic aircraft, the minimum longitudinal separation between turbojet aircraft, meeting the MNPS, and operating wholly or partly in MNPS airspace, shall 10 minutes, provided that …….ETC”

School of Computing and Mathematics, University of Huddersfield The FAROAS Project - Results (contract research from NATS Ltd): n encoded part of the requirements of a system that is to maintain separation between aircraft over the Atlantic Ocean in an expressive, structured logic n The kernel of this specification was written in about 500 logic axioms and is called the CPS n A validation environment was built around the CPS and helped “debug” it

School of Computing and Mathematics, University of Huddersfield CPS Auto-generated CPS lp [(one_or_both_of Segment1 and Segment2 are_flown_at_subsonic_speed) & ( the_Aircraft_on(Segment1) and the_Aircraft_on(Segment2) meet_mnps) & ( the_Aircraft_on(Segment1) and the_Aircraft_on(Segment2) are_jets & (the_Profile_containing(Segment1) & the_Profile_containing(Segment2) are_wholly_or_partly_in_the_ mnps_airspace) ] => [(the_basic_min_longitudinal_sep _Val_in_mins_required_for Segment1 and Segment2) = 10 …. ETC the_basic_min_longitudinal_sep_Val_ in_mins_required_for(Segment1,Seg ment2,10):- are_subject_to_oceanic_cpr(Segmen t1,Segment2), both_are_flown_at_supersonic_spee d(Segment1,Segment2), (both_are_flown_at_the_same_mach_nu mber_in_level_flight(Segment1,Seg ment2) ; the_Aircraft_on_segment(Segment1, Aircraft1), the_Type_of(Aircraft1,Type1), the_Aircraft_on_segment(Segment2, Aircraft2), the_Type_of(Aircraft2,Type2), Type1=Type2, are_cruise_climbed(Segment1,Segme nt2) ),.. ETC

School of Computing and Mathematics, University of Huddersfield CPS Expert Visual Inspection Automated Syntax Checking Automated Reasoning Automated Translation to Executable Software Batch Testing Simulation -an ATC requirements statement DETECT BUGS Opportunities for bug detection in a Formal Specification

School of Computing and Mathematics, University of Huddersfield One Major Outcome of FAROAS n Validation and Maintenance of Complex Models (Ontologies? Domain theories? Formal Specifications? KBs?) require automated tool support to identify bugs and help remove them. n Such “models” are not written like programs to allow systematic testing but are designed to decrease the semantic gap between model and what is modelled.

School of Computing and Mathematics, University of Huddersfield IMPRESS (EPSRC / NATS Ltd 96-98) IDEAS: From the Formal Specification standpoint: n The CPS is a “high level” specification - a kind of requirements domain theory - why not use theory revision (or other techniques from ML) to try to help improve the theory. From the ML standpoint: n The “fielding” of ML techniques is of great interest to the ML community.

School of Computing and Mathematics, University of Huddersfield Machine Learning (ML) The investigation and construction of systems which refine existing knowledge and/or acquire new knowledge One way for learning to take place is by feeding a system (performance component) with training examples, and letting a learning component use the results to improve the system’s behaviour.

School of Computing and Mathematics, University of Huddersfield Training Examples Performance Component Results + Execution Trace Analysis Blame and Credit Assignment Construct Refinements Example Architecture of a Learning System

School of Computing and Mathematics, University of Huddersfield Tests in Enveloped Form CPS in Enveloped Form Results+ Proof Trees Proof Tree Analysis Blame Assignment Inductive Refinement Algorithm Abstract Architecture of the CPS’s learning tool Meta- Interpreter

School of Computing and Mathematics, University of Huddersfield CPS: Many-Sorted Logic Specification + tests + queries PARSER + TRANSLATORS CPS Grammar CPS -logic program CPS - structured English Test Harness CPS ENVIRON- MENT: ABSTRACT VIEW Tests in Prolog TEST RESULTS Enveloped Logic Program and Tests Oracle Envelope Theory Revision CPS Refinements html

School of Computing and Mathematics, University of Huddersfield PROBLEMS! blame assignment: mark those clauses that take part in faulty proof trees - use a statistical measure to pick out those most likely to be faulty BUT operational version of CPS far from pure clausal form - e.g. with or's and not's ‘not’ a particular problem as it changes the ‘polarity’ of the proof tree..

School of Computing and Mathematics, University of Huddersfield PROBLEMS! clause revision:- use a hill climbing technique to help find appropriate revisions BUT any kind of conventional TR algorithm seemed doomed to complexity problems AND typical TR operators e.g. Dropping Condition may be too superficial.

School of Computing and Mathematics, University of Huddersfield Overcoming the Problems.. We explored a range of different approaches and eventually discovered a method that succeeded - it was based on TR operators that were: FOCUSSED and COMPOSITE

School of Computing and Mathematics, University of Huddersfield Focus: Ordinal Sorts.... Errors in the CPS tend to occur in complex groups of ordering relations (involving sorts like Flight Level, Time, Latitude etc ). These totally ordered sorts we call “Ordinal” - we focused on these when designing an algorithm for finding and removing errors.

School of Computing and Mathematics, University of Huddersfield Composite TR Operators We created a “refine” algorithm which - stores all the instances of clauses used in faulty proofs - chooses a clause and an ordinal variable to refine - induces “regions” of ordinal values from the variable’s values in the set of faulty instances of that clause - adds/subtracts these regions and evaluates the changes by executing the theory

School of Computing and Mathematics, University of Huddersfield Theory Refinement In: an imperfect theory T, training examples E Out: seq. of i revisions RS, updated theory T 1. i:=0; 2. repeat 2.1 call apply(T,E,Results,S0); 2.2 call blame(T,Results, RP); 2.3 call refine(T,RP,Results, R,Sm); 2.4 if = Sm > S0 then i:=i+1; T := R(T); RS[i] := R; end if until Sm =< S0 3. end

School of Computing and Mathematics, University of Huddersfield ML Example - Learning a new part of a Requirements Specification On May 10th, 1998, Margaret West working on the Impress project at Huddersfield University collected 237 examples of pairs of aircraft profiles “cleared” under the new RVSM requirements. She fed them to a learning system we had created and embedded in the CPS’s environment. By May 13th, the program had learned the general criteria for aircraft under RVSM conditions.

School of Computing and Mathematics, University of Huddersfield RVSM Example: the_min_vertical_sep_Val_in_feet_required_for(A,B,C,D,2000) :- are_subject_to_oceanic_cpr(B,D), ( the_machno_Val_on(B,H), H<1.0, the_machno_Val_on(D,G), G<1.0, (A is_above fl(290) ; C is_above fl(290)) ; ( the_machno_Val_on(B,F), F ge 1.0 ; the_machno_Val_on(D,E), E ge 1.0 ), (A is_at_or_below fl(430) ; C is_at_or_below fl(430)) ). CHANGE ADDED: not( (H ge 0.80, H le 0.86), (A is_at_or_above fl(330)), (A is_at_or_below fl(370)), (C is_at_or_above fl(330)), (C is_at_or_below fl(370)) )

School of Computing and Mathematics, University of Huddersfield Using the Theory Revision Tool: Results With one set of data, 100 per cent coverage was reached by the system making 3 revisions: 1 - a change introducing RVSM criteria, 2 - a change that removed a bug that hitherto had not been spotted 3 - a (?) meaningless change During the Impress project, the error rate of the CPS executable has decreased from hundreds of errors per 5000 tests to less than 10 errors per 5000.

School of Computing and Mathematics, University of Huddersfield TR Tool Research: Conclusions We showed the potential in using learning techniques to n find and remove bugs n help in the maintenance of a formal specification of requirements The main lessons we have learned in the fielding of a TR tool are that it should use refinement methods focused at the likely sources of error within the theory, and it should be designed so as to minimise the number of times the theory has to be applied to the whole training example set.