1. AC-B Air Traffic Control & Business Systems GmbH © AC-B GmbH 2008 AMHS Conformance Testing AMHS Implementation Workshop Chennai, India, 15 – 16 December 2008 Rainer Hoffmann, AC-B GmbH

2. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Presentation Contents  EUR AMHS testing strategy  AMHS conformance test environment  Test experience

3. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH  EUR AMHS testing strategy  AMHS conformance test environment  Test experience

4. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH EUR AMHS Testing Strategy  ICAO EANPG (AFSG) developed standards for AMHS testing  What exactly will be tested?  Which tests are performed?  How are tests executed (under which conditions)?  -> Make tests fair and transparent to everybody  3 Test phases are defined in the EUR AMHS Manual 1. Conformance testing 2. Interoperability testing 3. Pre-operational testing

5. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Test phases (1) 1.Conformance tests  wide test scope of standard (and optional) functions  soon after system installation (may be done in parallel with FAT/SAT) 2.Interoperability tests  limited test scope - basic and important functions  bilateral and trilateral tests  when the network connection is established 3.Pre-operational tests  a few check-ups close before cut-over  run AFTN and AMHS link in parallel

6. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Test phases (2)  examine one system against the specification (standard documents)  can be performed locally  examine two or more systems communicating with each other  distributed over two or more locations Conformance testingInteroperability testing IUT System A Test Tool IUT System A IUT System C IUT System B IUT:= Implementation Under Test   test automation possible  includes „negative“ test cases  bilateral coordination of staff presence  only „positive“ test cases

7. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Conformance testing saves overall test effort (1) Example: 59 interoperability tests with 8 partners (neighbour domains) Interoperability Tests 1 2 3 4 5 6 7 8 Number of tests 59 18 In this example 472 tests would need to be coordinated and performed

8. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Conformance testing saves overall test effort (2) Interoperability Tests 1 2 3 4 5 6 7 8 Conformance Tests...and reduce the number of interoperability tests Start with conformance tests... In this example the overall number has been reduced to 203 tests ( = 59 + 144 ) Interop. Tests Number of tests 59 18

9. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Test Strategy A: Perform comprehensive interoperability testing Number of test cases Test cases per day 596 Test casesTest days 2Partners11820 4Partners23639 8Partners47279 16Partners944157 40Partners2360393 Test effort comparison (1) All test cases have to be performed with each partner!

10. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Test effort comparison (2) Test Strategy B: Perform comprehensive conformance testing and a subset of interoperability testing Conformance testsInteroperability tests Number of test cases Test cases per day Number of test cases Test cases per day 5912186 Test casesTest days Test casesTest days Overall daysSavings 2Partners5953661144% 4Partners59572121757% 8Partners595144242963% 16Partners595288485366% 40Partners59572012012568%

11. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Reasons for a conformance test tool  Improved test coverage, more test capabilities  overcome scenario limitations encountered in the FIRST interoperability trials  enforce fault situations (= negative testing) to check IUT robustness, e.g. erroneous AMHS parameters  Better fault analysis  test tool can analyse captured AMHS messages, reports and probes  test tool can act as a reference system  Easier regression testing  less manual interaction, tests can be repeated very exactly (and quickly)  no need to use AMHS links to neighbour domains to verify your software updates

12. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH  EUR AMHS testing strategy  AMHS conformance test environment  Test experience

13. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AMHS Test Tool Concept  Build a test tool with all necessary components to simulate an operational AMHS environment of a COM centre  Provide a test platform for loadable test scenarios  modification or enhancements of standard tests  reuse of test messages and test sequences  easy creation of new tests  Minimise manual intervention by test automation

14. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH COM Centre AFTN/AMHS Gateway ATS Message Server (MTA) Generic COM-Center Environment (AMHS view) Local AFTN User Terminal Monitor & Control Position AFTN Environment AFTN User (s) AFTN User (1) AFTN User (2) AFTN Center AFTN (CIDIN) interface AMHS Environment Local AMHS User Agent (UA) User Agents (UA) UA 1 UA 2 UA 3 UA 4 UA m UA o AMHS (P1) interfaces Message Transfer Agents (MTA) MTA-1 MTA-n MTA-2

15. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AFTN/AMHS Gateway ATS Message Server (MTA) AMHS Test Interfaces Local AFTN User Terminal Monitor & Control Position AFTN Test Environment AFTN (CIDIN) interface AMHS Test Environment AMHS (P1) interfaces Local AMHS User Agent (UA) IUT Test MTA-1 Test MTA-3 Test MTA-2 AFTN Test Appl. Test Interfaces

16. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AMHS Test Tool AFTN/AMHS Gateway ATS Message Server (MTA) AMHS Test Tool Architecture Local AFTN User Terminal Monitor & Control Position AFTN Test Appl. AFTN (CIDIN) interface AMHS (P1) interfaces Test MTA-1 Test MTA-3 Test MTA-2 Local AMHS User Agent (UA) IUT Testlog Message Source/Sink Viewer/ Editor User Interface Test Control and Evaluation Application Test Interfaces

17. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Features of AC-B‘s AMHS Test Tool  Test interfaces  AMHS over TCP/IP  AMHS over ATN (option)  AFTN over CIDIN/X.25 PVC  AFTN over X.25 (option)  AFTN over TCP/IP (option)  Test cases  59 conformance test scenarios according to EUR AMHS Manual (App. D)  More than 200 prepared AMHS and AFTN test messages  Functions  Generation of IPMs, IPNs, reports, probes with user-defined parameters  Verification of IPMs, IPNs, reports, probes received from the IUT  Automatic response to received messages and probes: DR, NDR, RN, NRN

18. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AMHS Conformance Test Scope  Test groups  Submission operations  Delivery operations  Transfer operations  Gateway operations  AMHS -> AFTN  AFTN -> AMHS  Test classes  Verification of normal communication  Handling of communication errors (negative testing – robustness tests)

19. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AMHS Test Tool - HMI examples (1) Table tree view for test steps Details for a selected test stepRun control buttons Indication of test step success / failure

20. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH IPM AMHS Test Tool - HMI examples (2) Test Control and Evaluation Application AMHS Viewer/Editor MTE IPN Probe Report

21. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AC-B‘s AMHS Test Services  AC-B offers AMHS test services to ANSPs and AMHS suppliers  The basic AMHS conformance test service package includes  preparation of the AMHS test tool for the on-site test  on-site test session to run a set of standardized AMHS conformance test procedures  test evaluation and test report  test certificate (= conformance test statement)  Customized test service packages can be offered  to perform regression testing, or  to verify optional, customer-specific AMHS features

22. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH  EUR AMHS testing strategy  AMHS conformance test environment  Test experience

23. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AMHS conformance test sessions  9 conformance test sessions in 2006, 2007 and 2008  5 initial test session and 4 regression test  regression tests were due to  normal (planned) product release upgrades, or  unsatisfying initial test results  5 AMHS systems were tested  DFS (Germany), Austrocontrol (Austria), MATS (Malta), AFSBw (German Military), NAVIAIR (Denmark)  These 5 systems came from 5 different suppliers

24. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH AMHS conformance test statistics

25. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Experiences made  The first (initial) test of a product takes time  Connection set-up is never easy b e p a t i e n t... !  Inital test runs never flow smoothly do not get frustrated!  A renewed (second) test run can be done very efficiently (even in one day)  Some major problems that occured were not specific to the AMHS standard  When we defined AMHS tests, we believed that base X.400 MHS products are mature that was wrong!  Tests revealed problems also on X.400 level and even ISO transport layer s u r p r i se ?

26. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Conclusions and recommendations (2)  Conformance testing is only one step in a series of tests  stepwise testing at supplier and customer premises required  conformance tests do not replace FAT/SAT and are no guarantee for perfect AMHS operation (due to limited scope and specific test parameters)  The system‘s configuration is an issue  the conformance test environment uses test parameters, e.g. for mta name, domain names, AFTN and AMHS addresses  later configuration changes may result in different behaviour (e.g. gateway use of address mapping tables)  configuration control is the supplier‘s responsibility  Recommendation:  Perform the official conformance tests at the target system, best at ANSPs premises

27. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Conclusions and recommendations (2)  Today‘s standardized conformance tests are absolutely necessary but not necessarily sufficient  In many cases non-conformant behaviour was detected by the test tool and the software could be updated (improved) before the official test run  but...  test scope limited mainly to AFTN/AMHS gateway and MTA  only few MTA submission and delivery tests  no standardized test for UA-MS/MTA and UA-UA communication  Recommendation:  Enhance the scope of standardized conformance tests

28. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH Invitation  Please consider AMHS support from AC-B in your implementation projects  AMHS Conformance Test Services  AMHS Seminars  The AMHS Book

29. © AC-B GmbH 2008 AC-B Air Traffic Control & Business Systems GmbH www.ac-b.de/amhs