1. 1 Title slide Strategic Direction for Functional Test Automation Soft Test 2005 Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD

2. 2 AGENDA Future for Functional Testing History of Functional Automation Strategic Direction Automation Frameworks in Action Impact upon You?

3. 3 Future for Functional Testing

4. 4 Increased Demand………..? Global development, with large projects having multi site, multi geography and multi suppliers to contend with. Corporate and regulatory requirements growing all the time Business demands new applications, faster and cheaper to obtain competitive advantage Closer alignment of IT and business to repair lack of business confidence

5. 5..Or, increased pressures? Automation frameworks and greater business knowledge requirements, mean reduced numbers of traditional functional testers Agile development methods mean developers undertake more unit and component testing Growth of outsourced testing to different geographies, leading to greater competition for the roles Is the role of the functional tester (who is neither technical or business specialist) dead?

6. 6 History of Functional Automation

7. 7 Record and Playback “all you need to do…….…!” Easy to record the scripts Extremely fragile Expensive to maintain On average, each test run requires at least 50% of the scripts to be recorded again Please tell me no one does this anymore!!!

8. 8 Scripting Use the scripting language to write scripts that do what you want Build in as much robustness as possible But, you’re building an application to test an application, which also needs testing! Maintenance costs can be very high Needs programmer skills Cost of implementation not affordable at project level

9. 9 Table Driven Like scripting but more flexible and greater re-use Remove items from the script that change, e.g. data Reduces maintenance costs a bit Implementation costs about the same Still requires specialist skills to implement

10. 10 Functional Test Automation is Broken! Focus on technology rather than business needs 80% of functional testing still manual 60% to 70% of automation tools used for non- functional testing Typically, traditional functional automation stops at 100 scripts, regardless of test coverage requirement Critical factors; cost of implementation and maintenance, skills required and asset sharing over different technologies

11. 11 Strategic Direction

12. 12 Automation Frameworks Wouldn’t it be good if…….. – Tests could be documented in common format, regardless of whether they are manual or automated – The format for the tests resulted in quicker preparation time than traditional manual tests – Both developers and business testers could understand and use the same test format – Script maintenance was no longer a requirement – Tests could use any of the test execution tools required by the underlying technology Faster planning, faster execution, and far less technical skill required

13. 13 Business Analysts already using them, and use will grow Home grown frameworks built within organisations to meet business demands Niche suppliers providing frameworks; try a Google search – 512,000 hits this week – Seen a handful that appear mature – Latest review by Paul Herzlich (OVUM analyst) Market Leaders such as Mercury developing Business Process Tester (BPT) This is the industry direction now

14. 14 Automation Frameworks in Action

15. 15 The project Lloyds TSB Offshore in Channel Islands Migration from current banking platform IBIS to Temenos Globus G13 Tight deadlines due to EU Savings Tax Directive coming into force on 1 st July 2005 Business processes required additional Globus modules Multiple releases leading up to deadline required substantial testing SQS-UK contacted to provide consultancy

16. 16 The Team SQS-UK recognised the requirement for test automation Joint team from Odin Technology and SQS-UK assembled Working alongside Lloyds TSB staff Odin Technology provided testing technology SQS-UK provided implementation resource and testing know-how

17. 17 Project Approach Split project roles for Automation Technicians and Testers – Three phase technical approach Tool Evaluation & Technical Feasibility Technical Implementation Support – Test Analysis and Design throughout Tester Technician Tool Evaluation Technical Implementation Support Test Analysis & Design Project Time

18. 18 Project Phase 1: Tool Evaluation and Technical Feasibility

19. 19 Tool Evaluation Automation isn’t always straightforward! GUI Tools don’t work “out of the box” for all environments They will almost certainly require some custom functionality to interact with the SUT The Evaluation process: – Assess what custom functionality is required – Assess what involvement is required from an Automation Technician to provide the custom functionality – Assess if there are any other efficiency improvements that can be made through application of technology This provides the scope of technical implementation phase

20. 20 Project Phase 2: What would normally happen...

21. 21 Typical Automation Project Take existing Manual Scripts Rework by hand into automated scripts & function libraries The bulk of the work… Building the Object Map – Providing the Mapping between the business terms used to describe the SUT Interface and the test tools requirements Coding Test Scripts – Providing the instructions and associated data for the test tool to interact with the SUT and perform testing

22. 22 Object Mapping Providing the Mapping between the business terms used to describe the SUT Interface and the test tools’ requirements Application

23. 23 Example Manual Tester Test Tool “The Login Window” class: MSWDialog label: “Login.*”

24. 24 Example 2 Manual Tester Test Tool “Username” class: HTMLEdit id: Lgn_Uid

25. 25 Object Mapping in a typical Automation Tool Usually implemented under different names in different tools – GUI Map – Object Repository – Object Map Mapping an Object – Start the Object recognition tool – Point at the Object with the Mouse Pointer – Select the Object – Give the Object a meaningful business name – Make manual amendments to the properties if required

26. 26 Test Script Creation Providing the instructions and associated data for the test tool to interact with the SUT and perform testing AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1

27. 27 Scripting Approach Use the Tools Scripting Language to construct business processes Function Login(ByVal UID As String, ByVal PWD As String) Dim rc rc = wait_window(“Login”,30) if rc <> 0 Then log_msg(“Window Not Found”, FALSE) Return End if focus_window(“Login”) click_on_text (“UserName”) type(UID & “{Return}”) click_on_text (“UserName”) type(PWD & “{Return}”) End Function

28. 28 System Under Test Test Automation Tool AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then Exit Run(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc= AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then Exit Run(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc= AxeMainAPI.TestAbort Script 1.1 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use Script 1.1 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use Script 1.1 1. Open Application 2. Enter Userid for Authorised User 3. Enter Password for User 4. Click Login 5. Select Create new user from menu 6. Enter postcode and house number 7. Tick “Secure” user authority 8. Enter unique password 9. Hit Generate userid 10. Validate userid is 8chars long 11. Write down userid for later use Script 1.1 Reworking of Manual Scripts Application Object Maps

29. 29 Project Phase 2: What was done differently… Taught an old dog some new tricks…

30. 30 Test Design Model A simple structured way of defining tests that is independent of the interface and the means of execution

31. 31 Features of the Test Design Model Easy to design tests without technical knowledge Well structured Can be used to describe tests for any interface – Graphical User Interfaces (GUI) – Non-UI Components (WebServices, APIs etc.) Independent of Execution Mechanism – Manual testing – Automated testing Not theory - proven 7 year history of implementations In use in UK and Europe at many organisations for defining testing

32. 32 Test Design Model Test Test steps A single test step Object Action Data Sub-tests

33. 33 Test Model – GUI Example Test Login – enter user credentials Product search – enter product ID Product details – validate product info Logout Sub-test Login as user 1 Step Enter Username “jsmith”

34. 34 A test step Object + Action + Data Action – SET enter data, navigation – GETretrieve data – VALcompare expected/actual data

35. 35 How tests are designed Using Microsoft Excel Business processes are modelled Sequences of steps and data are built up in Sub-Test Tables Sequences of Sub-Tests model business processes

36. 36 Interpreting the Model There are products that can process the model – To generate Test Automation Scripts For UI Testing using GUI Automation tools For Non-UI Testing using Harnesses – To generate documentation for manual testing

37. 37 Interpreting the Model Manual ScriptsNon-UI Automated Scripts Automated Scripts

38. 38 Globus A Self Descriptive Application

39. 39 What is a Self-Descriptive Application? An application that, through an automated process, can provide a description of its interfaces Application

40. 40 The Globus User Interface How it works Globus Desktop Application A description of the interface for an application is stored in the database The Globus Desktop interprets the description and dynamically creates the user interface Windows and objects are presented to the user

41. 41 How Globus Describes its Interface Custom Interpreter Globus Desktop Application API API can be invoked to provide the description without presenting it

42. 42 A Sample of Globus Information APP: CUSTOMER_INST FIELDCUST_SNAME TYPEINPUT LABELSurname LENGTH20 SEQ1 FIELDCUST_GEN TYPEOPTION LABELGender LENGTH1 SEQ7

43. 43 How this Information was used The Description gives us 3 things: – The business names used for the objects – The properties the test tool required for object identification – The sequence of the objects in the window This could provide us with two things automatically: – The Object Map – The sequence of steps for our TDM sub-tests

44. 44 More on Self-Descriptive Apps Requirement on development to “name” objects in business terms Microsoft.NET Applications Windows Forms and Web Applications can provide us with – Object Maps – Sub-tests for the Test Design Model This can drastically reduce the cost of automation

45. 45 Axe Microsoft Excel Globus AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 AxeMainAPI.TestBegin "6" rc=AxeMainAPI.CheckDependency("5") If rc <> 0 Then ExitRun(rc) End If AxeMainAPI.BasestateBegin "Home" Browser("Browser"). Basestate(AXEDIR & "samples\OdinPortal\html\index.html") AxeMainAPI.BasestateEnd rc,"","" if rc <> 0 then rc=AxeMainAPI.TestAbort Script 1.1 Test Design Model & Object Maps Automation Scripts Overall - The Test System Design

46. 46 Project Phase 2: Implementation

47. 47 The Implementation Technical Implementation – 9 days of Test Tool Technician Using Automatic Interface Description in Globus – 1800+ Objects Mapped automatically – 90 Sub-test components generated 30-75 steps per Sub-test – Time to Generate: 42 seconds

48. 48 What was left for the Testers? Test Design! – Designing scenarios for testing the application – Providing Test Data – Sequencing Sub-Tests to form business processes

49. 49 Designing the Tests 4 Testers – Using the Test Design Model No previous automation experience 40 Days Elapsed time 1012 Complex Banking Scenarios Automated

50. 50 Raw Statistics – Test Execution Single Script – Automated = 80secs – Manual = 480secs – Automated is 6x Faster Full Regression Pack – Automated = 22 Hrs (1 Workstation) – = 5.5 Hrs (4 Workstations) – Manual = 132 Hrs – = 22 Man/Days (using 6hrs productivity per day) 22 Man/Days Testing in 5.5 Hrs.

51. 51 Impact Upon You

52. 52 Recommendations Find out your organisations direction Decide upon your own future 1. Increased management skills 2. Increased non-functional skills 3. Increased business skills Ensure you understand how Automation Frameworks support testing so you have a view when asked, because you will be….!

53. 53 Closing slide Thank You Susan Windsor WMHL Consulting Limited, MD susan@wmhl.co.uk www.wmhl.co.uk Case Study provided by Odin Technology Ltd www.odin.co.uk