1. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Developing and Testing in Servers Farm Environment Gil Gattegno Feb. 2013

2. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Agenda  Introduction –Our (mutual) Problem –Our development Environment & Procedure –TestNG™ – Our Previous Testing Framework –JSystem™ – Our New Testing Framework  The Way We Were…  Out With the Old… In With the New –AnthillPro™ in a Nutshell –Nightly Build & Test4Delivery - AnthillPro behind the scenes  The Way We Were (for a very short time)  Problems We Were Facing….  Solutions –Improving Test4Delivery performance –Improving Nightly Build performance  Where We Are Now 2

3. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Our (mutual) Problem  The need to constantly improve the product quality –Deliver high quality features  The pressure to reach the deadline –Deliver features in a reasonable time  Our goal is to allow good testing coverage of the code base 3 If we aim for perfection, we may just achieve excellence http://www.testing4success.com/softwarearticles.html

4. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Our Development Environment 4  Development is in Java, C++ & Matlab ® –We’ll focus on the Java development process  Configuration management: ClearCase ®  Developers working on local snapshot views  Common code base which supports several products

5. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Our Development Procedure  Our development process includes ( besides the actual development ) : –Automatic testing prior to delivery (Test4Delivery) –Nightly build of products’ main versions  Test4Delivery is performed after feature development or bug fix on the developer stream. –If fails it allows the developer to fix the problems prior to delivery to main  Nightly build is performed daily (21:00) on main which contains all deliveries made that day. –Although Test4Delivery was performed, by the time feature was delivered, others might also delivered to main so it all should be tested together  These steps help keep the main version stable 5

6. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Our Development Procedure  Nightly build includes: –Creation of a daily view for each product –Complete build (Java™, C++ & Matlab) –Testing –Additional (optional) tasks Distribution of compilation results to offshore development center Preparing installation KIT Checking module dependency violation (Lattix™)Lattix 6

7. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 TestNG™ – Our Testing Framework  Run using Ant™  Based on the backend paradigm –Set items and execute actions  Tests written by developers 7

8. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 TestNG – Our Previous Testing Framework  Tests didn’t find enough bugs –Insufficient coverage –Not simulating actual scenarios (working via UI) –Better HW simulators  Tests take too long to finish  QA team unable to create tests 8 Nightly build on main finishes at ~3 AM. Tests finish running at 1 PM

9. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 JSystem™ – Our New Testing Framework  JSystem is a professional open-source framework for writing and managing automated system tests –Jemmy based architecture (methods to reproduce all user actions which can be performed on UI components) –Active / Passive GUI testing ability Passive – GUI running in background. Actions invoked on backend. Ability to close popups. Active – pressing buttons on screen, checking button enabled/disabled, etc. –Support tests hierarchy Each tests suite can be constructed from other suites or basic building blocks 9 Java VM JSystem Agent Application JSystem Application

10. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 JSystem – Our New Testing Framework –GUI based tool for creating & running scenarios QA compatible (assuming building blocks already exist) –View status of running tests In which suite/test/building block we are now, what done, what left to do Input/Output per building block Error detailed information in case of failure 10

11. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 The Way We Were…  Test4Delivery (feature development and bug fixing) procedure Developers could choose between 2 options: Run test locally on their work station Get a token and run test on remote station 11

12. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 The Way We Were…  Test4Delivery (feature development and bug fixing) procedure –Ease of use User had to make sure view is ready for testing (if using dynamic view – build it or copy compilation results from snapshot view) –Productivity Try and catch a token or leave it running overnight (work stations were unusable during test) –Duration Took all night (no one really stayed to count the minutes) 12

13. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 The Way We Were…  Nightly build –Ease of use Sanity duty was exhausting (users ran sanity tests manually) Collect the required information / results from different location –Productivity Took long time to perform Problem solving was delayed due to late results –Stability / Robustness Specific server dedicated to specific view Failure to compile non-relevant part meant no test results 13

14. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Out With the Old… In With the New  We use a build farm (~50 blades) to perform Test4Delivery and Nightly build tasks  These tasks are handle by AnthillPro server 14

15. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 AnthillPro™ in a Nutshell 15 “AnthillPro combines Continuous Integration with automated deployments and test orchestration to create a Continuous Delivery pipeline”  Build, deploy, test, and release automation framework  Run a "test build" of changes in the build environment without committing the changes to source  Supports, on demand and recurring scheduled builds  Server-agent architecture coupled with workflows that support parallel job processing  Efficient event-based and repository triggers  Invoke tests, notifying testers of the results and trend the data  Integrated repository stores artifacts for reuse and later lifecycle stages http://www.urbancode.com/html/products/anthillpro/

16. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 AnthillPro in a Nutshell 16 View Server Anthill Server Farm Blades  Continues integration is where we’re heading but not there yet….  We use AnthillPro for distribution and parallelism of specific tasks on the farm

17. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Nightly Build - AnthillPro behind the scenes 17  Parallel building of modules  Limiting test suites under one hour each

18. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Test4Delivery - AnthillPro behind the scenes 18 Developer’s [Build and] Test for Delivery Request is sent via web client (explorer / EasyLife)EasyLife Local agent on developer’s station is identified If requested, snapshot view is compiled on local developer’s station Dynamic view is created (if not exists) and compilation results are copied (clone_view) Tests are distributed by platform / scenario and executed on farm Identify User Agent Copy build results to dynamic view Run Test workflow Send Build & Test Request Compile Partial view Locally [optional]

19. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 The Way We Were (for a very short time)  Test4Delivery –Ease of use Select view and submit –Productivity Send request and keep working as if nothing happened. Receive mail once completed –Duration 2.5 hours 19

20. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 The Way We Were (for a very short time)  Nightly build –Ease of use All the data available via Easy Life JSystem replaces manual testing –Productivity AnthillPro finishes nightly build around 7am (JSystem manual tests included) –Stability / Robustness Build and test distributed automatically. Can recover from failure Test can executed even if compilation partly failed 20

21. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Problems We Were Facing…. 21 Resources limitation More and more developers are joining…. More and more views scheduled to run nightly Network bandwidth Current workflows using network extensively

22. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Solutions 22 Resources limitation More and more developers are joining…. More and more views scheduled to run nightly Network bandwidth Current workflows using network extensively Better utilize the farm resources Improving T4D performance Improving Nightly Build Performance

23. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Improving T4D performance 23 Running JSystem tests locally Instead of using dynamic view, the artifacts are being copied to the farm Using Anthill artifacts repository to distribute compilation artifacts Using local logging Logs are written on local resource during the test Once test ends, logs are copied to dynamic view (TestNG) / network drive (JSystem) Using local logging (started on Jun 20)

24. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Improving Nightly Build performance 24 Nightly build using snapshot view Instead of building dynamic view, build snapshot view and copy artifacts to dynamic view Reuse snapshot view (move stream) Running JSystem tests locally Getting same improvement as in T4D View Server Snapshot view Blade Anthill Server Copy Artifacts Move stream & Build

25. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Where We Are Now  Bottom line: QA reports version stability improved dramatically  Work never ends…. 25 Dealing with increasing number of tests Build a weekly plan and run specific tests on specific days according to farm capacity Improving offshore development process Provide same capabilities of farm utilization Refining JSystem test dependency on specific build modules Run specific scenarios immediately when corresponding module finished compilation

26. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60

27. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Lattix™ 27 Suggested Components Separation Components’ Dependencies Rule Violation (a component which is lower in the hierarchy uses a higher one) Detailed analysis and drill-down of dependencies

28. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Lattix™ 28 Data layer depends on Application layer Specific Data layer Class and its dependency on Specific Application layer Classes

29. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 EasyLife™ 29  Although the server has a web access we use EasyLife to submit tests  Test results data (TestNG & JSystem) is collected and presented

30. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Submitting Test4Delivery Task 30 (feature development and bug fixing) 1.Select view –Partial view if compilation prior to testing needed –Dynamic view (if view is already compiled) 2.Submit 1 2

31. Applied Materials External Use R 140 G 140 B 140 R 220 G 220 B 220 R 69 G 153 B 195 R 254 G 203 B 0 R 255 G 121 B 1 R 234 G 40 B 57 R 155 G 238 B 255 R 146 G 212 B 0 R 75 G 75 B 75 R 6 G 30 B 60 Submitting Test4Delivery Task 31 (feature development and bug fixing)  Once Test4Delivery finishes you’ll receive a mail  Results are available in EasyLife 1.JSystem sanity tests 2.JSystem scenarios 3.TestNG 1 2 3