How to Improve Releasing Efficiency via i18N/L10n Test Automation. View CART Case Study

Beyondsoft Overview: 20 Years’ Footprint to Excellence

Beyondsoft Overview: Full-Cycle G11N Services

VMware and Beyondsoft Beyondsoft has worked with VMware since Jan 2014. Services provided : G11N Testing Service Provider Team size - 23-30 Engineers L10N Engineering Service Provider Team size - 2-10 Engineers Marketing Partner Developing VMware new product solutions for Local Chinese Clients

View CART Introduction View CART is a term that describes View Clients for various platforms. The Project Objective is to test : View Client features Interaction between View Client and Virtual Desktops. L10N Keyboard Input Testing: It means user can use native keyboard to input various characters to Virtual Desktop from the client. Client UI Testing: Except Keyboard Input testing, other testing on client UI are belong to Client UI testing, such as: login, logout, do specified operation via menu, check the strings displayed on UI. PDP: Product Delivery and Packaging, it is similar with installation testing.

Areas where efficiency can be improved Many similar type of bugs are found in Keyboard Input Testing Multiple platforms need to be covered Very important old features must be tested every cycle. Time required to upgrade each test environment Manual test is performed during office work hours only Ever increasing number of new features, and manpower is limited. Automation is the most suitable way to solve the above issues!

Automation Assessment Strategy to select test areas for automation SN Rationale Area selected 1 Areas with high possibility of finding bugs Keyboard UI test 2 Basic and important features P0 features 3 Areas which require long execution time Download build Installation Deployment Upgrade Copy or move large files 4 Areas which are not going to be automated New, incomplete and unstable features Non-P0 features Translation: it means rearrange the order of steps in the manual test cases

Automation Assessment Reorder Manual Test Cases for Automation SN Tool Selection Rationale Tools selected 1 Rearrange the test cases into a test set in a specified order For Android keyboard input, manual cases are arranged by category, such as: Number keys Letter keys Symbol keys Child keys All number, letter and symbol keys have child keys: To simplify automation script development, each child key test case is split into three automation scripts for number keys, letter keys and symbol keys. This helps to improve the efficiency of : automation script development Test execution Translation: it means rearrange the order of steps in the manual test cases

Automation Assessment Automation Tools selection criteria SN Tool Selection Rationale Tools selected 1 Reduce learning curve Tools which VMware has experience with 2 Reduce creating tools from scratch Open source automation tools or framework OS native automation tools Translation: it means rearrange the order of steps in the manual test cases

Automation Tool Selection Tool selection for Windows OS Tool Advantage Disadvantage Selected Windows Ranorex Support software, web on IE, Chrome, Firefox, safari, flash. Good report generation. Support record and play Programming language is C# Less time to deploy and install Can execute with a executable file. Commercial License required √ UIAutomation Support Software only and web on IE. Programming languages - VB, C#, etc. Does not support record and play Need more development base on API QTP Support software, web on IE, Chrome, Firefox, safari, flash Programming language is VB Script Commercial license required Need more time to deploy and install

Automation Tool Selection Tool selection for MAC and Android OS Tool Advantage Disadvantage Selected Mac Apple Script No License required Can operate software by properties of controller For some controllers, need assistance from other tools. √ Sikuli Can operate any controller by using image Low recognition efficiency based on coordinates Need a lot of time to maintain images (eg screenshots, etc) Android UiAutomator Can action by other tool during running. Can test cross software Don't resign release apk. only support android 4.0 and above. Need identify controller by text for Android 4.0 and 4.1. Robotium Cannot operate cross application Cannot insert action from other tool while running. Need re-sign APK with debug certification

Framework Selection Challenges Operating software in VM OS Issue encountered Beyondsoft solution Windows Need change IME operate notepad in VM copy test result from VM to local desktop. Develop a tool to perform these tasks and execute it on VM using STAF Simulate keyboard input OS Issue encountered Beyondsoft solution Windows Caps Lock and Num Lock can’t work when send key code from local desktop to VM Use STAF to simulate keyboard input : Send Scan Code for Caps Lock Send Key code for numeric keypad since to Num Lock scan code doesn’t work. Android Can’t use key code to simulate all characters/symbols input Use virtual keyboard key coordinates to simulate keyboard input

Automation Development Environment Ranorex for Windows UIAutomator for Android Apple Script for Mac OS Automation Tools STAF + Client Scripts + VD side Scripts Automation Framework C# -> Windows JavaScript -> Mac OS Python & Java -> Android Program Language Why we use STAF: When try to input some special characters via automation from client to VD(virtual desktop), such as: Num lock, it will be failed by unknown reason. So we developed a VD side code, and use STAF to connect the client and VD, the STAF can help to: Lunch the application in VD, e.g. : Notepad Lunch input method in VD, and transfer input command from client to VD to simulate the keyboard input. Copy the txt file from VD to client, then it can verify the actual result on client side.

Framework Overview Client(Local PC) VM(Remote PC) Network Config Open VM in View Client 1 1. Open VM in View Client in test computer (Action on local desktop) STAF 2. Change default IME in test computer (Action on local desktop) Change IME Automation Test Controller 2 Copy Assistant Tool 3 Copy Command 3 3. Copy Assistant Tool to VM (Action on remote desktop) 4 Execute Command 4 Change IME 4. Change default IME in VM (Action on remote desktop) 5 5 Open Notepad 5. Open Notepad in VM (Action on remote desktop) 6. Send Key code/scan code to Input to Notepad in VM (Action on local desktop) Send Key Code / Scan Code to VM 6 7 7 Save Notepad 7. Save Notepad with input content in VM (Action on remote desktop) 8. Copy the saved Notepad from VM to test computer (Action on remote desktop) 8 8 Copy Saved Notepad 8 Copy Command 8 9 Verify Test Result 9. Compare test result in the saved Notepad in test computer (Action on local desktop) Repeat steps for each case Report 10. Repeat step 4-8 until test is completed.

Efficiency Gained 5 days 12.8% Note: As more cycles are executed, these numbers will become higher Days saved for this project 5 days Efficiency improvement 12.8%