Copyright © 2003 by Trend Micro Inc. Introduction to Test Automation Terrence Chou 周存貹 趨勢科技研發部協理

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.2 Agenda  Test Automation  Software Testing in an Enterprise Networking Environment  Group Discussion Review

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.3 Test Automation  What is Test Automation?  Make the test procedures to be executed automatically

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.4 Agenda  Why Test Automation?  False Expectations for Test Automation  Keys to Test Automation  Test Automation in PLC (Product Development Life Cycle)  Test Automation Considerations

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.5 Why Test Automation  Make the testing procedures systematic, consistent and 100% reproducible  Minimize the error-prone manual testing  Save effort of QA Engineer: fast, in the night and short testing cycle  Some testing is not able to be done manually More Cost Effective!

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.6 Benefits of Automated Testing  Production of a reliable system  Improved performance testing  Improved load/stress testing  Quality measurements

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.7 Benefits of Automated Testing  Reduction of the test effort and minimization of the schedule  Test plan development  Test procedure development  Test execution  Test result analysis  Report creation

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.8 Benefits of Automated Testing  Improvement of the quality of the test effort  Improved build verification testing(Smoke Test)  Improved regression testing  Improved multi-platform compatibility testing  Improved software compatibility testing  Improved execution of routine tasks  Improved focus on advanced test issues  Increased testing coverage  Execution of tests that can’t be done manually  Ability to reproduce software defects  After-hours testing  Easy to take over

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.9 Keys of Test Automation  Well-defined testing process  Involve at the beginning of project  Introduce test automation into process.  Repeat and continuous improvement.  Test Automation Consideration  Reuse  Measurable Values

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.10 False Expectations  False Expectations for Test Automation  Immediate Test Effort Reduction  Immediate Schedule Reduction  One Test Tool Fits All  Tool Ease of Use  100% Test Coverage

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.11 Some Questions Before Involvement  Scope the test automation  Which part should be automated  What is the priority  Which part can't or not worth automating  To buy or not?  Use commercial tool  Write program/script by ourselves  How to do automation?  How to organize these binaries with test data  How to deploy them to each test machine  How to gather test result and generate report ROI Analysis

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.12 Test Development in PLC  High Level Design  Involved Activities Test Plan Formulation Test Plan Review UI Review I  Outputs Test Automation Plan –Test Automation Decision –Tool Selection –Test Software (Tool) Development Requirements (TRD) –Performance Benchmark Plan –Risk Assessment

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.13 Test Development in PLC  Detail Design  Involved Activities Design Document Formulation Test Case Design and Review –Tool Introduction Test Software Design  Outputs Test Software Detail Design Document Test Software Schedule and Milestones

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.14 Test Development in PLC  Construction  Involved Activities Conduct Unit Test –Provide auto-test framework Test Software Development  Outputs Test Software Auto Test Scripts –Write auto test scripts for selected test cases or by requests –Conduct auto test scripts into automation smoke test process

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.15 Test Development in PLC  Pre-Release  Involved Activities Tune Performance  Outputs Performance Report/Recommendations

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.16 Test Development in PLC  Closure  Involved Activities Endorsement Meeting Post-Mortem Meeting  Outputs Test Automation Document

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.17 Test Automation Checklist  Can the test sequence of actions defined clearly?  Is it possible to automate the sequence of actions?  Is it possible to “semi-automate” a test?  Is it necessary to repeat the sequence of actions many times?  Do you need to run the same tests on multiple hardware configurations?  Are you testing non-UI aspects of the programs?

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.18 Test Automation Considerations  When to involve in a project  Don't lose sight of testing goals  Don't duplicate automation of the AUT's program logic  Analyze the automation effort  Analyze the reuse potential of automated modules  Focus on repetitive tasks  Focus on data-driven tasks

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.19 Testing Tools  Rational: Visual Test (original from Microsoft)  Rational: Robot  Microsoft: Visual Basic, Visual C/C++  Script language: Perl…  Others:  Pure Coverage  SendMail, MailBomb…  Burnit  …

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.20 Discussion  Does test automation save time?  Time invested v.s. Time saved  Who invests the time v.s. Who saves time  The testing schedule is shortened v.s. Neither fewer testers nor faster release cycle.

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.21  Q & A

Copyright © 2003 by Trend Micro Inc. Software Testing in an Enterprise Networking 2003 NTU Seminar

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.23 RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line  Enterprise Networking Topology  Software Performance Testing  Software Security Testing  Q&A

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.24 RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line  Internet Segment - Provide internet services to customers.  Extranet Segment - Provide B2B services to business partners.  Intranet Segment - Provide enterprise services to internal users.

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.25

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.26 RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line  Software Performance Testing -Testing focused on both “how much” and “how fast” the software can handle.  Software Load Testing -Testing focused on “how much” the software can handle.  Software Stress Testing -Testing focused on “varying loads” that the software can handle.  Tools for Performance Testing -Introduction of industry’s enterprise-class load testing tools to simulate thousands of users to measure software performance and get server resources data (CPU, MEM, Disk I/O, Network I/O...) in real-time for performance improvement and bottleneck analysis.

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.27 RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line  Step 1: Creating a script with the Recorder. -Capture the test script of traffic transferred between a client application and a server then do a trial run.  Step 2: Customizing the generated test script. -Customizing the test script you have generated allows you to take the actions performed by a single user and from them create a realistic simulation of multiple users.  Step 3: Replaying the customized script. -Select the workload model and prepare the workload to run a full load test.  Step 4: Viewing the results of the load test. -Explore response time and throughput results of the load test and display performance results in real time to analyze performance from the client and server point of view.

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.28 RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line  Software Security Holes Testing -Testing focused on finding well known software security holes in the audits database.  Software Buffer Overflow Testing -Testing focused on finding the software buffer overflow problems on listening ports.  Software Hacking Testing -Testing focused on simulating hackers to find unknown software security holes.  Tools for Security Testing -Introduction tools to scan software security problems and create reports for software security risk analysis.

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.29 RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line  Step 1: Check the audits database. -Keep the audit database up to date before software security holes testing.  Step 2: Select the security test policy. -Select the test policy you need (scan perform protocol identification on found open ports or scan all ports in range) then run a complete scan and view results to analyze risk.  Step 3: Create and customize a new audit. -Create a new audit and customize for a hacker simulation test.  Step 4: Run the buffer overflow checks. -Run the DoS checks to test listening ports (FTP, HTTP, SMTP, POP3) and find the host down service.

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.30  Q & A

Copyright © 2003 by Trend Micro Inc. Group Discussion Review Terrence Chou 周存貹 趨勢科技研發部協理

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.32 Agenda  MRD (Marketing Requirement Document)  Project Plan  Communication  Change Control  Confidence level  What are you going to do in next 6 weeks  What should you prepare at 6/10

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.33 MRD  Ideas come from:  Past project related  Improve Trend’s programming contest  Team interest  Many creative ideas  Good customer survey  Dropped ideas…  Has priorities

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.34 Project Plan  Only 6 weeks left  Most teams have high level design already  Most PM, RD and QA have plan  What is plan:  PM, RD and QA time table  QA and RD must sync the work  PM must check the schedule constantly  Everybody should response for the prediction

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.35 Communication  How many meetings already?  Everybody contribute his/her thoughts?  Listen and respect others.  No mis-understanding.

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.36 Change Control  What should you do when one of team member gone?  Every team will cover the job by others.  What should you do if the requirements changed?  What should you do if the milestone missed?

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.37 Confidence level  Most team has 85% ~ 100% confidence level  Too optimistic?  Don’t take too conservative  Willing to take the challenge

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.38 Conclusion  Creative  Above my expection  Communication  Find the way to communicate  Change  Willing to accept the change  Customer focus  Need approve

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.39 What are you going to in next 6 weeks  PM: Monitoring  RD: Coding  QA: Testing  All: Documentation

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.40 What should you prepare at 6/10  A working product!  MRD  Project Plan  PM, RD and QA time table  Source code  Test cases  Bug statistic  Presentation file  Try to see this product

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.41 What should you prepare at 6/10  Optional:  Usability study  User’s manual  Change control history  Architectural design  Focus Group  L10N/G18N plan  Future plan  …

RUNNING HEADER, 14 PT., ALL CAPS, Line Spacing=1 line Copyright © 2003 by Trend Micro Inc.42  Q & A