1. Make your Test Automation
SMARTER
Jim Trentadue
Software Quality Consulting Director
Original Software

2. Agenda 1 2 3 4 5 6 Key advantages of Test Automation
Additional advantages of Test Automation not incorporated 3
Limitations of Test Automation 4
Topics for SMARTER Test Automation
There are two ways this slide could be operated:
You may hyperlink the various Agenda Items to the slides that encompass their respective content (do not forget to add a ‘back’ button linking back to this slide)
Alternatively you may copy & paste the agenda slide besides the respective content slides so that when you press next, it shows the agenda slide once more, but this time, the upcoming topic will be highlighted.
NOTE: This slide is suitable for 5 menu items max, in case you need to add more menu items, then first group all the textboxes (including the numbered ones) together and resize them to make em smaller. The text in these text boxes has been set to “shrink to fit” settings, so now if you reduce the size of the textboxes the font size will change automatically, thereby making it easier to add more menu items without any problems. 5
Case Study 6
Session recap

3. Key advantages of Test Automation
In addition to time & cost savings
Robust Test Automation
Team effectiveness
Data
Round the clock
More scenarios for functional, negative, equivalent class, boundary, etc.
Better Test Objectives - requirements / user story
Better Test Metrics -effectiveness and data coverage
Work with UAT earlier; incorporating customer tests

4. Additional advantages of Test Automation
Going beyond automated testing through your SUT
Test data setup –
not validation
Automate your updates and inserts
SQL Editor for table query results
Backend data functions update
Component-level (message queues, scripts, web services, stored procs)
Test data injection

5. Limitations of Test Automation
Understanding limitations can better serve its purpose
UI layout
Witnessing the look & feel of an application; responsive design in a pure definition
Test Environments
Finding hardware or network gaps across the test environments that are not equivalent to production
Condition injection
Inserting an exploratory test condition during the execution to check system behavior

6. SMARTER Test Automation Roadmap
Strategy
Understand Application Behavior M
Methodology
Implement the logical model A
Adaptable
Scale across platforms & browsers R
Roadmap
Order test sequence & priority T
Toolbox
Working with processes & technologies E
Experienced
Define requirements for personnel
Repeatable
Design for future success
Source: Celtic Testing Experts, Inc.

7. Conditional Execution
STRATEGY
Strategy for understanding changes in application behavior
Conditional Execution
Execute only tests that relate to changes made?
Manual Tests
Strategy
Replicate static Manual Tests?
How are errors handled?
Application Changes
Results Analysis
Quickly report defects against the changes…
Error-Handling
Strategy

8. STRATEGY Consider conditional execution as found below Test Suite
Test Case 1
- Module 1 (Scenario 1)
- Module 2 (Scenario 2)
(if PASS, execute Test Case 3
if FAIL, execute Test Case 2)
Test Case 2
- Module 3 (Scenario 3)
(if PASS, go to Test Case 4)
- Module 4 (Scenario 4)
(if FAIL, stop the testing)
Test Case 3
- Module5 (Scenario 5)
(if PASS, iterate through Test Case 1 for data-driven tests)
- Module6 (Scenario6)
(if FAIL, execute Test Case 2)

9. METHODOLOGY Methodology for implementing the logical model
LOGIN
MENU
ORDER
INVENTORY
PROCESSING
CONFIRMATION
REPORTS
24 full automation paths (recording or coding)
2 (3)* 3 2 2 1 1 1
*1 Negative scenario that doesn’t proceed
Test 1
Test 2
Test 3
Test 4
Test 5
Test 6
Test 7
Test 8
Test 9
Test 10
Test 11
Test 12
DATA-DRIVEN; only 1 module required

10. USING COMMON STRUCTURES FOR OBJECT RECOGNITION
ADAPTABLE
Adaptable across different platforms and browsers
USING COMMON STRUCTURES FOR OBJECT RECOGNITION
DESKTOP: UI Automation (UIA)
Microsoft Active Accessibility (MSAA)
WEB: Document Object Model (DOM)

11. ROADMAP Roadmap for outlining the sequence and priority of tests
Test Automation Prioritization
TOPIC
Manual Test Case readiness
Automation Difficulty
Automation Effort
# of releases automated test case(s) will be used
TOTAL
SCORE (1-10)
Functional (Manual) Testing Prioritization
TOPIC
# of test cases for functional area
Traceability from req. thru to test
# of releases manual test case(s) will be used
Release frequency of functional change
TOTAL
SCORE (1-10)
Business Prioritization
TOPIC
Volume of defects in production
Volume of defects in test
# of defects opened in last six months
# of defects opened within the last 12 months
TOTAL
SCORE (1-10)

12. Continuous Integration
TOOLBOX
Toolbox to work with technologies
Application(s)
Under Test
(AUT)
Test Management
Test Automation
Agile Planning
Continuous Integration

13. EXPERIENCED Personnel requirements for those to support changes
SHIFT-RIGHT
Developers:
Codes
Review with QA
Standards
Unit tests
System Testers:
SME knowledge
Test Plans
Test Cases
Executes system tests
Analyzes results
Reports
Understands:
Object Libraries
Code Functions
Debugging
Knowledge of:
Testing Process
Test Documents
App Integration
TEST AUTOMATION PROFESSIONALS

14. REPEATABLE Repeatable design for future success POINTS TO CONSIDER
Test Harness Development & Release
Test Case Development & Execution
Automated Reporting
Results & Performance Analysis
Stakeholder Reporting
Automation Candidates
Finalize Framework Study
Automated Test Creation
Review & Validation
Inclusion in Test Suite
POINTS TO CONSIDER
Static Conditions with Dynamic Data
Error-handling at every turn
Elimination of a manual test / task

15. Case Study: Improving Automation
Background:
Decision to automate was made for the following:
Save Time and Speed Up testing
Without an independent test team, the project team had to do all of the testing
It was thought that repetitive testing of the application was dull
The tools existed & looked fun!
Software Under Test:
Large scale sales & marketing automation program.
Client / Server application that ran on different Operating Systems and Relational Databases
Automated solution used:
SQA Robot
Case Study by Steve Allott, Software Test Automation: Fewster & Graham

16. Case Study: Generations of Automation
First Generation
Initially tried using Capture / Replay
Functions written in SQA Basic to enable complex test activities in one command
Second Generation
12 environments to support, mgmt. kept with automation
Automation was only for simple navigation & basic record; data-driving tests
Modular approach: Each component has a main GUI screen & main procedure
Third Generation
Creation of larger dedicated test team
Designed new automation infrastructure and mapped into a test automation estimation matrix
(expanded in next slide)
Evolution of Implementing Automation
ISSUES FACED
Basic use of the tool was limited; required coding
LESSONS LEARNED
Solution requires coding expertise
Need a champion
Solution lags with technologies
ISSUES FACED
Object recognition & event actions were unreliable
LESSONS LEARNED
Difficult to build automated region
Tests had to be released to test environment
ISSUES FACED
Technology used outgrew automated solution; had object recognition issues
LESSONS LEARNED
TA managed as a project initiative
Understanding of maintenance costs

17. Case Study: Levels of Prioritization
Characteristics of three levels of prioritization & estimation
Exercise the deepest level of functionality in a module, including those that interface to other components
Difficult if not impossible to test manually
Hard to automate
Unlikely to run successfully, repeatedly
Very likely to find a bug
Level 3
Explore all module aspects except interfaces to other components
It is possible but time consuming to test manually
It looks easy to automate, but doesn’t always turn out so
The automated test is likely to have bugs
It sometimes finds a bug
Level 2
Exercise the simplest aspect of the functionality in a module
It is usually straightforward to test manually
It is easy to automate
The automated test is likely to work
It is unlikely to find a new bug
Level 1

18. Case Study: Key Points
Key points and recommendations in this case study
Sell the ideas and benefits of automation; manage expectations carefully
Have a reasonably mature before you start to automate
Consider your business, technical and resource requirements
Evaluate in your environment
Build your prioritization and estimation scales for automation

19. Presentation Recap Reviewing the main points of the presentation
Understand all of the advantages
Know the limitations
Build your own definitions using the SMARTER acronym
State an objective for each keyword along with a real or anticipated scenario
Use case studies – like the one referenced here

20. Thank you for attending this presentation!
Jim Trentadue
Software Quality Consulting Director
Original Software