1. Automation

2. 70% of software projects are unsuccessful
Chaos Report
Automation projects ARE software projects

3. 90-100% of automation projects fail
Why do automation projects fail?
Technical Factors
Record Playback Trap
Slow execution times
UI automation unstable
Maintenance

4. Non Technical Factors My personal experiences Where I failed
Lessons I’ve learned

5. Management buy in required
Case Study (The salesman)
Large testing team
Small mix of testers that can code and developers
Strong team buy in
No management buy in
Management buy in required
There was an unrealistic expectation of what automation can do. Due to lack of management approval there was literally no billing code. More time was spent selling the idea than building on it. In fact I find that in every automation project I had to sell it one way or another. LessonManagement buy in required.

6. Team buy in required Case Study (The Expert)
Large team of testers (non coders)
The expert
Management buy in
Lack of tester participation
Team buy in required
Automation code and test entry separated. Double the maintenance. Train testing team to use the tool. Lunch and learns, sit face to face but the turnout was disappointing. Lesson learned team buy in required.

7. No silver bullet Case Study (Baggage) Large testing team
Experienced team of automators
Big shot Silicon Valley Manager
Management and testing team buy in
No continuous integration
No silver bullet
Trying too hard to make it perfect. Trying to mimic what was previously done in a successful project. Very low ROI. Lesson learned no silver bullet. What worked previously may not work again. That perfect framework, page object model, junit integration, one click execution. No so perfect after all. Not everyone is Ebay.

8. Keep testers close Case study (Telephone game)
Large testing team (non coders)
Strong automation team
Lots of buy in
Great development practices
Offshore developers, on/offshore testers, on/offshore automators, exploratory testing
Keep testers close
Communication was too slow. Too many team leads. Separation of responsibility was sickening. Offshore automation programmer needs to talk to automation lead to talk to the architect to talk to the project test lead to talk to the offshore test manager to talk to the tester in order to clarify test case details. Lesson learned, cut out middlemen. Testers should be as close to automators as possible.

9. Know when to stop Case Study (The Beast) Large team of testers
Experienced team of automators
Mature development practice
Equally mature testing practice
Know when to stop
Everything was added to the framework. Everything was custom including data setup and test case entry, results capture and even reporting. Diminishing returns on ROI. Refactoring was a bad word. Lesson learned. Know when to stop. Leverage existing tools where ever possible

10. Fast is better than perfect
Case Study (Man out of time)
Tiny testing team all testers can code
Management and team buy in
Development was a well oiled machine
One week sprints
Fast is better than perfect
Builds are a continuous onslaught. Feature were here one day and gone the next. Lots of automation but testing was always a few builds behind. Project was over as soon as framework had matured. Lesson learned, fast is better than perfect. Need to balance minimal maintenance with quick ROI. Solid framework is very important. But don’t be afraid to hack. More and more projects are becoming this way. I blame our friend Agile. Development is getting faster, we have to be even faster.

11. The Goal Automation is not meant to replace testing
It is used to free up time for other activities more suitable for humans. Exploratory testing, context-driven testing
AI is a separate discussion
Find All Pizzas
Find All Pepperoni Pizzas
Find All Edible Pepperoni Pizzas
Find All Delicious Pepperoni Pizzas. How do you determine Deliciousness?
Find All Foods My 4 year old would like to eat
I purposely ordered them this way from easiest to hardest for a machine to do. The last question. Find all foods My 4 year old would like is the most similar to the question. Does this feature work the way my end users expect. That’s why you can’t replace everything with automation. Not yet anyway.

12. Surprise Surprise

13. Ingredients for success
Keep it small
Keep testers close to automators
Make sure you have buy in
Keep it small 62% success!!
Quick ROI
Don’t be afraid leverage existing tools and hack
Did I mention keep it small? Make it Micro

14. Micro Automation
Testers automate small everyday tasks intended to save time. Over time these scripts are bundled together to form the automation framework.

15. Micro Automation Choose a flexible framework ie programming language
Your testers are your automators
“Narkov” automates-his-job
Proven ROI
A reason to invest in a framework
Train your team
Teach them to use scripts
Teach them a language that can parse text
Teach them to use Virtual machines

16. Micro Automation Skynet launches nuclear missiles against humanity.
No automation, over the fence development
Simple code snippets, automated deployment
Automated regression, continuous integration
Continuous testing, virtual environments, remote control, artificial intelligence
Skynet launches nuclear missiles against humanity.

17. Discussion samstruong@gmail.com Sam Truong
Don’t forget the extra “s” in the middle.
Sam Truong