Automation

70% of software projects are unsuccessful Chaos Report Automation projects ARE software projects

90-100% of automation projects fail Why do automation projects fail? Technical Factors Record Playback Trap Slow execution times UI automation unstable Maintenance

Non Technical Factors My personal experiences Where I failed Lessons I’ve learned

Management buy in required Case Study (The salesman) Large testing team Small mix of testers that can code and developers Strong team buy in No management buy in Management buy in required There was an unrealistic expectation of what automation can do. Due to lack of management approval there was literally no billing code. More time was spent selling the idea than building on it. In fact I find that in every automation project I had to sell it one way or another. LessonManagement buy in required.

Team buy in required Case Study (The Expert) Large team of testers (non coders) The expert Management buy in Lack of tester participation Team buy in required Automation code and test entry separated. Double the maintenance. Train testing team to use the tool. Lunch and learns, sit face to face but the turnout was disappointing. Lesson learned team buy in required.

No silver bullet Case Study (Baggage) Large testing team Experienced team of automators Big shot Silicon Valley Manager Management and testing team buy in No continuous integration No silver bullet Trying too hard to make it perfect. Trying to mimic what was previously done in a successful project. Very low ROI. Lesson learned no silver bullet. What worked previously may not work again. That perfect framework, page object model, junit integration, one click execution. No so perfect after all. Not everyone is Ebay.

Keep testers close Case study (Telephone game) Large testing team (non coders) Strong automation team Lots of buy in Great development practices Offshore developers, on/offshore testers, on/offshore automators, exploratory testing Keep testers close Communication was too slow. Too many team leads. Separation of responsibility was sickening. Offshore automation programmer needs to talk to automation lead to talk to the architect to talk to the project test lead to talk to the offshore test manager to talk to the tester in order to clarify test case details. Lesson learned, cut out middlemen. Testers should be as close to automators as possible.

Know when to stop Case Study (The Beast) Large team of testers Experienced team of automators Mature development practice Equally mature testing practice Know when to stop Everything was added to the framework. Everything was custom including data setup and test case entry, results capture and even reporting. Diminishing returns on ROI. Refactoring was a bad word. Lesson learned. Know when to stop. Leverage existing tools where ever possible

Fast is better than perfect Case Study (Man out of time) Tiny testing team all testers can code Management and team buy in Development was a well oiled machine One week sprints Fast is better than perfect Builds are a continuous onslaught. Feature were here one day and gone the next. Lots of automation but testing was always a few builds behind. Project was over as soon as framework had matured. Lesson learned, fast is better than perfect. Need to balance minimal maintenance with quick ROI. Solid framework is very important. But don’t be afraid to hack. More and more projects are becoming this way. I blame our friend Agile. Development is getting faster, we have to be even faster.

The Goal Automation is not meant to replace testing It is used to free up time for other activities more suitable for humans. Exploratory testing, context-driven testing AI is a separate discussion Find All Pizzas Find All Pepperoni Pizzas Find All Edible Pepperoni Pizzas Find All Delicious Pepperoni Pizzas. How do you determine Deliciousness? Find All Foods My 4 year old would like to eat I purposely ordered them this way from easiest to hardest for a machine to do. The last question. Find all foods My 4 year old would like is the most similar to the question. Does this feature work the way my end users expect. That’s why you can’t replace everything with automation. Not yet anyway.

Surprise Surprise

Ingredients for success Keep it small Keep testers close to automators Make sure you have buy in Keep it small 62% success!! Quick ROI Don’t be afraid leverage existing tools and hack Did I mention keep it small? Make it Micro

Micro Automation Testers automate small everyday tasks intended to save time. Over time these scripts are bundled together to form the automation framework.

Micro Automation Choose a flexible framework ie programming language Your testers are your automators “Narkov” http://www.businessinsider.com/programmer- automates-his-job-2015-11 Proven ROI A reason to invest in a framework Train your team Teach them to use scripts Teach them a language that can parse text Teach them to use Virtual machines

Micro Automation Skynet launches nuclear missiles against humanity. No automation, over the fence development Simple code snippets, automated deployment Automated regression, continuous integration Continuous testing, virtual environments, remote control, artificial intelligence Skynet launches nuclear missiles against humanity.

Discussion samstruong@gmail.com Sam Truong Don’t forget the extra “s” in the middle. Sam Truong