How We Do Test Automation at HomeAdvisor A last-minute production

Who Am I? Nicholas Rosevear - QA Automation Architect At HomeAdvisor for 11 months Previously worked at Shutterstock, Pearson, IQNavigator and EchoStar

A Look at Our Company Customer Facing Application 2 Sided Marketplace Domestic US/Canada User Base Dealing With a Rapidly Increasing User Base Large Scale Integration With Social Media Companies

A Look at Our Architecture Java Oracle DB & ElasticSearch Coherence Memory Cache Kafka Data Pipelines Node – Front End DevOps Managed through Jenkins Google Analytics, Adobe Omniture Moving to a Massive Microservice Architecture

A Look at Our Test Architecture Java/Spock - Unit/Integration Tests Java/TestNg/Appium – Mobile Automated Tests NodeJs/Nightwatch/Mocha – UI Automated Tests NodeJs/Supertest/Mocha – API Tests NodeJs/BenchRest - Performance Testing NodeJs/Express/Casperjs – Performance Testing NodeJs/Express – Assorted Other Monitoring/Assistance Tools Java/TestNg/Selenium - Legacy Automated UI Tests Jmeter – Legacy Performance Tests

How Did We Get Here? Deciding Between Needs and Wants High Test Coverage Usable/Maintainable Tests Fast Performance Testing and Trending Analysis Test Data Management for Complicated Test Setup/Teardowns Wants Better CI/CD Infrastructure Useful Documentation Simplified Test Execution and Reporting More Thorough Security/Compatibility Testing

So What Did We Do? Built a new UI framework to replace unstable long running legacy tests Built a dynamic API test framework to generate tests on its own and enforce consistency Built easy JSON config file driven performance testing Built internal Test API and sites to manage data creation/validation across tools and for manual testers

So What Did We Do We took advantage of simplicity of Express Easy to Display and Document Easy to Train and Teach

The Blurred Line Between Automation and DevOps At some point we drifted out of Test Engineering Everything can be automated Tools for the company Monitoring CI/CD Efforts

UI Automation - Nightwatch Built in NodeJs with Nightwatch Built to focus on readability and simplicity to create And easy to understand output and test results Built to consider switching technologies

Mobile Automation – Unnamed Page Object Built as a One-Off for the Mobile team Basic Page Object style framework built on top of Appium Same concepts as Nightwatch Easy to use Maintainable Flexible across mobile devices

API Automation – Api-Tite We are dealing with 500+ unique API Endpoints Legacy and New Endpoints Multiple teams, multiple standards, multiple test coverage Created a framework to sit on top of swagger and build a database of all APIs and dynamically generate tests Readability not prioritized Guaranteed Coverage Immediate Test Detection Security Coverage Ability to enforce consistency

Performance Testing - Scattershot Had a couple of close calls with performance issues Rapid development cycle means a need for performance testing with little advanced notice Built a framework built around using simple config files to generate tests Tracking performance across different numbers of concurrent connections and iterations Allowing dynamic setup for testing to test more difficult endpoints like DELETE Long term analytical engine for trending and comparison

QA/Automation Support - Zamboni With more tools we were running the risk of duplicated effort There were security concerns and other testing efforts that required quick access to data We created a tool Our own test server complete with Endpoints for data creation/validation Abstraction of common DB queries/API calls to consolidate and visualize information Ability to quickly add support for testing efforts

Performance Testing - Ichabod We want to do more exotic testing Chaos and Disaster Recovery Testing Realistic Load-Runner style load testing We Built A Tool To allow a network of machines to generate load on an environment Using UI testing scripts made headless for efficiency Manageable from a single hub site Easy to expand scripts and configurable to choose what type of load to add to the system Real Time results and updateability

Reporting/Monitoring – Jenkins Crawler We have a lot of tests running Across a number of Jenkins jobs Determining overall health or specific health for certain applications teams is difficult Building a Jenkins Crawler Pulls data from various Jenkins jobs and stores To allow analysis of success, number of test pass/fail historical accuracy To allow simple email reporting for overall health based on Jenkins results Potentially to implement full CI/CD pipelines

QA And Automation Support - HostManager Finally just a quick example of using automation to solve problems Created a simple host server to manage host files across company Gives us a way to guarantee everyone has the same host entries Allows automation to be able to change hosts remotely

So What’s Next? Docker, docker, docker Integration with Sauce Labs Simplified Test Creation – No Programming Required RabbitMQ message queues to manage jobs, data creation and reporting A specific security testing suite Maybe switching technologies And of course more dashboards