Performance Considerations in Agile Projects CMG-CA 2016

How Agile works? (scrum version) Product Owner Scrum Team Product Backlog Functionality 1 Functionality 2 Functionality 3 …. Functionality n Scrum Team Team select as much functionality as it can be done in 1 sprint Task Breakout Task Breakout Task Breakout Code Release Sprint Planning Meeting Sprint (1 to 4 weeks) Sprint Review Sprint Retrospective Prioritized High level

Why a project moves to Agile? Complex Domain Probe, Sense, Respond Explore to learn the problem Unpredictable environment Creative/innovative approach e.g. : new market products Complicated Domain Sense, Analyze, Respond Metrics / experts for control / insight Good practices Predictable environment e.g. : Performance optimization Chaotic Domain Act, Sense, Respond Act immediately, then inspect Many decisions to make, no time No one knows, no clear cause/effect e.g. : Bank panic Simple Domain Sense, Categorize, Respond Response based on established practices Stable domain Correct answer exists and is well known e.g. : offshore outsourced process

What means Agile for an organization? (strict interpretation) The most radical change in IT departments since the personal computer arrives. For testing and development is a profound and radical transformation It modifies the organization in teams, without hierarchies, and without a clear distinction between developer and testers or technical support people, there is only team members in Agile. The development environments are build on the fly, as they are needed. Continuous integration (code compilation and link every 30 seconds), continuous regression testing (at least every night) The team is all in the same physical location (in site or offshore). There are not more manual testing, all testers are capable of doing automation testing Every sprint (1 to 4 weeks) should finish in “potentially deployable code” means ready for production The end of the project manager as we know it What it means for technical support teams Less control of the number of environments and how are they administered Smaller, more frequents deployments Less documentation about the systems to be deployed More unpredictable workloads and performance

What is happening with Agile implementation in the real life The change is slowly but firmly implemented in the financial institutions in GTA Currently most of the Agile projects are partially Agile Use Agile tools like Jira, Jenkins, etc. Follow the Scrum version of Agile Use a collocated place where all the members are in the same place and share information face to face (that by itself improves the results of the project). Implement code in production not in each sprint but by groups of Sprints, with a last sprint previous to release called “Hardening Sprint” There is a “cloud” of support people that surrounds the Scrum team (test architects, development architects, technical support, etc). Some of these are off shore. The Scrum team are less stable than the “canon” said, members sometimes rotate after a couple of Sprints, as each Agile project is used as a way to win experience in the process. There is discussion about if the whole IT organization could be Agile or not.

Why Agile and performance appears to move in different directions? For performance specialists (capacity planners, Performance testers, etc.) Less control of what happens in the testing environments No clear estimation of the functionality or the implementation of that functionality until 4 weeks before deployment If there is no performance requirements explicitly defined, performance is not part of the team deliverables That’s no so different of what happens now Functional quality of the code is generally really good, performance it depends… Code is delivered faster than before, the workload in production is lot less predictable But the impact of the changes is lower as each deployment is smaller than in the waterfall model

Agile : Challenges for Performance It talks about quality, but most of the time it means “functional quality”, not quality in term of performance It shouldn’t be this way, but usually is what it happens It never talks about performance testing. Most of the Agile teams says that performance testing is too hard or too risky to even attempt (Scott Barber “An explanation of Performance Testing on an Agile Team”) Sometimes the most evident (and easier to do) software architecture is not scalable at all Agile projects has the tendency of doing very good software, but with serious problems of scalability We can said the same about resiliency. If nobody does a load testing, the defect will be found only when the application is in production. In all the Agile manifest there is not only one reference to performance. Sometimes is considered that if performance is important for the user it will be explicitly considered in one of the cycles, but usually it’s the kind of requirement that an agile process is not very well prepared to cope with if there is not an explicit practice to analyze and design alternatives in order to achieve the desired performance. Agile makes emphasis in speed, but most of the times the fastest solution to build is not the better solution in terms of performance and scalability.

Performance requirements in Agile There is no “requirements” in Agile lingo, they are called “user stories” and they are oriented to functional behavior Performance requirements could be an “user story” by itself or something that is added to each user story Personally I prefer the second approach. Each user story should have their performance requirements defined as: Number of transactions estimated Desirable response time that should be the base for an SLA Availability requirements “Balance what is economically possible with the business needs. If you can’t tell him/her that this requirement will be negotiated later during the design, but never after the design is finished” This phrase of a waterfall context is not valid anymore in Agile. The moment of negotiation is now.

How to collect performance requirements? For each user story ask the product owner: What is the response time required? Remember: “As fast as possible” is not a response, is a desire and is too expensive. Never accept “x seconds 100% of time”, its always at least “x seconds 90% of the cases”. If it is necessary take in account the level of workload. How many, of each particular user story, will the system execute per period of time? (days, hours, etc) Always explicit the unit of time (hours, days). Is this value a peak? Is it an average? Is there an seasonal difference? (Christmas, REER period, vacations, etc). Remember that in these case you will not be asking for transactions but for complete functionalities Where it has to be measured? (in the end user interface? At the exit point of a component?) How much it will grow in the next 6 months after the deployment? And in 4 years? There is some topics here related to how to define a performance requirement, specially when it refers to response time. Specially important is the fact that expressions like “less that 1 second” are more expressions of desire but are really poor performance requirements. That expression also supposes that all the transactions should take less than 1 second, and there is always weird cases, that doesn’t fit in that, doesn’t matter how good you system was designed there is 0.0001% of cases that are not in the norm. You should protect of that. In the end imagine that the performance requirement collected should also be the base number for a SLA negotiation. The level of workload should be deduced from the workload defined from the business requirements perspective, but in our experience you should also ask what is the volume expected at the use case level in order to compare them with the numbers that is expected in terms of business. Rarely are the same…

Agile and performance testing As each iteration has to generate a software good enough to enter in production, with a minimal functionality, in consequence that software has to have defined performance requirements to meet and also to be tested. This diagram show the relation between the construction iteration and testing in an Agile development. Load testing is a test as any other that should be done after each construction iteration. Also performance requirements should be collected as all the other requirements at the beginning of the iteration, and the information passed to the load tester, to generate the test scenario. Source : Dr Dobb’s Portal : “Agile Testing Strategies” by Scott Ambler

Agile and performance testing (other approach) As each iteration has to generate a software good enough to enter in production, with a minimal functionality, in consequence that software has to have defined performance requirements to meet and also to be tested. This diagram show the relation between the construction iteration and testing in an Agile development. Load testing is a test as any other that should be done after each construction iteration. Also performance requirements should be collected as all the other requirements at the beginning of the iteration, and the information passed to the load tester, to generate the test scenario. Source : Dr Dobb’s Portal : “Agile Testing Strategies” by Scott Ambler

The performance oriented professional in an Agile project The performance specialist should help the rest of the team to think about performance during the whole project Product backlog: introducing the idea of Performance Requirements in each user story During Sprint Planning meeting: consider the impact of the performance requirements (written or not) in the estimation of the Sprint backlog Sprint: working with the developers in the team to find performance oriented architect solutions into the code (performance antipaterns, software performance engineering, etc) Sprint Review & Retrospective: asking questions about performance and how good is the product in terms of performance and efficiency Think like the Performance Expert in Neil Gunther book “Guerrilla Capacity Planning” Insist about the need of having a performance tester or a performance specialist as a part of the team if each spring generates implementable code or to do performance testing in the “hardening sprint”.

Questions?