Continuous Evolution of Performance Test Assurance Cliff Zattoni FCA Capacity, Availability & Performance Manager

Team Setup Capacity Team x 3 Hybrid outsourced Tower model Internal governance practise Service Design + Transition

Challenges Outsourced partners in Support don’t take part in testing Governance is heavily geared towards process and not technical Leverage is not financial Captive audience Regulatory change Performance Monitoring

For assuring the quality of Performance Testing – support side Our Current Model For assuring the quality of Performance Testing – support side NFR engagement with Business and Service Design Risk assessment Execution analysis and tuning Service Technical Debt clinic Service Transition board

Evolution Insourcing of core capacity function Creation of Service Design Service Design Principles NFRs – Re-launch Release & Environments management re-launch BAU Testing Application Performance Monitoring Empower teams to self-service Better business engagement

Continuous Evolution of Performance Test Assurance Isabelle Cosar FCA Non-Functional Test Assurance Manager

NFT Setup NFT Team: OAT and Performance Test Assurance Specialists Performance Delivery testers from supplier OAT delivery (Infrastructure partners) Test Leads / Managers (Coordination)

For assuring the quality of Performance Testing Our Current Model For assuring the quality of Performance Testing NFT Team supports NFRs definition and review Risk Assessments & Scoping Templates, processes & guidelines Test Assurance Board Review PT deliverables Provide consultancy and guidance

Recent Evolution(Last 12 months) New unique supplier consultancy to deliver DEV & testing: new Performance testers and better capability Key Core systems moving to a Continuous Delivery Model (Scrum etc..) with a central team New Performance testers, better capability More Continuous Improvement focus Piloting of practices and toolset required for a continuous delivery pipeline

What’s the next evolution ? With moving to continuous delivery pipeline then DevOps Impact of CDM & DevOps on performance Testing Have your performance test approach evolved? What challenges did you face ? Test in Production? Put live first then continue to test in production? More frequent testing in BAU/Prod to establish consistent baselines? Less emphasis on process, documentation and sign offs? NFRs evolve as testing becomes less waterfall driven? Impact on Performance Test Assurance: How has your performance test assurance evolved? Have you been able to maintain the quality of performance testing deliverables when moving to CDM with shorter delivery cycles? New model of governance?

- Less testing & QA ? With more confidence in the quality due to smaller and shorter delivery cycles, fixing forward quicker or quicker rollback. Increase in the risk appetite with more tolerance of failures? Who drives the risk appetite / what is the Operations involvement? Better tooling to mitigate risk of less testing? Challenges ? Technical challenges (People skills..) Cultural challenges (Scare of change, of risks)