©Test Management Forum 2006Slide 1 Post-Deployment Monitoring Is it part of Test? Paul Gerrard Technical Director, Systeme Evolutif Systeme Evolutif Limited 9 Cavendish Place London W1G 0QD, UK Tel: +44 (0)

©Test Management Forum 2006Slide 2 Paul Gerrard Paul is the Technical Director and a principal consultant for Systeme Evolutif. He has conducted consultancy and training assignments in all aspects of project assurance, software process improvement and testing. Previously, he has worked as a developer, designer, project manager and consultant for small and large developments. Paul has engineering degrees from the Universities of Oxford and London, is Web secretary for the BCS SIG in Software Testing, contributed to BS and BS7925-2, host of the Test Management Forum and Founder Chair of the ISEB Software Testing Certification Board which has been taken by nearly 20,000 candidates. He is a regular keynote speaker at both development and testing seminars and conferences in Europe, US and Australia, and won the "Best Presentation" award at EuroSTAR ’95 and "Best Presentation of 2002" award for the BCS SIGIST. Paul devised the term Project Intelligence for an innovative approach to results-based management supported by the disciplines of software testing. He is Women’s coach for Maidenhead Rowing Club and webmaster and ISP administrator for several websites including

©Test Management Forum 2006Slide 3 What is Post-Deployment Monitoring? After deployment, systems need to be monitored, like any production process ‘User Point of View’ – Extranet – visible to the outside world – Intranet – visible to internal users – “Customer Perception” ‘Infrastructure Point of View’ – Technically oriented measurement of availability, resource usage, queue length etc. – The “Vital Signs”

©Test Management Forum 2006Slide 4 What can be monitored – user view Functionality Response times Availability Usability/Accessibility Security Etc etc. In fact, most of the non-functional aspects can be monitored from a ‘users’ point of view Tools and external services widely available.

©Test Management Forum 2006Slide 5 What can be monitored – infrastructure view System and network resources 100s of attributes can be monitored – Cpu, memory, page faulting, io, queue length etc. – Network latency, bandwidth, availability – Etc. etc. Production support have had tools to monitor, log, raise alerts, do event monitoring for years Not usually focused on the users experience, though.

©Test Management Forum 2006Slide 6 Who does the monitoring? Testers, having a more user-oriented view, believe they could monitor the user experience, if asked Production/Support team, having long experience (and tools to manage the process) are already responsible What should the testers’ role be?

©Test Management Forum 2006Slide 7 Arguments against testers Testing only superficially related to production monitoring Some shared tools for observing and measuring But production monitoring goes so much further – Includes outage detection, event management, call management, capacity planning, etc. Post-deployment monitoring does not fit into test Production support monitor and FIX problems Testers should stick to the phase where they can add value – pre-deployment.

©Test Management Forum 2006Slide 8 Arguments for testers? Who monitors outsourced hosted systems? – Hosting company will monitor infrastructure – But might not spot user outages – Hosting company are internally focused, not independent Should ‘independent’ testers having some domain knowledge, experience of tools specify remote monitoring? – Done by a 3 rd party? – Done by client’s IT people?

©Test Management Forum 2006Slide 9 Why monitor? Monitoring is partly to detect failures But it is really focused on change: – Changes to infrastructure disturbs performance, availability, resilience, security – Changes in user behaviour cause peaks in load – Growth in volumes load systems and slow them progressively as resources are over stretched – Skill of hackers increases risk of security vulnerabilities – Changes to functionality or design disturbs functionality, usability, accessibility, security (but shouldn’t it have been detected earlier?) Some caused by projects (testers already involved) Some caused by the passage of time.

©Test Management Forum 2006Slide 10 In summary… Testers come to monitoring from their point of view – is it really a testing issue? If independent monitoring is required, should testers be involved? Is monitoring really the same as regression testing? Are testers involved because customers haven’t planned it right and have no one else to consult? What is the role of testers in monitoring?