1. Post-Deployment Monitoring Is it part of Test?
Paul Gerrard
Technical Director, Systeme Evolutif
Systeme Evolutif Limited
9 Cavendish Place
London W1G 0QD, UK
Tel: +44 (0)

2. Paul Gerrard
Paul is the Technical Director and a principal consultant for Systeme Evolutif. He has conducted consultancy and training assignments in all aspects of project assurance, software process improvement and testing. Previously, he has worked as a developer, designer, project manager and consultant for small and large developments. Paul has engineering degrees from the Universities of Oxford and London, is Web secretary for the BCS SIG in Software Testing, contributed to BS and BS7925-2, host of the Test Management Forum and Founder Chair of the ISEB Software Testing Certification Board which has been taken by nearly 20,000 candidates.
He is a regular keynote speaker at both development and testing seminars and conferences in Europe, US and Australia, and won the "Best Presentation" award at EuroSTAR ’95 and "Best Presentation of 2002" award for the BCS SIGIST. Paul devised the term Project Intelligence for an innovative approach to results-based management supported by the disciplines of software testing.
He is Women’s coach for Maidenhead Rowing Club and webmaster and ISP administrator for several websites including
(IN PACK BUT TO REMOVE FROM DISPLAY VERSION)

3. What is Post-Deployment Monitoring?
After deployment, systems need to be monitored, like any production process
‘User Point of View’
Extranet – visible to the outside world
Intranet – visible to internal users
“Customer Perception”
‘Infrastructure Point of View’
Technically oriented measurement of availability, resource usage, queue length etc.
The “Vital Signs”

4. What can be monitored – user view
Functionality
Response times
Availability
Usability/Accessibility
Security
Etc etc.
In fact, most of the non-functional aspects can be monitored from a ‘users’ point of view
Tools and external services widely available.

5. What can be monitored – infrastructure view
System and network resources
100s of attributes can be monitored
Cpu, memory, page faulting, io, queue length etc.
Network latency, bandwidth, availability
Etc. etc.
Production support have had tools to monitor, log, raise alerts, do event monitoring for years
Not usually focused on the users experience, though.

6. Who does the monitoring?
Testers, having a more user-oriented view, believe they could monitor the user experience, if asked
Production/Support team, having long experience (and tools to manage the process) are already responsible
What should the testers’ role be?

7. Arguments against testers
Testing only superficially related to production monitoring
Some shared tools for observing and measuring
But production monitoring goes so much further
Includes outage detection, event management, call management, capacity planning, etc.
Post-deployment monitoring does not fit into test
Production support monitor and FIX problems
Testers should stick to the phase where they can add value – pre-deployment.

8. Arguments for testers? Who monitors outsourced hosted systems?
Hosting company will monitor infrastructure
But might not spot user outages
Hosting company are internally focused, not independent
Should ‘independent’ testers having some domain knowledge, experience of tools specify remote monitoring?
Done by a 3rd party?
Done by client’s IT people?

9. Why monitor? Monitoring is partly to detect failures
But it is really focused on change:
Changes to infrastructure disturbs performance, availability, resilience, security
Changes in user behaviour cause peaks in load
Growth in volumes load systems and slow them progressively as resources are over stretched
Skill of hackers increases risk of security vulnerabilities
Changes to functionality or design disturbs functionality, usability, accessibility, security (but shouldn’t it have been detected earlier?)
Some caused by projects (testers already involved)
Some caused by the passage of time.

10. In summary…
Testers come to monitoring from their point of view – is it really a testing issue?
If independent monitoring is required, should testers be involved?
Is monitoring really the same as regression testing?
Are testers involved because customers haven’t planned it right and have no one else to consult?
What is the role of testers in monitoring?