1. Click to edit Master title style nFocus Software Testing e-Innovation Centre Shifnal Road Telford TF2 9FT Tel: 0870 242 6235 www.nfocus.co.uk Notes MAY accompany this presentation, please select notes page view © nFocus Ltd 2009 all rights reserved. These materials can be reproduced only with nFocus’ official approval. Approvals may be requested via email – info@nfocus.co.uk Test Management Summit January 2010 Institute of Directors, Pall Mall London Sam Clarke - Principal Consultant nFocus Sam_Clarke@nfocus.co.uk Giles Davies - Developer Tools Technical Specialist Microsoft Giles.Davies@Microsoft.com How can I tightly integrate my testing into the Application Development Lifecycle?

2. 2 Agenda Introductions Why are we testing? Coping with change Demonstration of how Visual Studio 2010 enables integration of testing focussing on change Workshop - critical success factors © nFocus Ltd 2010 all rights reserved

3. Why should we test software? To see if it works To make sure it’s secure To show it hasn’t damaged other things To see if it’s reliable In fact there are many reasons depending on your role and professional interest. To see if it performs OK To show we can manage it To ensure we can install it To see if it’s usable To debug my program Confirm we meet regulations You can assert that testing: 1.Reduces risk of failure 2.Increases confidence of success 3.Supplies key information Objective Get the right level of quality, at the right price, at the right time Testing is not just about debugging programs! IT Systems are likely to have faults and you can’t test everything SME Challenges Budget, skills, critical mass © nFocus Ltd 2010 all rights reserved3

4. Fundamentally testing and quality assurance helps protect the reputation of the application, the business, the people and the company 4 ReputationApplicationBusinessPeopleCompany So where does testing fit in? Delivery as promised Minimal disruption Reliability as expected Press Share price Morale Pride Low attrition Early adopters Increased market share Minimised disputes Cost of maintenance as budgeted Profit as expected © nFocus Ltd 2010 all rights reserved

5. Testing adds value by reducing the risk of bugs in production, increasing confidence of a successful deployment and delivering objective information on the state of the system and the process by which it was developed Risk ConfidenceInformation Best Value Not delivering business benefits Damaging existing systems Implementing an unreliable system Developed to plan Complete Meets requirements Quality acceptable Quality of the system Quality of development process Residual risks after testing Constraints of: Cost Skills Resources Time Testing is a balancing act © nFocus Ltd 2010 all rights reserved5

6. However to fulfil these objectives testing must be an integral part of the Application Lifecycle Management Process from start to end delivering appropriate information that is applicable to all methodologies. 6 ConceptDesignDevelopDeployMaintainRetire Information – Building the right thing and building it right Test What does this mean for the project? Confidence of success and reduced risk of failure © nFocus Ltd 2010 all rights reserved

7. To deliver best value and key information, testing must have a repeatable process which integrates with the project and covering analysis, design, preparation, and execution of tests. Incidents can be raised at any time. Test repository Incident management system Project management repository Requirements repository Detect and record Incidents IssuesRisksDefects Information on the quality application and the dev. process Input to test (requirements, design, system, etc) Business and technical functional and non functional, Analysis Design and Specify tests PrepareExecuteLog results But something is missing...... Independent of methodology Code repository Designing and prototyping Stabilising Designing and building © nFocus Ltd 2010 all rights reserved7

8. Information can be difficult to provide if each of the test, project and development repositories are delivered by separate systems. This can be most apparent when trying to find the impact of a code change. Where is the data? In the head! Office tools Open source Proprietary systems 8© nFocus Ltd 2010 all rights reserved Quality of the application Progress Quality of the process Impact of change RisksTasks Dependenc ies Issues Process defects Application defects TestsTest logsChanges Specificati- ions ManualsCode Requirem- ents User stories What if something changes? What about unforeseen effects? Changes Tests Test logs Impact of change

9. Change is pervasive it effects reliability and testing has to deal with the issue of retesting function that has been affected. You cannot retest everything on each build so choice of tests by impact analysis is vital. Change is inevitable Agile approaches embrace change Development using Continuous Integration is delivering a continuously changing application Can’t wait until end of development to run tests Ideally all passed functional and system tests need to be run again on each build Need to quickly know impact of a change Change affects reliability increasing risk of damaging our Reputation Logistically we can’t retest everything after every change Automated regression testing helps by allowing lots of tests to be rerun in a short time Choice of tests to rerun can be difficult 9 So what do we retest after a change? © nFocus Ltd 2010 all rights reserved

10. If the Application Lifecycle Management System is fully integrated it is easy to provide consistent information about the impact of a change allowing informed decision to be made on risk, schedule and cost. 10© nFocus Ltd 2010 all rights reserved Quality of the application Progress Quality of the process Impact of change RisksTasks Dependenc ies Issues Process defects Application defects TestsTest logsChanges Specificati- ions ManualsCode Requirem- ents User stories Single integrated repository Consistent customised reporting

11. In summary to understand which black box tests have been impacted by an application change can be difficult. Knowing which code has been exercised by which test makes regression testing a much more effective and efficient process reducing risk and protecting reputation. Problematic subjective judgments taken Unrelated repositories (excel word, code bases) etc. Unknown code coverage with black box functional testing If you don’t know where you have been how do you know to go there again? What we need is an easy method and toolset to allow us to list tests that need to be rerun on a specific build of the system. 11© nFocus Ltd 2010 all rights reserved

12. Demonstration 12

13. 13 Features Code, unit test, build Fail Run smoke test and regression test Test error fixes and new function Report progress Update regression tests. Prepare next feature tests System test Errors Incremental development of the automated tests with constant rerunning of the test set increases confidence that the tested function remains intact and shows immediately if there are any adverse effects. Accept- ance test Production test Run regression tests update as necessary Which tests to run? © nFocus Ltd 2010 all rights reserved

14. Q&A Any questions before we move on to the workshop session Are you convinced that full integration of testing into the ALM process is necessary? 14© nFocus Ltd 2010 all rights reserved

15. Workshop objective is to share your knowledge and experience to understand critical success factors of implementing testing into ALM What do you see as your challenges in integrating testing into ALM life cycle in the following areas? Attitude to testing Application development and test process Application Lifecycle Management Tools Personnel - skills, Roles & Responsibilities, organisation structure Test management Reporting and metrics Test Environments and data © nFocus Ltd 2010 all rights reserved15

16. Workshop Critical Success Factors Rank the factors that MUST be fulfilled for success from previous lists Attitude ALM and process Test Technology (including ALM tools) Personnel and Organisation Test Management Reporting and Metrics Test Environments and Test Data 16© nFocus Ltd 2010 all rights reserved

17. 17 nFocus Ltd Consultants Automation Specialists Test Managers Test Analysts Process Outsource Tools www.nfocus.co.uk © nFocus Ltd 2010 all rights reserved