HP QuickTest Professional פרויקט גמר בניהול מערכת תפעולית מגיש : אבישי בן שבתאי ת " ז :

Agenda Short Introduction Arena Tier 3 support process Description and analysis of the process Introduction of main problems Define key performance indicators for successful improvements of the process Suggest improvements to process

What is Functional Testing? Testing software based on its functional requirements. It ensures that the program physically works the way it was intended and all required menu options are present. It also ensures that the program conforms to the industry standards relevant to that environment; for example, in a Windows program, pressing F1 brings up help.

How Does QuickTest Help? When performing functional testing by automatic scripting you get: Scenarios are easily reproduced. More coverage since it can run 24 hours. Regressions are easily detected. More accurate verifications.

Legend Horizontal Integration Vertical Integration Company Products Customer

Arena SAPOracle Oracle and SAP sells their products with Tests suite for their products that can be run under QuickTest Professional. Service Test is another testing tool that is developed in HP Software which Perform functional testing on SOA systems and is sold with QTP as unified functional license testing tool. QuickTest Professional Service Test IT Orgnization

Market Analysis

Tier 3 Support Process Tier 2 Creates an escalation R&D Analysis R&D Creates DLL with Solution R&D creates patch QA performs testing on verification Customer Verification ~3 Days ~1 Day ~0.5 Day ~2 Days

ISO 9000 ISO 9000 is a standard that says that the process needs to be defined and enforced. Currently HP SW Tier 3 support process, that was described, meets ISO 9000 standard. The maturity of the process is 4, since it is currently the 7 th year that it is enforced and fine tuning is done at the beginning of a new QuickTest Professional version release.

6σ6σ It is hard to talk about variety in the process since each customer has its special case and each case is resolved by different R&D engineer. We can pin point a few problems with the current: a) Cases that are “forgotten” to be handled by TIER 2 or TIER 3 Suggestion: Monitor engineers by their response time when handling cases. b) Cases that are reassigned to other R&D engineer since the first analysis where the problem resides was wrong. Suggestion: Add a logging system to the product and teach the engineers how to pin point the problem from the log.

Performance When talking about the performance of our support process we take into consideration the following parameters: Productivity: each customer pays about 3,000$ for support license. Assuming that each customer has approximately 4 support calls that handled by Tier 3 (R&D) when each support cycle is about 5 working days, the cost is about 300$ per day. It comes that our profit is about 1500$ per support license. Improvement: according to the process, reducing the maximum time spent on analysis by half can increase the profit by 30%. This can be done by the following: a) creating a simple diagnostic program that generates a report. b) fixing our logging system. c) teaching our TIER 2 support to analyze the logs.

Performance cont. Overhead – Since there is no direct communication between customer and R&D engineer, most of the time is spent on cycles between R&D engineer and Tier 2 support person. Improvement: Performing WebEx with customer to demonstrate his problem. First analysis can reduce cycles and help with analysis stage. Response Time – When tier 3 escalation is created it takes about 7 working days to close it. But the cycles between R&D engineer and customer increase a time to close the tier 3 escalation to about 20 days. Improvement: With the above suggested improvements it is possible to reduce the response time by 50%.

Performance cont 2. SLA – about 84.5% of the TIER 3 escalations have resolved in the given SLA. Improvement: With the above suggestions we can increase the above number to 92%. Quantity – Each product version has about 320 Tier 3 escalations and in every moment there are about 70 cases which are opened. Each R&D engineer has about 2 escalations that he is currently working on. Improvement: more regression testing in QA and working with more real world application, when performing the QA, will reduce the number of total escalations that needs to be handled by tier 3. This reduction will be by 25%.

Performance cont 3. Quality – in this indicator we speak about to aspects: Robustness – the quality of the robustness of the process is measured by how many Tier 3 escalations are not met by the SLA, this number is currently about 15.7% of all Tier 3 escalations. Improvement: with the above suggestions we can reduce the number of errors in the process to less than 10%. Agility – there are small number of cases where a customer needs a special treatment to solve his problem either by working directly with R&D or by increasing man power. This number is rather smaller (less than 5%). Improvement: no improvement needed here.

Problems in current process Since the number of cycles between R&D and TIER2 is the main cost of the process reducing that time can provide much more profit. The following problems in the process cause many cycles: a) Reassignments to between R&D engineers b) Problematic logging system. c) Escalation was not justified.

Pareto Analysis on Problems

Ishikawa Analysis Lack of knowledge in Analyzing Logs Lack of DB to store Customer’s Logs Problematic Logging System Not all Scenarios are logged No R&D enforcement for Writing to Logs

Actions to be taken After a short discussion we have concluded that the following steps should be taken: ResourceTaskTime Technical ArchitectResearch for new logging system 2 weeks IT managerCreate DB for logs1 week R&D Team leaderInstruct R&D engineers with to consider logs in their code review 1 week L&D Team leaderCreate a new course for analysis of logs 4 weeks

Changes to the arena With the introduction of the WEB 2.0 the support process became complex because of the following: a) Variety of toolkits. b) Toolkits are open source and it is hard to keep track with changes in them. That is why HP has decided to create a taskforce that needs to provide methodologies and patents to solve the problems introduced by Web 2.0.

References ctional+test&i=43588,00.asp ctional+test&i=43588,00.asp All the information provided in this presentation is taken from HP Software and is used for the academic purpose only and should not be forwarded or used in any other way.