Application Problem Resolution The “Invisible Problem” or True Process Improvement Opportunity? Bob Levy Sr. Systems Engineer – Process Improvement ©2007 BMC Software

Problem Resolution – Manual Process The manual process is the way problem resolution has always been performed We MUST improve this process with automation Document / Gather Recreate Analyze Fix Automated Virtually Eliminated Accelerated Unchanged ©2007 BMC Software

Why is Problem Resolution so Difficult? Problem Resolution is all about getting root cause 80% 20% ROOT CAUSE ANALYSIS FIX GATHER  RECREATE  ANALYZE Because symptoms rarely reflect the root cause, the process of root cause analysis is manual, based on trial-and-error and therefore highly inefficient and time-consuming. ©2007 BMC Software

Root Cause Analysis Stages Gather Information Recreate Problem Analyze Information Problem information gathering is manual and consumes a very long time Information includes screen prints, textual descriptions, etc. Incorrect problem categorization results in gathering the wrong information Problems must be recreated on-demand, multiple times throughout the process Often, symptoms are recreated rather than true root cause Every re-creation is a manual process that takes a long time Based on categorization of the problem Multiple iterations are often required for obtaining additional information Multiple specialized tools are often used ©2007 BMC Software

Problem Resolution in Pre-Release Workflow Testing Development   - Problem is uncovered during a test Problem information is gathered and documented in a defect tracking tool  - Developer receives a defect report - Requests additional information  - Tester/user recreates the problem - Gathers additional information  - Developer requests different data  - Recreates the problem - Gathers additional information  - Recreates the problem - Analyzes the problem - Fixes the problem - Delivers a patch ©2007 BMC Software

Problem Resolution in Maintenance Workflow Support/Operations End-Users/ Production   A problem is reported to support/operations  Support/operations engineers gather information and try to resolve the issue  Developers analyze the problem, determine root cause and fix it  Most problems are resolved – small percentage of problems are escalated to development  Development communicates with support/operations multiple times to gather information and attempt to recreate the problem Development ©2007 BMC Software

Proposed Solution: A black box application recorder like the blackbox on an airplane... ©2007 BMC Software

Proposed solution to Application Problem Resolution Root Cause Analysis Capture full problem information (like a black box on an airplane) Eliminate the need to manually gather Information Eliminate need to recreate problems or problem environments Eliminate need to access production data Becomes a standard part of the process Provide 25%–35% efficiency to the development organization ©2007 BMC Software

Pre-Release Problem Resolution Automation Testing Development Test Lab  integrated into defect tracking process   records all problem information during test execution  - Developer receives a defect report Opens the information in recording - Analyze without recreating  - Developer fixes problem ©2007 BMC Software

Maintenance Workflow with Automation Support/Operations End-Users/ Production  A problem is reported to support/operations  Support/operations engineers gather information and try to resolve the issue  A very small percentage of problems are escalated to development  Development activates recording and retrieves problem log Development  Development analyzes recording, determines root cause and fixes the problem ©2007 BMC Software

Problem Resolution Automation Summary Eliminate problems recreation Eliminate non-reproducible problems Reduce back and forth Integrate with existing defect/problem process Deliver consistent and accurate and complete information Deliver improved quality Deliver improved customer satisfaction Deliver faster time to market My contact information Bob Levy blevy@bmc.com 708.205.5373 ©2007 BMC Software