Optum DTR Process Overview

Agenda UHG Application Tuning DTR: The Foundation of UHG Tuning Efforts Key Participants DTR Lifecycle UHG Results The Secret Sauce Areas of Ongoing Focus Questions/Discussion

UHG Application Tuning Program Business demand and transaction volumes grew with no end in sight Two Choices: Throw ever-increasing amounts of hardware at the problem, often to no avail Tune high-value, high ROI consumers In 2004 UHG chose the latter, and from this decision the DTR program/partnership was developed.

DTR: The Foundation of UHG Tuning Efforts In response to corporate mandate to control CPU consumption, a process was developed by which tuning actions could be created, reviewed, tested and implemented. The Detailed Tuning Recommendation (DTR) program was the output from this exercise. A DTR is a tuning recommendation created for application jobs, transactions, SQL, or system resources for the purposes of: Reducing Business Chargeback Improving Response Time Improving Cycle Time (Batch Throughput) Reducing Abends/Contentions Decreasing system overhead and/or peak hour demand Summarized: Improve efficiency of UHG assets Standardized format makes for easier processing DTRs are intentionally detailed for the purpose of providing clarity for the recommended change so as to minimize the amount of discovery work needed by the application team The process also includes a reporting database (compiling DTR savings & costs) containing all DTRs for current year as they progress through the DTR life cycle. Every DTR is assigned a unique serial number so its progress can be tracked.

Key Participants APM Team (Tuner): Identifies the tuning opportunity Writes the DTR (Detailed Tuning Recommendation) Supports the Application team throughout development lifecycle Application Team: Validate the recommendation If funding is needed, estimate the work and gain approvals Complete any additional analysis needed to confirm scope/impacted components Complete necessary build/development activity Complete necessary testing Schedule and execute implementation activities DTR Administrator: Maintains inventory of all DTRs Tracks DTRs throughout lifecycle Measures and publishes tuning results Creates scorecards for aggregated results Other Contributors: System teams (CICS, DBA, Capacity Management, etc.): Identify hotspots or areas of interest Business teams: Communicate SLA issues and/or areas of growth Problem Management: Issues that require performance attention

DTR Process Workflow Opportunities are identified in several ways: Identify Opportunity and Create Tuning Recommendation Application Implementation Measurement Publish Results Opportunities are identified in several ways: Application team has a performance issue where tuning expertise is needed Problem identified as a result of a Production Incident Instrumentation/exception reporting identifies a situation where tuning could address consumption or performance Create Tuning Recommendation: APM team creates Detailed Tuning Request (DTR) that outlines suggested changes that will address opportunity

DTR Process Workflow Application Implementation: Identify Opportunity and Create Tuning Recommendation Application Implementation Measurement Publish Results Application Implementation: DTR information is forwarded to the Application Service Manager Application Team: Assesses DTR for effort and timing Request funding if needed (ROI is a significant factor in decision to fund) Keep DTR Administrator updated on progress Schedule and implement DTR Confirm implementation

DTR Process Workflow Measurement: Identify Opportunity and Create Tuning Recommendation Application Implementation Measurement Publish Results Measurement: DTR Administrator gather performance metrics across a standard measuring period following implementation (usually two weeks).

DTR Process Workflow Publish Results: Identify Opportunity and Create Tuning Recommendation Application Implementation Publish Results Measurement Publish Results: DTR Administrator team notifies to interested stakeholders with the results of the DTR and updates annualized totals

UHG Tuning Results – Executive View Chargeback savings roll-up with breakout by: DTR Phase Area/type of savings Tuning results reflect: Traditional success in tuning DB2 usage Active work to manage DB size through purging, archiving, etc. Distributed-related chargeback for storage and memory

UHG Tuning Results – Application View Ability to report results by Application

UHG Tuning Results – Inception to Date Since inception of DTR program in 2004, over 1900 DTRs have been implemented with cumulative compounded CPU savings of +56 million CPU hours. We would require double the current amount of mainframe capacity without the DTR program.

The Secret Sauce Enterprise Support for program: Proven results Partnership with Application, Systems and Business teams to identify and implement changes Funding: Common pool of funding established to remove “no money to tune” barrier from the application teams while maintaining centralized control and visibility into where tuning efforts are directed. Business willingly funds because they receive 17:1 ROI – for $1 spent on tuning their chargeback bill is reduced by $17 (average ROI 2004-2017) Lightweight Process APM vs. Application roles: The APM team makes a recommendation; the Application team maintains ultimate ownership of their code Defensible measurements with supporting documentation The APM Team: Singularly focused on Application performance Consolidated into one team, shared learning and information Highly skilled former developers with deep system and operations knowledge Flexible – not dedicated to a single app; can “run to the fire”

Areas of Ongoing Focus Aging DTRs: Improved Predictability of Outcomes As inventory ages, recommendations run risk of becoming obsolete Improving internal processes to keep inventory fresh and relevant Improved Predictability of Outcomes Ensuring All Tuning is Captured via DTR “If no DTR then the savings did not happen” Non-Chargeback Metrics Most current aggregated reporting is mainframe and chargeback-based (CPU or Storage) Opportunities exist to improve how we produce and report aggregated results for: Abend/contention reduction Response time improvements Distributed environmental measurements Improved/Common Access to DTR Information Enhanced partnership with Application teams: Performance assessment/tuning earlier in SDLC Business Logic

Questions/Discussion