Does Change Management Include Patches? Joel Howard, RingMaster Software Northern California OAUG San Ramon 2004

Agenda 1.Change Management 2.Answer the Question 3.Realities of 11i 4.APM - a Change Mgmt Solution

What does Change Management cover? Upgrades – Software Upgrades – Hardware Document Management Configuration Management Software Development Maintenance

Change Management Why is Change Mgmt important? Disparate components share a relationship to one another… SHARED DEPENDENCIES!!

Changes…Dependencies Maintenance Pack >> Apache Pre-reqs Apache upgrade >> DB Pre-requisites One off patch >> Pre- and Post-requisites Apps Upgrade >> DB Upgrade Does CM include patches? YES !!

The biggest, most complex release ever – Release products – Release products – Release products – Release products – Release products ALL OR NOTHING! Release 11i Dependencies

– Release > 20,000 bug fixes – Release > 40,000 bug fixes – Release > 64,000 bug fixes – Release > 100,000 bug fixes Release 11i

What are the two most difficult issues in maintaining software? 1.Patching 2.Not Patching Not Unique to Oracle

The Impact of Patching Problems Resolved New, more stable technology Added Functionality Improved Security Faster Performance Who is impacted?EVERYONE.

The Impact of Patching Large amounts of new code introduced Exact Impact unknown…undocumented distinct bug fixes in an average family pack 2-50 different modules impacted User unaware of changed processes/attributes Disabled forms/reports Extensive testing Disruption of production system

The impact of NOT patching Why not patch? Stability… Perceived From Metalink FAQ: “At a minimum, apply maintenance packs to stay within two maintenance releases” of the most current release. Timely support: Call Oracle support for a problem and you’re not patched current? Performance Latest performance updates as part of a patch.

The impact of NOT patching Security Latest security updates: issued in the form of patches New Functionality Improvements in functionality and stability Politics User complaints, frustration over a lack of fixes, requests for added functionality

Why avoid it? Multiple manual steps per patch, per environment No detailed impact analysis No structure to enforce pre-requisites Inability to flag affected customizations No mechanism to manage testing, approvals or workflow

Manual Steps Download Patch Transfer Patch UnZip Patch Review “Readme” Issue search Review Drivers Parse Actions Compare Envir’s Apply to test Transfer Patch Server Login UnZip Patch Prepare Envir. Review Log File Review Invalid DB Objects File System Impact Database Impact Customization Impact Create Summary Update Spread Sheets Check Applied Patches Run Patchsets.sh Each Tier Run Adpatch Each Driver For Each Language Run Adadmin Run Adctrl Run Datainstall Find Pre-reqs Maint Pack Download Change Testing Maint Pack Research Maint Pack Application Change Reporting

Strategy 1.AUTOMATE the maintenance of your applications 2.Patch Proactively: Stay current on your watch… not Oracle’s Software issues are typically encountered when you least have time for them –Month end (Mgmt reports due, Payroll filing) –Quarter end (SEC Reporting) –FDA or other certification requirements –Year end (1099s, W-2s, Sarbanes Oxley) Not patched current = major project when you can least afford it

APM A Change Management Solution Identify the change Control the change Communicate the change

APM Real-time analysis by APM  file system impact (APPL_TOP)  database impact (table,view,sequence,index)  list of affected customizations  required prerequisite patches  is the prereq obsolete?  determine “patch impact ratio”  summary of bug numbers in patch  links to Oracle bug database  list environments where patch/bug already applied  Patch Prediction Change requested Research performed on Metalink Patch obtained Patch impact analyzed Dependency information obtained Knowledge disseminated Implementation planned Plan implemented Testing performed Patch Approved Patch Migrated Concurrent Documentation Change Management

APM Automatic scheduled download of patch from MetaLink Notification of obsolescence Enforcement of pre- requisites Establish pre-reqs Automate manual tasks Change requested Research performed on Metalink Patch obtained Patch impact analyzed Dependency information obtained Knowledge disseminated Implementation planned Plan implemented Testing performed Patch Approved Patch Migrated Concurrent Documentation Change Management

Automated processes within APM patch download transfer patch to multiple servers schedule single patch or patch group merge patches in background ADADMIN, ADCTRL, sqlplus/shell scripts compile / list invalid objects start/stop Oracle processes /pager notification Change requested Research performed on Metalink Patch obtained Patch impact analyzed Dependency information obtained Knowledge disseminated Implementation planned Plan implemented Testing performed Patch Approved Patch Migrated Concurrent Documentation Change Management

Workflow / Approval Process PROD Test Patch TEST1 DEV2 DEV1 TRAIN DBA No STOP Yes Ok? DBA Yes DBA Yes Ok? Yes

Change requested Research performed on Metalink Patch obtained Patch impact analyzed Dependency information obtained Knowledge disseminated Implementation planned Plan implemented Testing performed Patch Approved Patch Migrated Concurrent Documentation APM Automatic Alerts: s, pager, etc. Central Repository for all environments for all patching information Dozens of useful reports Environment comparisons, patch detail, patch impact analysis, patches applied by date, by environment, by application, by person, etc. find patchset levels, find where patch was applied, compare your patches against those released from Oracle Change Management

Application DBA Functional Analyst Patching Information Information Flows

Environment Comparison

The Impact of APM Functional Group Reduced, focused user testing Verify testing took place Faster support from Oracle Technical Group Reduced workload through automation Reduced risk through impact analysis Complete, accessible audit trail Faster problem resolution Organizational Effect Enforced procedures Decreased workload Communication, accountability between business groups Improved problem resolution Reduced costs

Oracle and RingMaster , Oracle Application Manager (OAM) Improved Reporting No automation, scheduling, workflow No pre-requisite checking No mechanism to manage testing or approvals No detailed impact analysis No environment comparison reports , Oracle Application Manager (OAM) Improved Impact Analysis on file system only Pre-requisite checking No automation, scheduling, workflow No mechanism to manage testing or approvals No impact analysis for DB changes No environment comparison reports

1995: Founded as a consulting organization with expertise in Oracle Applications 1998: Began developing and selling software specifically for Oracle Applications Users Offices in Boston, Dallas, Los Angeles, Atlanta, Chicago, St. Paul, and Vermont More than 100 customers in the US, Canada, and Europe RingMaster Software Corporation

Representative Customers