1. Oracle Fleet Management: Patch and Upgrade Your Databases
Gary Henderson
Vaithianathan Soundararajan
Good afternoon
You are in the Oracle Fleet Management session where we are going to talk about patching and upgrading databases at scale ... IE get allot done with few people, with minimal downtime and minimal risk.
I am Gary Henderson and this is my co-worker Vaithianathan Soundararajan.
We Both work for

2. Nationwide Insurance and Financial Services Company
Fortune 100 Company
Founded in 1925
43 Billion in Revenues*
221 Billion in Assets*
33,135 Employees*
#53 on Fortune’s "Best Companies to Work For"
#27 in Computer World’s "Best Places to Work in IT."
*Fortune 500
Nationwide is an Insurance and Financial services company
Headquartered in Columbus Ohio
Nationwide is 66 on Fortune’s list of top 100 companies
43 billion in revenues last year
And #53 on Fortune’s "Best Companies to Work For" list,
#27 in Computer World’s "Best Places to Work in IT."
Vaithianathan and I work as part of the Oracle DBA team
an experienced team of about 20 individuals
So before we begin … let me get a feel for who’s here.
Who in the audience is or has been a DBA? …
Who has Real Applications Clusters (RAC)?
that’s a real joy to patch
How many have heard about OEM Fleet Maintenance?
Anybody currently using it?

3. Agenda Why Patch? Our Configuration Patching Alternatives
Our Constraints
Our Results
How We Did It
Our Timeline of Progress
Summary
Questions
In this session we’ll cover
Why patching is important
Nationwide’s configuration
The existing patching alternatives
Nationwide’s patching constraints
The results we’ve had for the past 3 years
How we accomplished it
The progress we’ve made over the years.
Then summarize and open the floor for questions.
So Why is patching important and need done?

4. Security..Security..security
Center for Internet Security (CIS)
1 Oracle Database Installation and Patching Requirements One of the best ways to ensure secure Oracle security is to implement Critical Patch Updates (CPUs) as they come out, along with any applicable OS patches that will not interfere with system operations.
 1.1 Ensure the Appropriate Version/Patches for Oracle Software Is Installed Profile Applicability:
• Level The Oracle installation version and patches should be the most recent that are compatible with the organization's operational needs.
Rationale: Using the most recent Oracle database software, along with all applicable patches can help limit the possibilities for vulnerabilities in the software, the installation version and/or patches applied during setup should be established according to the needs of the organization. Ensure you are using a release that is covered by a level of support that includes the generation of Critical Patch Updates.
In these days of security breaches security has become all important.
The Center for Internet Security (CIS) is a global standards and best practices organization with guidelines on securing IT systems and data against attacks
This excerpt is verbatim from their Oracle benchmark
Most all security expertise list currency patching as A top priority to ensure IT security.
Besides the remediation of vulnerabilities .. This patching would also be improving performance and stability.

5. Oracle’s Annual Release Roadmap Doc ID 742060.1
This graphic shows Oracle’s roadmap for future versions.
If you look near the bottom one thing is apparent A new version every year.
These new versions may necessitate an upgrade... And these upgrades may come as fast as yearly
And recently announced
NOTE: Oracle Database 18c will be supported for two years after Oracle Database 19c platforms have released on premises. Oracle Database 19c is the Long Term Support Release or “Terminal Patch Set” for Oracle Database 12.2.
Typically in the past oracle supported versions for 3 years.
So not only are new versions coming faster, they may be supported for less time.

6. 2014 Clustering Technologies vs DB Versions
This represents what our environment looked like in 2014.
This is about 1200 databases spread across about 400 db servers
Each Clustering Technology may present its own patching challenges
Reduce/Eliminate those that don’t provide strategic value/capability
Over the next three graphs … Notice RAC 1 Node clustering starting at 17%.
While our DBA’s and customers love the functionality of RAC 1 Node ... It is so much more difficult to maintain and patch than other database types.
The number of database versions you support is also an important factor. You should try to stay on supported “CURRENT” versions
Also …. PAY close attention to database versions and
… notice they were each 1% of the population as of 2014.

7. 2016 Clustering Technologies vs DB Versions
This represents our configuration in 2016
Many of these database upgrades were accomplish with OEM’s deployment procedures ....wizards.
Notice that the variety of environments and more importantly database versions is reducing.
This variety in your environment make scripting solutions more challenging and removes focus from your central offerings.
Less variation and fewer versions will be more consistent, more secure, and easier to patch.

8. 2018 Clustering Technologies vs DB Versions
And here is our current picture.
Notice how Rac 1 Node has become our primary offering with 79% and db version at 90%.
Also notice that database version 18.3 appears. This is our next major database version.
I would expect non-clustered solutions to rise in the years ahead with Cloud offerings and on premise virtualization become a new focus.
So getting the db environment more consistent is both beneficial to the patching effort .. And the patching effort will help attain more uniformity.
Before we get into our results, I think it is important to layout some of the major offerings available to patch your databases.

9. Patching Alternatives
Opatch
In place
Challenging error recovery
Longer Outage window
All databases in the Home have to be patched together
Rollback challenging
Requires less storage than out of place options
*no extra licensing
Multi-tenancy
Out of Place
Unplug from old and plug into updated/patched
May require double the memory resources
*Multi-Tenant license required
Rapid Home Provisioning
Gold Image Homes/ Standardization
Minimal Outage window
*LifeCycle License Required?
OEM Fleet Maintenance
Version 1: Switch
Version 2: Db software maintenance
Gold Image Homes / Standardization
Utilizes OEM’s deployment job system
Scalable
*LifeCycle License Required
Cloud DBaaS
Cloud Provider applies Patches
Patching Alternatives
Opatch ... I think most of you will be familiar with this offering ... If you were happy with this option .. I doubt you would be here. This is the only offering that does not have some licensing or cost associated to it.
Multitenancy .. While not primarily a patching vehicle .. The ability to unplug and plug into an updated or upgraded container database does give you an alternative.
- remember if you plug multiple databases into a CDB .. Licensing needs considered.
The next two options (Rapid Home Provisioning and OEM Fleet Maintenance) are very similar to each other and if it was not for several of the benefits of OEM fleet we might have gone with Rapid Home Provisioning.
Both are out of place patching that utilize gold image deployments and a quick switch from the current home to the new patched codeset.
And both require the Lifecycle Management Pack license to fully leverage there capability.
OEM Fleet Maintenance is our currently patching method and what this session will concentrate on
Cloud Provider Database as a Service. This option has you offload most of the DBA responsibility to a cloud provider and they perform the patching.
At this point I will let Vaithianathan explain some of the constraints we’ve faced, and our patching results.

10. Quarterly Patching Windows Switch/Update
Each Quarter patch to previous quarter’s PSU
Development patched during Wednesday window
Using a team of three:
DBA – Db patcher
DBA – Clusterware patcher/Tester
OEM - Administrator
Production patched during IRW window
Our Patching model is “Quarterly Cycle”.
Each quarter we patch Databases  to previous quarter’s PSU.
We use a Team of 3 individuals,  1st Team Member responsible for Patch Schedules and Database Patching, 2nd Team Member responsible for Testing Enterprise Manager, Cluster Patching and Gold Image Creation
and the 3rd Team Member responsible for Enterprise Manager Administration.
 We patch or Dev, test, and PT databases on wednesdays
Our Company limits production changes to 1 weekend per month (aka ..Infrastructure Release Weekend)
Our window is 6 hours beginning midnight/early morning … which we share with Linux Team.
The Average number of Prod. Databases patched during IRW was 145 (including Standby) and Maximum was 284 (including Standby).
Yesterday was our October IRW and it was a light patching schedule,  we patched 16 Clusters and 46 Databases.
Our Goal is to reduce and Standardize Patch to as few windows as possible.

11. Patching Results Windows Success % # Dbs Tot Exec Time (Hrs)
Window Time (Hrs)
2016:Devl 31
100
693
98.15
38.35
2017:Devl 44 99
1080
160.67
52.77
2018:Devl 42
1575
183.56
48.14
2016:Production 12 98
340
44.36
18.34
2017:Production
706
79.38
20.85
2018:Production 10
1000
105.35
18.24
Prior to using OEM to Patch .. We attempted to patch all databases using scripted opatch with 10 DBA’s within a year  ... We only got about 70% done.
This tables shows our results for the past 3 years using 3 part time resources
 The average time to patch a database is btween  7-12 minutes
We became comfortable using this method ,  so,  we patched half of our Infrastructure every quarter in 2017 … covering all Dev. and Prod. Databases.
Looking at our results during 2017,  our management asked us if we can patch all the Dev. and Prod. Databases  in one Quarter  (Reason for more # of Dbs. In 2018)
and we were able to patch all Dev. and Prod. Databases during 2nd Quarter of 2018.
We never  ran over a “window” … Most of the time  … we were able to complete under 3 hrs.
Failures – Are mostly operational (Example : During Cluster Patching ...  Oc4j process don’t  shutdown/Stop ...
Another Example … During  Post Database Patch step ... Utlrp” takes more time or it may appear to be hanging/hung  ... 
In both cases/situation … we fix the problem manually and retry the failed step on OEM).
We always monitor the database patching progress using “Cloud Control -> Procedure Activity” .
With that I turn the presentation back to Gary.

12. OEM Fleet Maintenance (EMCLI)
Version 1: Switch
Version 2: Db_Software_Maintenance (Update)
Switch_database
SWITCH_GI
createSoftwareImage
subscribeTarget
checkApplicability
performOperation
DEPLOY_GI_SOFTWARE
UPDATE_GI
DEPLOY_DB_SOFTWARE
UPDATE_DB
DEPLOY_RAC_SOFTWARE
UPDATE_RACDB
DEPLOY_CDB
ATTACH_CDB
ROLLBACK_DB …RACDB ..GI
CLEANUP_SOFTWARE
You may have already picked up that this capability exists in OEM but is currently only exposed via the Enterprise Manager Command Line Interface (EMCLI)
We have been using the Switch options since they were first available in 2016.
The switch operations solely move you from one Oracle home to the successor home you specify. Actually it launches a deployment procedure you can monitor in the procedure activity page of OEM.
But last quarter we moved on to Db_software_maintenance
Our adoption was delayed due an issue with deploying homes for RAC 1 node databases. .. As I’ve said before RAC 1 Node is unique.
DB_Software_Maintenance allows gold image creations
It has operations to deploy the homes
Switch or update to the new home
Clean up the homes when they are no longer in use
Batch multiple deployments or updates into a single command (target_list).
**As part of the update you can upgrade to new db versions.
All these operations launch deployment procedures that can be monitored in OEM..
Fleet Reference:

13. Software Standardization Advisor
One of the first steps in this process is to looking at your environment and assess where you are. The software standardization advisor does a good job of this. It is well hidden ..
under the targets drop down .. You go to the database page
once there, under the administration drop down you have the software standardizations advisor.
Now this slide is a little embarrassing for me .. As 26 unique configurations is allot.
but a single patch on a home will cause a unique configuration.
these 26 configurations include 9 configurations that are not hosting any databases ...
kept for restoring old versions.
testing and lab servers
92% of our databases are spread among 4 unique configurations. Which is something I’m proud of.
This page acts as a good grade card to see how many unique configurations you have... How consistent are your oracle homes ... How far have they drifted from your standard install.

14. DB Software Maintenance
Create Gold Image/Version
createSoftwareImage
Prior Quarter
Associate Target to Corresponding Image
subscribeTarget
One time (two weeks prior)
Deploy new Oracle Home
DEPLOY_RAC_SOFTWARE
Two Weeks prior
Patch Database moving it from the old to the new patched home
UPDATE_RACDB
When the old home is no longer used .. Remove it.
CLEANUP_SOFTWARE
Patching is cyclic and DB Software maintenance supports that cycle.
After you have assessed you environment and downloaded the latest patchset, you patch to create a gold install.
How many of you are familiar with using gold images for agent deploys?
... This follows that same model. Studying the agent gold image model helped us understand db software maintenance better.
You begin the cycle by using createSoftwareImage to capture the gold image
You subscribe databases to the gold image that corresponds to it’s configuration
You deploy the gold image software to the databases server ... Oem deploys it based on the target type (database_instance or rac_database) . If multiple databases share the home, you only have to deploy it to one of them.
When you issue an update on the database it will patch or update it based on the image it’s subscribed to.
Once all the databases have been updated to the successor software home you issue a cleanup_software for the database to properly remove the old unused software from the servers and OEM.

15. Gold Image Version – Lineage/Swim Lanes
12.1 DB Standard RAC
Ver
11.2 DB Standard RAC
12.1 Grid Standard RAC
12.1 DB Standard Stand-Alone
Ver
11.2 DB Standard
Stand-Alone
12.1 Restart Standard Stand-Alone
12.1 DB
One-Off RAC
Ver
One-Off Stand-Alone
Ver
Ver ??
So you keep hearing about gold images ... But how many .. Answer .. As few as you can get away with. The Fewer the better.
So lets look at my set up.
When I look at my RAC environment database code, database code, and clusterware.
So I also have a non-clustered environment ... That doubles the number of gold images.
And at this point you say ...I have a few customers that have one-off requirements. They would never let me move them to a standard image ... And I don’t want to put their one-offs in all my homes.
Oracle’s and my recommendations is try for a single standard install ... But if you can’t as we couldn’t ... Try to combine all the one-offs into one standard one-off. If a customer has a one-off requirement ... They have to consume all the one-off’s. If the one-off they require is incorporated in a future PSU .. You unsubscribe and SWITCH_DATABASE them back to the standard home.
BUT BE CAREFUL of moving between image lanes.
So when the next PSU or Release Update/Revision is available .. You patch your gold install and new version is captured ... And when you start using it you make it current. And start the cycle again.
Databases that stay within their image lane do not have to be subscribe again.
And next quarter ... Another version.

16. This screen shot is the update_RACDB part of db_software_maintenance
Clus_server1
We’ve talked about db_software_maintenance launching a deployment procedure that can be monitored from OEM.
This screen shot is the update_RACDB part of db_software_maintenance
This is what sells me on this option. Each operation is broken down into smaller component steps. Each step in the flow has a status and is timed. The check box before the step will show you the log for that particular step.
So you’ve heard us talk about failures. This pin points the failure .. Look at the log and you will see the cause of the specific failure ... And the action button at the top right of the log pane ... Give you options to retry or skip that step.
So my favorite story about this is a dba was upgrading a cluster going from 11.2 to 12.1.
She had negotiated a Tuesday 8pm maintenance window and at the appointed time kicked off the deployment.
She went to the above window to watch and the process and it died almost immediately.
the Cluster had been reconfigured improperly and corrupted the inventories.
Worked to Oracle to correct.
next day asked customer when she could continue.... Next Tuesday.
A week later found the deployment ... Went to the failed step and retry.
Completed patching but as it was bringing up clusterware hit a memory problem (huge pages) ... Worked with the linux SA .. Retry .. And it completed.
Imagine that with Opatch.
Clus_server2

17. Our Automation Leveraging Db Software Maintenance
So putting it all together
We built a small application express app that lets us create windows and place databases into those windows.
Two weeks prior to the patching window, we subscribe the databases to the appropriate image. This is automated
Then deploy any homes needed for the future activity. Again queries against OEM and our own database tables determine this and it has been automated as well.
When it comes time to patch we review that all databases are up using the screen above then launch the window.
The launch job cycles through the scheduled tasks issuing the emcli command when the predecessor job is complete..
As we’ve said before we monitor this screen and the OEM procedure activity screen to watch the progress of the patching. And troubleshoot any patching task encountering problems.

18. Progress – Crawl, Walk, Run … Fly
2014 Gold Image Provision of Database Homes
2015 RAC Clusterware Upgrade from 11.2 to 12.1 (60)
Some Database upgrades /3 to
Q1 & Q2 Mandate ALL DB homes OEM provisioned
Mandate ALL RAC Clusters built with OEM
Continue upgrades &
Q3 & Q4 Patch 1033 databases using Switch Database
databases patched using Switch Database
databases patched (so far)
All databases patched in 2nd quarter
3rd Quarter moved from Switch to DB_Software_Maintenance
I like this slide ... Because it shows that we did not do this overnight.
Your biggest challenge with doing this is having faith in OEM. It took us years to get where we are ...
But the thing that helped the most ... Is the gold image provisioning of database homes. This was one mundane job that DBA’s did not mind handing over.
-- we had one database build that had 34 one-off patches ... It took hours to build manually minutes with OEM... Un-supervised... Unengaged .. You launch the deployment then check back later.
Then test out the switch database routine, this will familiarize your with out of place patching and the procedure activity screen.
Then look at db_software_maintenance to see if it can help you scale your patching effort as it has us.
I began this presentation showing you the variety of databases both versions and configurations types (multinode rac, RAC 1 Node, stand alone databases and of course dataguard) ... OEM handles this variety better than the DBA’s .. There are a few extra considerations with dataguard .. But again it handles it well.
OEM has been a great enabler to automation for NW.  Based on our success with deployment and patching we are always looking for ways to leverage our OEM investment.
One other thing that I did not highlight .. The lifecycle pack that provides this patching can also be used to provision RAC Clusters, database homes, create databases and many other Oracle activities

19. Karthika Thirumalasamy
Summary:
Recognition:
Missing Nationwide team member John Norman
Oracle OEM Product Team: Harini Srinivasan
Bharat Paliwal
Martin Pena
Saurabh B Jain
Oracle OEM Dev Team: Harmeet Kaur
Paras Narang
Karthika Thirumalasamy
Oracle Senior Enterprise Account Executive Stephan Saade
Oracle Key Account Director for Nationwide Joe Johnston
Other Sessions:
Hands On Lab (HOL6350) Next-Generation Database Patching - Wedneday 11:15
OEM kiosks at the Demo Grounds (MGMT-WU2)
Oracle Fleet Reference Manual:
Questions ???
Summary:
Based on our experience, security requirements are driving the need to patch large number of assets more frequently and automation is key to successfully accomplishing that.
With fleet Maintenance the payoff has been significant ... instead of dedicating the entire DBA staff to constantly patch and coordinate schedules we have been able to patch more in less time with less risk
Before I open the floor to questions... There are some people whos significant assistance I need to acknowledge.
The missing member of our patching team. John Norman. His scheduling of the databases has been one of the more challenging aspects of this journey.
The OEM Product Team
The Oem Lifecycle Development team
The hands on lab ...
The OEM kiosks in the demo grounds
Fleet reference
While questions are being asked .. I want to display a hint sheet ... This has some key information about various things that helped our effort.