Oracle RDBMS Patching Brian Hitchcock OCP 8, 8i, 9i DBA Sun Microsystems NoCOUG Brian Hitchcock May 6, 2004Page 1

NoCOUG Brian Hitchcock May 6, 2004Page 2 Why Patch the RDBMS?  To upgrade – For example to  One-off patch – Fix a specific bug  Security patches – Fix specific security issues for specific products – This is the focus here… – But notice that I end up patching to as well…

NoCOUG Brian Hitchcock May 6, 2004Page 3 Patching In General  Is becoming a bigger issue – More patches more often – More patches for more products – Think this is bad? – Oracle apps patching makes this look easy – Apps 11i patching is more complex  Many more modules, interactions

NoCOUG Brian Hitchcock May 6, 2004Page 4 Patching In General  And, more fun… – No way to back out of a patch  In general  Specific patches may say you can deinstall…  But what if that patch required ? – Once applied, only one way to go back…  Full restore of ORACLE_HOME from backup – No way to tell what patch level a database is at  Other than version such as  You must manually keep track of patches applied

NoCOUG Brian Hitchcock May 6, 2004Page 5 Patching In General  How often do you patch? – Every time a new security patch is available? – Quarterly?  Security risk until latest patch(es) applied? – Testing for each patch?  For bug fix patch, testing is clear  For other types of patches ­None? ­Complete? ­In between?

NoCOUG Brian Hitchcock May 6, 2004Page 6 Patch Testing Details  What is your policy? – Apply all needed patches, test? – Apply one patch and test? – If testing shows problems, what to do? – Need to test  Your app software  Vendor app software  OS issues  Security, chroot, other software components

NoCOUG Brian Hitchcock May 6, 2004Page 7 How Do You Know…?  What patch(es) do you need to apply? – Security alerts from Oracle  Must review each one manually – Metalink – Your environment has hit a specific bug – Need specific functionality  Feature isn’t available until

NoCOUG Brian Hitchcock May 6, 2004Page 8 How Do You Know…?  For security patches – Oracle sends out security alerts  Each alert applies to specific products  Your site doesn’t need all of them  No source for a single list of which patches you need – I like to file a TAR to confirm the patches I need  Some patches require other patches  Fun, fun, fun!

NoCOUG Brian Hitchcock May 6, 2004Page 9 Example, for  Get current with all security alerts – Political – Nothing was done for a long time – A manager read about a recent oracle alert – Suddenly we have to apply lots of patches

NoCOUG Brian Hitchcock May 6, 2004Page 10 Why Discuss ?  is not cool!  Cool DBAs only talk about 10g!  But real world has X databases  The older a db version becomes the more patches you will need to stay current  Same issues are happening for 9i – Will happen for 10g  Process is the same, starting version doesn’t matter

NoCOUG Brian Hitchcock May 6, 2004Page 11 Finding Security Alerts  Metalink  FAQ for security alerts – Doc id – Item I, generic questions  Number 10, what security patches do I need for my database?  Points to number 13, security patch matrix ­ doesn’t need patches below #48 ­ doesn’t need patches below #59 – When I did this I needed 48, 49, 50, 51, 54  Security alert #62 hadn’t been issued at that time – Today I would need #62 as well…

NoCOUG Brian Hitchcock May 6, 2004Page 12 Finding Security Alerts  FAQ for security alerts (cont’d) – Item II, list of security alerts and notes  Lists security alerts #18 through #66  Review each security alert for patch # – Security alert #66 is most recent as of today  Check Metalink frequently – changed may 07, 2004 while I was creating the previous slide – Note that more products means more patches  Database plus app server etc.

NoCOUG Brian Hitchcock May 6, 2004Page 13 Security Alerts  Listing of security alerts from doc id II. List of Security Alerts and Notes (since Nov 2001) II.1. Security Alerts: Doc Security Alert #66: Vulnerabilities in Oracle Application Server Web Cache Doc Security Alert #65: Security Vulnerability in Oracle9i Application and Database Servers Doc Security Alert #64: Buffer Overflow in Oracle9i Database Server Doc Security Alert #63: Security Vulnerabilities in Oracle9i Lite Doc Security Alert #62: SSL Update for CERT CA and older SSL issues Doc Security Alert #61: SQL Injection Vulnerability in Oracle9i Application Server Doc Security Alert #60: Unauthorized Access to Restricted Content in Oracle Files Doc Security Alert #59: Buffer Overflow in Oracle Binaries Doc Security Alert #58: Buffer Overflow in the XML Database of Oracle9i Database Server Doc Security Alert #57: Buffer Overflows in EXTPROC of Oracle Database Server Doc Security Alert #56: Buffer Overflow Vulnerability in Oracle E-Business Suite Doc Security Alert #55: Unauthorized Disclosure of Information in Oracle E-Business Suite Doc Security Alert #54: Buffer Overflow in Oracle Net Services for Oracle Database Server Doc Security Alert #53: Report Review Agent (RRA/FNDFS) Vulnerability in Oracle E-Business Suite Doc Security Alert #52: Two Vulnerabilities in Oracle9i Application Server Doc Security Alert #51: Buffer Overflow in the Oracle Executable of Oracle Database Server Doc Security Alert #50: Buffer Overflow in Oracle Database

NoCOUG Brian Hitchcock May 6, 2004Page 14 Security Alerts Doc Security Alert #49: Buffer Overflow in Oracle Database Doc Security Alert #48: Buffer Overflow in Oracle Database Doc Security Alert #47: Vulnerabilities in Oracle 9i Application Server Doc Security Alert #46: Buffer Overflow in iSQL*Plus (Oracle9i Database Server) Doc Security Alert #45: Security Release of Apache Doc Security Alert #44: Unauthorized Access Vulnerability in the Oracle E-Business Doc Security Alert #43: Oracle9i Application Server - Web Cache Administration Tool Crash on Malformed Request Doc Security Alert #42: Security Vulnerability in Oracle Net Doc Security Alert #41: Oracle9i Application Server Oracle Java Server Page Demos Vulnerability Doc Security Alert #40: Oracle Net Listener Vulnerabilities Doc Security Alert #39: Oracle9i Application Server - Web Cache Administrator Password Not Encrypted Doc Security Alert #38: Security vulnerability in Oracle Net Doc Security Alert #37: OpenSSL Security Vulnerability Doc Security Alert #36: Security Vulnerability in Apache HTTP Server of Oracle9iAS Doc Security Alert #35: Buffer Overflow Vulnerability in Oracle9iAS Reports Doc Security Alert #34: Security Vulnerability in Oracle Net (Oracle9i Database Server) Doc Security Alert #33: User Privileges Vulnerability in Oracle9i Database Server Doc Security Alert #32: Unauthorized Access Vulnerability in the Oracle E-Business Suite Doc Security Alert #31: Oracle Configurator Security Issue: Potential Cross-site Scripting Attacks Doc Security Alert #30: SNMP Vulnerability in Oracle Enterprise Manager, Master_Peer Agent Doc Security Alert #29: ALERT: Oracle PL/SQL extproc in Oracle 9i, Oracle 8i and Oracle8 Database

NoCOUG Brian Hitchcock May 6, 2004Page 15 Security Alerts Doc Security Alert #28: Vulnerabilities in Oracle mod_plsql and JSP in Oracle 9iAS V1.0.2.x Doc Security Alert #27: Vulnerabilities in Oracle 9i Application Server Web Cache Doc Security Alert #26: Potential DoS Vulnerability in Oracle9i Application Server Doc Security Alert #25: Vulnerabilities in MODPLSQL No Doc Security Alert #24: Skipped Multiple Doc (Security Alert #23 is split into 3 documents on MetaLink) Doc Security Alert #23: Oracle Home Environment Variable Buffer Overflow Doc Security Alert #23: CHOWN Path Environment Variable Vulnerability Doc Security Alert #23: Oracle Home Environment Variable Validation Vulnerability Doc Security Alert #22: Security Implications of the Oracle9iAS v Default SOAP Configuration Doc Security Alert #21: Oracle Label Security Mandatory Security Patch Doc Security Alert #20: Oracle File Overwrite Security Vulnerability Doc Security Alert #19: Oracle Trace Collection Security Vulnerability Doc Security Alert #18: Oracle9iAS Web Cache Overflow Vulnerability

NoCOUG Brian Hitchcock May 6, 2004Page 16 Patches Needed  For security alerts – 48, 49, 50, 51, 54 – Review each alert to find needed patch info  Need patches – ( ) – (alert 48) required – (alert 49) required – (alert 50) required – (alert 51) required – (alert 54) required

NoCOUG Brian Hitchcock May 6, 2004Page 17 Patches Needed  Create stage directory for each patch  Ftp from oracle  Patches require patches – To apply some of these security patches  You must be at  Patch to before applying these patches  Note that I had no plan to patch to – One patch leads to other patches…

NoCOUG Brian Hitchcock May 6, 2004Page 18 Getting Patches  Metalink – Patches – Simple Search  Enter specific patch number  Specify platform – Download  Patch zip file  Readme file

NoCOUG Brian Hitchcock May 6, 2004Page 19 Getting Patches  What is patch number for patch? – Should be simple to find… – Metalink  Patches  Simple search ­Product: Oracle Database Family ­Release: ­Patch type: Patchset/Minipack ­Platform: Solaris Sparc 32-bit ­24 results – Correct patch? – Patch set for oracle data server

NoCOUG Brian Hitchcock May 6, 2004Page 20 Patching Process  What does it take to apply a patch? – Dot release   Oracle installer (OUI) – One-off, security patches  README shows steps to install patch  Example, security patch ­Shutdown database, listener ­Execute patch.sh supplied as part of patch

NoCOUG Brian Hitchcock May 6, 2004Page 21 Patching Process  Production – Must backup ORACLE_HOME – Full backup of database – Document the db  This will come up later  I use dbdoc script, see Managing Multiple Databases… on NoCOUG website – If patch fails  Restore ORACLE_HOME from backup

NoCOUG Brian Hitchcock May 6, 2004Page 22 Patching Process  Development – Full export – Document the db – If patch fails  Reinstall Oracle software  Import export – However,  If practicing prod patching on dev db  Should practice the prod db process

NoCOUG Brian Hitchcock May 6, 2004Page 23 Fresh Install?  Before creating any databases – Install Oracle software – Apply all needed patches – Much quicker – Many post patch steps only apply if database already exists

NoCOUG Brian Hitchcock May 6, 2004Page 24 Patch Install Steps  Can be simple  Can be complex – Example, patch – May require use of Oracle Installer  May require use of OUI that is part of the patch – Patch may require certain patch level  Example, patch can only be applied to  You must review the README file for each patch – Script the steps for each patch

NoCOUG Brian Hitchcock May 6, 2004Page 25 Cases  1) OraInventory not in place  2) Installer not in place  3) 64-bit oracle  4) chroot  5) not following instructions

NoCOUG Brian Hitchcock May 6, 2004Page 26 Case1 -- OraInventory  Existing database  Patch to latest security alert – At the time, this was security alert 54 – Downloaded all needed patches  – (alert 48) – (alert 49) – (alert 50) – (alert 51) – (alert 54)

NoCOUG Brian Hitchcock May 6, 2004Page 27 Case 1 -- OraInventory  Review readme – Existing database – Many post patch tasks – Before applying  Backup db  Shutdown db  Shutdown listener

NoCOUG Brian Hitchcock May 6, 2004Page 28 Case 1 -- OraInventory – Script the steps  Patch readme file README_8174.html  How to install this patch set  Steps 6 through 18 ­Oracle Label Security ­Disabling system triggers ­Check JIS ­Catalog.sql, catproc.sql ­Set trace ­Java objects ­Enable system triggers ­Recompile invalid objects

NoCOUG Brian Hitchcock May 6, 2004Page 29 Case 1 -- OraInventory  Start installer – Installer not installed – Find original cpio files from install – Run installer (OUI) from there – Script inputs for installer  File locations ­Source ­Destination ­UNIX group name

NoCOUG Brian Hitchcock May 6, 2004Page 30 Case 1 -- OraInventory  And now? – Dependencies – There are no patches that need to be applied from the patch set Oracle 8i  Huh?  Off to Metalink – Doc ID – OraInventory is missing

NoCOUG Brian Hitchcock May 6, 2004Page 31 Case 1 -- OraInventory  What is OraInventory? – Documents exactly what was installed – Created as part of software installation – Created by the installer  What does it do? – When installing a patch – Installer checks OraInventory – Verifies that patch should be applied  Example, patch on Oracle_home

NoCOUG Brian Hitchcock May 6, 2004Page 32 Case 1 -- OraInventory  Where does it live? – Installer creates in Oracle_base  (my experience)  What happened here? – oraInventory didn’t exist – Installer couldn’t tell what had been installed – Installer decided it couldn’t install anything  No inventory, can’t apply any patches

NoCOUG Brian Hitchcock May 6, 2004Page 33 Case 1 -- OraInventory  Ok, but what caused this? – To save time, copy existing oracle installation  Tar up oracle_home  Move to new machine  Untar – Lovingly referred to as “Tar&Toss”  my manager came up with that – This isn’t supported by Oracle – This saves time initially  Wastes time later

NoCOUG Brian Hitchcock May 6, 2004Page 34 Case 1 -- OraInventory  OK, that’s weird, but what now?  How to re-create the inventory? – There is only one way – Reinstall the Oracle software – In this case, a full reinstall of  Reinstall will over-write oracle_home – Anything you can’t lose?  Tnsnames.ora, password file – Don’t place anything of your own in oracle_home – Document your database before patching

NoCOUG Brian Hitchcock May 6, 2004Page 35 Case 1 -- OraInventory  How to be sure – Nothing unique in oracle_home? – Can’t be sure – Make backup  I had enough disk space – Copy oracle_home to another filesystem  Now need to reinstall – Disk space to stage the software?

NoCOUG Brian Hitchcock May 6, 2004Page 36 Case 1 -- OraInventory  After software reinstalled – Install patch  Works this time! – Apply the 5 patches in order – Startup the database – Test application – Everyone is happy!  But this took much longer than we planned

NoCOUG Brian Hitchcock May 6, 2004Page 37 Case 2 -- Installer Not In Place  Applying same patches to another machine – Installer not installed – Base software ( ) not on disk – Not enough disk space for software CD image – Have to free up disk space just to  Copy the CD image to get the installer on disk – Proceed with the patching process  Saves disk space in the short term – Wastes time later

NoCOUG Brian Hitchcock May 6, 2004Page 38 Case bit Oracle  Different scenario – No security patches – Simple patch from to  No problem – Stage the patch to the db machine – Downtime for patching is almost here – Reviewing dbdoc output  Select * from v$version shows  Oracle 8i … - 64bit Production

NoCOUG Brian Hitchcock May 6, 2004Page 39 Case bit Oracle  64-bit Oracle? – This is a development db – Production is 32-bit – I assumed dev would be 32-bit – I staged the 32-bit patch  20 minutes to – Download 64-bit patch from Oracle web site – Check README for 64-bit, same as 32-bit – Calm down  No one can explain why…

NoCOUG Brian Hitchcock May 6, 2004Page 40 Case 4 -- chroot  Yet another environment – All set to apply patches – Shutdown database, listener – Start installer  Can’t display OUI GUI back to my workstation  Chroot – Removes many OS libraries – Have to manually identify which are needed – Copy from another system

NoCOUG Brian Hitchcock May 6, 2004Page 41 Case 5 – Complete the Patch  User calls – Dev db doesn’t work – Error is ‘blah blah blah’  Metalink – Error seen when patch partially applied  Call user – “Did you apply a patch?” – “Yes” – “Did you complete all the post patch steps?” – “Oh, umh, ok, thanks!” – Didn’t hear from the user again

NoCOUG Brian Hitchcock May 6, 2004Page 42 Lessons Learned  Verify – OraInventory exists  If not, enough disk space to backup oracle_home? – Installer is installed  If not, disk space for source CDs? – Correct patch(es)  32-bit versus 64-bit – Installer GUI can display to your workstation – Finish all patch install steps  Document this

NoCOUG Brian Hitchcock May 6, 2004Page 43 Lessons Learned  For a new install – Oracle_home not a top level directory – Oracle_base /u01/app/oracle – Oracle_home $ORACLE_BASE/product/ – Oracle_home /u01/app/oracle/product/ – Install the installer  A 10 minute patch can become a 5 hour mess  Verify things before the scheduled patch time  Document all the steps – Takes time the first time – Saves time on all the other servers – Saves time when you have to redo things