1. © 2002 Solution Beacon, LLC. All Rights Reserved. Release 11 i Administration and Maintenance Solution Beacon, LLC John Stouffer http://www.solutionbeacon.com

2. © 2002 Solution Beacon, LLC. All Rights Reserved. 2 Introduction Welcome and Introductions Overview The focus will be to share “Real-World“ experience as well as tips and techniques to administer and maintain the complex Oracle Application environment delivered with Release 11i Audience Profile Current Platform and O/S, Database Version, Applications Release, Application Interface, Language, Installation/Reinstall/Upgrade Plans and Timelines

3. © 2002 Solution Beacon, LLC. All Rights Reserved. 3 What is Release 11i? The Biggest and Most Complex Oracle Applications Release Ever ! There have already been five releases with major changes to tools and technology stack Each release addresses bug fixes AND adds new functionality

4. © 2002 Solution Beacon, LLC. All Rights Reserved. 4 What is Release 11i? Completely New Architecture and Tools All New Technology – True Three-Tier Internet Computing Architecture (ICA) Technology Stack (Forms/Java/Apache) Oracle 8i/9i – CBO, Partitioning Multiple Application Server Components Forms-Based (ERP – Back Office) HTML-Based (Self-Service, Workflow, “i” Products) Business Intelligence (Discoverer, Reports)

5. © 2002 Solution Beacon, LLC. All Rights Reserved. 5 Release 11i Modules Increased Modules, Functionality and Features Release 10.740 products Release 11.0.x50 products Release 11.5.4 161 products Release 11.5.6175 products Release 11.5.7 ??? products Note: While there is an 11.5.6 ERP Family Pack and an 11.5.6 CRP Family Pack, Oracle announced on 1/16/02 that there will not be a 11.5.6 Maintenance Pack or CD Bundle. The next release will be 11.5.7, tentatively planned for April 2002 (subject to change).

6. © 2002 Solution Beacon, LLC. All Rights Reserved. 6 Release 11i Guiding Principles Keep It Simple and Supportable Possible vs. Practical KISS

7. © 2002 Solution Beacon, LLC. All Rights Reserved. 7 Release 11i Concepts All or Nothing – Oracle/Applications Software and Database – No Sharing of Software/Objects Interdependencies of Products – AD, FND, AK, HZ Multiple Component Requirements – Database Software (3), Forms, Reports, Java, JDBC, Apache Hybrid Application – OLTP/Batch Coexist

8. © 2002 Solution Beacon, LLC. All Rights Reserved. 8 Architecture – 11.5.1/11.5.2 Client  Oracle JInitiator (1.1.7.27) Middle Tier  Webservers (WebDB 2.2 and WebDB 2.5)  Jserv (1.1)  JRE (Java Runtime Environment) 1.1.8  JDK (Java Development Kit) 1.1.8  Web Listener 2.2  Oracle Forms/Reports 6i (6.0.8.8.0) Database Tier  Oracle8i Enterprise Edition 8.1.6  Other Oracle Home Software Directories – App(8.0.6), WebDB(8.1.6)

9. © 2002 Solution Beacon, LLC. All Rights Reserved. 9 Architecture – 11.5.3 Client  Oracle JInitiator (1.1.7.27) Middle Tier  Apache Web Server (1.3.9, Full - jsp, jServ)  JServ (1.1)  JRE (Java Runtime Environment) 1.1.8  JDK (Java Development Kit) 1.1.8  Oracle Forms/Reports 6i (6.0.8.8.0) Database Tier  Oracle8i Enterprise Edition 8.1.6  Other Oracle Home Software Directories – App(8.0.6), iAS(8.1.6)

10. © 2002 Solution Beacon, LLC. All Rights Reserved. 10 Architecture – 11.5.4 Client  Oracle JInitiator (1.1.8.7) on Client Middle Tier  Apache Web Server (1.3.9, Full - jsp, jServ)  JServ (1.1)  JRE (Java Runtime Environment) 1.1.8  JDK (Java Development Kit) 1.2.2, 1.3 Recommended  Self Service Framework 5.X  Oracle Forms/Reports 6i (6.0.8.12.1 - Patch 3) Database Tier  Oracle8i Enterprise Edition 8.1.7.1  Other Oracle Home Software Directories - App(8.0.6), iAS(8.1.6)

11. © 2002 Solution Beacon, LLC. All Rights Reserved. 11 Architecture – 11.5.5* Client  Oracle JInitiator (1.1.8.13) on Client Middle Tier  Apache Web Server (1.3.9, Full - jsp, jServ)  JRE (Java Runtime Environment) 1.1.8  JDK (Java Development Kit) 1.3  Self Service Framework 5.2  Oracle Forms/Reports 6i (6.0.8.14.2 - Patch 5) Database Tier  Oracle8i Enterprise Edition 8.1.7.1  Other Oracle Home Software Directories - App(8.0.6.3), iAS(8.1.6) * Even with this Release, significant patching is required.

12. © 2002 Solution Beacon, LLC. All Rights Reserved. 12 11i Architecture - Components Release 11.5.5 Client Tier Web HTTP Servers Application Technology Stack Middle Tier *optional 8.1.6 iAS ORACLE_HOME 8.0.6.3 ORACLE_HOME Database Tier 8.1.7.1 ORACLE_HOME JInitiator ApacheAdmin Server Concurrent Processing Server Discoverer Server * Forms Server Reports Server 8.1.7.1 Database

13. © 2002 Solution Beacon, LLC. All Rights Reserved. 13 Single Server Configuration Single Tier Definition - All Application components installed on a single server Pros Ease of administration (installation and process control) No network traffic Ease of maintenance (single tier patching) Lower hardware and O/S license cost Cons Scalable only within the single server - must provide for present and future growth needs

14. © 2002 Solution Beacon, LLC. All Rights Reserved. 14 Multi-Tier Definition – All Applications components installed on two or more servers  Separate database server  Load balancing  Specific component level performance tuning  Ability to add additional front-end servers as user load increases Multi-Server Configuration - Pros

15. © 2002 Solution Beacon, LLC. All Rights Reserved. 15  Increased network traffic  Increased administration - process control and backups  Increased maintenance - more patches/platforms  More architectural complexity  Higher O/S license costs  More complicated to handle load balancing (OLTP vs. Batch vs. Web) Multi-Server Configuration - Cons

16. © 2002 Solution Beacon, LLC. All Rights Reserved. 16 Server Sizing Configurations Disk Space - 25GB for the software and database - 10GB for the software staging areas - 10-25GB for patches, log/out files, archives, disk backups, temporary files, etc. - RAID for disk drives is necessary Typical ERP Server Configuration - Combined Single Server – 4CPU, 4GB RAM (Scalable) – 50 GB disk effective (after RAID)

17. © 2002 Solution Beacon, LLC. All Rights Reserved. 17 Server Sizing Recommendations Memory Oracle Recommendation – 8MB/user Real World Recommendation - 15MB for every active user - 15MB for concurrent manager workers - 20MB for database background processes - 500MB SGA (pinning packages/sequences)

18. © 2002 Solution Beacon, LLC. All Rights Reserved. 18 Server Sizing Recommendations CPU Oracle Recommendation – 70 users/CPU Real World Recommendation - 15 Active Users/CPU (super users) - 50 Logged Users/CPU - 1 CPU Dedicated – Database and Application Processes (DB, Forms, Web, Concurrent Managers)

19. © 2002 Solution Beacon, LLC. All Rights Reserved. 19 Desktop Sizing – Oracle Minimum Windows 3.11Windows 95+Windows 2K CPU - MHz Pentium 120+Pentium 133+ Pentium 200+ Memory 32 Mb 48 Mb 64-96 Mb

20. © 2002 Solution Beacon, LLC. All Rights Reserved. 20 Desktop Sizing – Real World Windows 95/98/2000 CPU400MHz+ MemoryMinimum 256MB Color PaletteMust be able to run a minimum of 256 colors (a Java requirement), but ideally 65,536 (64K) JInitiator Latest release that is certified for the Oracle Applications Release (1.1.8.13 released with 11.5.5) BrowserEither Netscape Navigator 4.7x where x >=3 or higher or Internet Explorer* release 5 or higher * Why IE? – Preferred by Oracle - MetaLink

21. © 2002 Solution Beacon, LLC. All Rights Reserved. 21 Desktop Considerations Java is particularly CPU intensive and a faster CPU has a more noticeable affect on performance (all other things being equal) JInitiator 1.1.8.7 and certain other Oracle tools are incompatible with Pentium 4 Processors (minimum of 1.1.8.10 required) Automatic download and installation of the JInitiator requires temporary changes to IE security settings Periodic “cleaning” of cache and other temporary files is necessary to reduce browser/memory issues while accessing the Oracle Applications. (See Clean Cache in Free Tools section at http://www.solutionbeacon.com/)

22. © 2002 Solution Beacon, LLC. All Rights Reserved. 22 Server Kernel Parameters Sun Solaris 8 (2.8/5.8) Example 4096 entries in semaphore map (SEMMAP) 4096 semaphore identifiers (SEMMNI) 4096 semaphores in system (SEMMNS) 4096 undo structures in system (SEMMNU) 4096 max semaphores per id (SEMMSL) 4096 max operations per semop call (SEMOPM) 4096 max undo entries per process (SEMUME) 32767 semaphore maximum value (SEMVMX) 16384 adjust on exit max value (SEMAEM) 1073741824max shared memory segment size (SHMMAX) 1min shared memory segment size (SHMMIN) 400 shared memory identifiers (SHMMNI) 200 max attached shm segments per process (SHMSEG) HP/Compaq/SUN Manual Configuration IBM AIX Automatic

23. © 2002 Solution Beacon, LLC. All Rights Reserved. 23 Single UNIX Account Owner Single User – oraappl oraSID or any other single account other than oracle* Pros Ease of access Ease of administration Ease of maintenance (specifically patching) Cons Less auditability Less control - more access to entire system * Why not oracle?

24. © 2002 Solution Beacon, LLC. All Rights Reserved. 24 Multiple UNIX Owner Accounts Multi-User - oraSID (database, database software) and applSID (application/web software) Pros Separation of duties / responsibilities by account Auditability - increased security, restricted access Cons Need root access for installation Increased complexity of Apps process management Application vs. database maintenance must be closely coordinated and controlled Directory and file permissions may need to be adjusted for dual access/management Increased sharing of database/application files

25. © 2002 Solution Beacon, LLC. All Rights Reserved. 25 Account Recommendation Use a single account instead of multiple accounts which typically increase the overall amount of time until an Applications issue is resolved. For most companies, the distinction between Applications management/maintenance and control vs. Database management/maintenance and control is constantly being blurred by business considerations that force support staff to respond quicker to Application issues. At a minimum, if multiple accounts are used, the accounts must use the same group.

26. © 2002 Solution Beacon, LLC. All Rights Reserved. 26 “Simplified” OFA Rapid Install Directories ORACLE_SID@servername > du -sk * (K bytes) 8,456,150SIDappl (APPL_TOP = 10GB) 275,750SIDcomn (COMN_TOP)- for Java/HTML 13,448,775SIDdata (13.5GB = Database Files) 1,289,678SIDdb (1.3GB = 8.1.7, oraInventory and JRE) 1,418,717 SIDora (1.5GB - 8.0.6 (APPL_TOP), iAS (APACHE) Oracle Homes 25GBTotal (Approx)

27. © 2002 Solution Beacon, LLC. All Rights Reserved. 27 Staging Software Directories 867, 577 ora8i 3,455,164 oraAppDB - compressed datafiles 1,877,108 oraApps 866,947 oraiAS 474,716 redCD 7.5GB Total Staging Area required is approximate and various based on platform. Only Application Software - no other Software, Doc or Patch CDs Loaded.

28. © 2002 Solution Beacon, LLC. All Rights Reserved. 28 Supplemental Directories*../SIDadm - Administrative../SIDarch - Archive Logs (Large)../SIDexp - Exports../SIDinst - Installation../SIDlog - Log Files (Large)../SIDout - Output Files (Large)../SIDptch - Patches (Large)../SIDscr - Scripts../SIDtmp - Temporary (Large) As compared to:../SIDcomn - $COMN_TOP * DBA creates

29. © 2002 Solution Beacon, LLC. All Rights Reserved. 29 Applications Process Port Grid Note: Some Apps will reserve multiple ports

30. © 2002 Solution Beacon, LLC. All Rights Reserved. 30 Environment Management Three Specific Oracle Environments for supporting the 11i Applications (Database, Applications, Web) Manually create profiles for each environment (profile.SID.app, profile.SID.db, profile.SID.web) Create a list profiles program to automatically list all available environment profiles upon signon Source the appropriate profile for the task(s) to be performed (including start/stop/backups/patching)

31. © 2002 Solution Beacon, LLC. All Rights Reserved. 31 OAEM Environment Profile Management - Oracle Applications Environment Manager (oaem) Facilitates.profile establishment (DB, App, Web) No need for multiple UNIX.profile scripts Provides start/stop/status – all components Download from Free 11i Tools section of Solution Beacon website

32. © 2002 Solution Beacon, LLC. All Rights Reserved. 32 Installation Overview All Products Installed Regardless of Licensing – database and software Oracle Seed Database – 15 GB Simple Installation – 2/4 Hours Validates Configuration Starts All Processes Core – admin, ad, ax, fnd

33. © 2002 Solution Beacon, LLC. All Rights Reserved. 33 Pre-Installation Checklist Summary Establish Initial Hardware Configuration Verify Certification Matrix Verify Licensed Products Check Upgrade/Install Note on MetaLink daily Check Release Notes – Generic Check Release Notes – Platform Specific Order Software Bundle – 11.5.5 or latest for your platform Download Java and O/S Patches Contact Support for CSI Detailed Report for Licensed Products See R11i Free Tools at http://www.solutionbeacon.com/ for a sample Pre- Installation Checklist

34. © 2002 Solution Beacon, LLC. All Rights Reserved. 34 Rapid Install Overview Check /tmp - must have 750MG of free space minimum, preferably 2GB Stage Software Directory Naming, $TEMP, X-Windows emulator Patch Rapid Install – Check MetaLink Select Products – Check Detail, Licensing Select Country/Language/Character set Specify Directories – tmp Define Software Account(s), Group Define Ports/Java 1.3 Location Review Settings Save Configuration/Install Files Automatic Validation Check (Port, Mount Points, Space, Temp, Host, System Utilities, Java) Automatic Process Startup Finishing Digital Signature/Repackaging Regenerating Jar Files – Force = Y JVM (JInitiator) Client Installation

35. © 2002 Solution Beacon, LLC. All Rights Reserved. 35 Post-Installation Checklist Summary Disk/Tape Backup Cleanup Installation Files Verify Product Licensing – License Manager Apply Initial Patches (AD, FND) Run ADADMIN Tasks – Create Environment File Disk/Tape Backup Change Process Log File Destinations Run Patchsets (patchsets.sh from FTP Site) Apply Technology Stack Patches Apply Product Family Packs Apply Fully Installed Product Mini-packs Run ADADMIN Tasks Disk/Tape Backups Check for Latest Patchsets (patchsets.sh) Update Install/Patch Documentation See R11i Free Tools at http://www.solutionbeacon.com/ for a sample Post- Installation Checklist

36. © 2002 Solution Beacon, LLC. All Rights Reserved. 36 Product Installation Check $APPL_TOP/SID.env Validate $APPLFULL/$APPLSHAR - AD $AD_TOP/sql/adutconf.sql FND_PRODUCT_INSTALLATIONS Relinking (applprod.txt) Patchsets.sh http://machine.domain:Port/OA_HTML/US/ICXINDEX.htm http://machine.domain:Port - Portal pagehttp://machine.domain:Port

37. © 2002 Solution Beacon, LLC. All Rights Reserved. 37 Environment Verification Temporary Files init.ora - utl_file_dir Common Log/Out Files - APPLCSF Process Log Files Forms Parameters - Timeout - To prevent FRM-92100 set FORMS60_TIMEOUT=60 (1 hour) or 1440 (1 day) in APPLSYS.env (sid.env) file located in the $APPL_TOP. Default is 15 minutes.

38. © 2002 Solution Beacon, LLC. All Rights Reserved. 38 Administration Utilities License Manager ($AD_TOP/bin/adlicmgr.sh) AD Configuration Utility ($AD_TOP/sql/adutconf.sql) AD Administration ($AD_TOP/bin/adadmin) AD Controller ($AD_TOP/bin/adctrl) AutoUpgrade ($AD_TOP/bin/adaimgr) AutoPatch ($AD_TOP/bin/adpatch) ddrtest/RDA/TUMS Utilities

39. © 2002 Solution Beacon, LLC. All Rights Reserved. 39 Database Start/Stop Scripts Start #!/bin/ksh. $SCRIPTS_TOP/profile.SID.db cd $ORACLE_HOME/appsutil/scripts./addbctl.sh start./addlnctl.sh start SID Stop. $SCRIPTS_TOP/profile.SID.db cd $ORACLE_HOME/appsutil/scripts./addlnctl.sh stop SID./addbctl.sh stop abort./addbctl.sh start./addbctl.sh stop

40. © 2002 Solution Beacon, LLC. All Rights Reserved. 40 Applications Start Script #!/bin/ksh. $SCRIPTS_TOP/profile.SID.app cd $COMN_TOP/admin/scripts./adalnctl.sh start apps_SID./adcmctl.sh start apps/apps./adfrmctl.sh start./adrepctl.sh start./adtcfctl.sh start*./adapcctl.sh start Or./adstrtal.sh apps/apps

41. © 2002 Solution Beacon, LLC. All Rights Reserved. 41 Application Stop Script #!/bin/ksh. $SCRIPTS_TOP/profile.SID.app cd $COMN_TOP/admin/scripts./adtcfctl.sh stop*./adrepctl.sh stop./adapcctl.sh stop./adfrmctl.sh stop./adcmctl.sh stop apps/apps./adalnctl.sh stop apps_SID Or./adstpall.sh apps/apps

42. © 2002 Solution Beacon, LLC. All Rights Reserved. 42 Environment Management Oracle Enterprise Manager Multiple Instances Applications Manager Management Pack – Applications Oracle Applications Manager Commercial Tools Patrol DBArtisan Spotlight Scripts

43. © 2002 Solution Beacon, LLC. All Rights Reserved. 43 Housekeeping Apply Minimal Patches Latest AD Consolidated - starting with 11.5.5 AD mini-packs and AD consolidated are the same 8.1.7 Database Patch #1831771 Password Utility - patch required Cloning Utility - patch required Workflow 2.6 - 5 patches required ADI Security - patch required Backup AD/FND bin Directories Modify processes, rbs, temp, redos, init.ora Resize the Database Run ADADMIN Steps Cold Backup to Disk/Tape

44. © 2002 Solution Beacon, LLC. All Rights Reserved. 44 Patch Definitions Maintenance Pack (AU – 11.5.5) Entire Release for All Application Modules No Technology Patches Included Lots of Pre/Post Tasks, VERY Long-Running Family Pack (PRC_PF F) All Product Modules Mini-packs For the Specific Family Lots of Pre/Post Tasks, VERY Long-Running Product Mini-Pack (GL D) Entire Release for Specific Application Modules One-Off Bug Patch Fixes Specific Issue, No Regression Testing

45. © 2002 Solution Beacon, LLC. All Rights Reserved. 45 AutoPatch Review AutoPatch ($AD_TOP/bin/adpatch) Bug Fixes Additional Functionality to Existing Products New Products – ADSPLICE AutoPatch Checklist Research on Patch # – MetaLink, OAUG Archives Readme.txt, README.html Shutdown Application Processes Pre-Requisite Patches/Tasks, Post-Patch Tasks C Driver D Driver Checkfile Driver (.cf) – AD E Required G Driver Migration to Other Environments

46. © 2002 Solution Beacon, LLC. All Rights Reserved. 46 AutoPatch Maintenance Standards – Patch Naming, Driver Log Files, Patch Status, Issue Tracking Spreadsheet Process – Discovery, Evaluation and Examination, Pre/Post Patch Tasks, Testing, Migration Approval, Freeze Patch Levels (Install – Int. Testing vs. Upgrade - Latest) Installation – Rapid Install, AD Consolidated, FND Mini-Pack Ongoing Maintenance – Frequency, Business Requirements/Impact

47. © 2002 Solution Beacon, LLC. All Rights Reserved. 47 ftp://oracle-ftp.oracle.com/apps/patchsets/PATCHSET_COMPARE_TOOL/ Updated Nightly By Oracle Can Be Used For 10.7, 11.0.x And 11.5.x Displays Currently Available Patchsets Reads the applptch.txt File or the database Produces Reports On Differences Automate FTP Script Patchsets.sh Utility Patchset Comparison Utility Found on Oracle’s FTP Site:

48. © 2002 Solution Beacon, LLC. All Rights Reserved. 48 Patching – Oracle Tools Manual (Interactive) Automated (Non-interactive) Merge Patch Utility (Automated) OEM (Future) OAM (Oracle Applications Manager)

49. © 2002 Solution Beacon, LLC. All Rights Reserved. 49 Patch Management Manual (applptch.txt) Manual (spreadsheet) Manual (patchsets.sh) Database (AD patch) Commercial (Cinap, Kintana)

50. © 2002 Solution Beacon, LLC. All Rights Reserved. 50 Platinum What is Platinum? Platinum is the latest patched release of CRM and ERP software designed to greatly simplify and expedite the installation of a fresh E-Business environment. The Platinum Environment, apart from being nearly fully patched, also includes all prerequisite patches and critical bug fixes. The Platinum Environment is a tool for rapid installation and implementation – not upgrades. All the applications are patched but the technology stack is NOT patched.all prerequisite patches and critical bug fixes Oracle e-Business Partners Only – HP/SUN/NT/Linux Only - http://platinum.oracle.com

51. © 2002 Solution Beacon, LLC. All Rights Reserved. 51 Basic Definition and Configuration Internal Conflict Resolution Standard Transaction Control Scripts Modifications Abort - Stop Wait - Stop Sleep - Start Concurrent Managers

52. © 2002 Solution Beacon, LLC. All Rights Reserved. 52 Define Workshifts Modify Buffers/Cache, Sleep/Workshifts Define Time-Based Managers Utilize Specialization Rules Concurrent Manager Tuning

53. © 2002 Solution Beacon, LLC. All Rights Reserved. 53 Concurrent Managers - Advanced Troubleshooting (CLASSPATH, /tmp) Managing/Cleaning (MetaLink) - Shutdown Advanced Analysis

54. © 2002 Solution Beacon, LLC. All Rights Reserved. 54 Definition of Cloning: The process of reproducing an entire environment, including the Oracle software ($ORACLE_HOME) and Application software ($APPL_TOP) directory structures. Whitepaper: MetaLink Note 135792.1 Feb 2002) Limitations Architecture Technology Stack Cloning

55. © 2002 Solution Beacon, LLC. All Rights Reserved. 55 Cloning Checklist Clone Patches PERL Path High-Level Tasks Target Installation adclone.pl Pre-Clone Copy Directories (HTML/JAVA) Copy Database adclone.pl Post-Clone Post-Clone Tasks

56. © 2002 Solution Beacon, LLC. All Rights Reserved. 56 Manually Dozens Of Configuration Files Numerous Site Profile Values Not Supported Oracle’s adclone.pl Utility Cloning Methods

57. © 2002 Solution Beacon, LLC. All Rights Reserved. 57 adclone.pl: MetaLink Note 135792.1 Patch 2115451 + Pre-requisite Supported Limitations – Identical Configuration, Database and Technology Patches Not Cloned AD Clone

58. © 2002 Solution Beacon, LLC. All Rights Reserved. 58 Thank you! Hope you enjoyed the R11i Administration and Maintenance Presentation! Remember: KISS Keep It Simple and Supportable