Effective Update Manager Application: Lessons Learned in PeopleSoft 9 Session #CON6144 Oct 2nd, 2014
Speaker Info Daniel F. Monzon Sandeep R. Vemireddi Senior Infrastructure Specialist, Adventist Health System Dan.Monzon@ahss.org Sandeep R. Vemireddi Senior Consultant, Deloitte Consulting LLP sandereddy@deloitte.com Peter A. Christiaans Specialist Leader, Deloitte Consulting LLP PChristiaans@deloitte.com
Adventist Health System Adventist Health System is a faith-based health care organization headquartered in Altamonte Springs, Florida Each facility operates independently in delivering care and services to best meet the needs of the local communities they serve. While each entity is unique, all remain united in one mission of Extending the Healing Ministry of Christ.
1973 Adventist Health System was established 70,000 Employees and physicians 4.5 million+ Patients served annually 22 Home health and hospice agencies 16 Skilled nursing facilities 45 Hospital campuses operated in 10 states
500+ Physician Offices 7.4 billion+ Total Operating Revenue 1.2 billion+ Expenses 8,300 Licensed Beds 817,000+ Invoices 385,600+ Purchase Orders (1.6 Million Lines)
iSynergy Scope PeopleSoft 9.2 (Tools 8.53.13) Interaction HUB 9.1 (Tools 8.53.13) Financial Management Supply Chain Management Human Capital Management CRM Accounts Payable Accounts Receivable Asset Management Billing Cash Management Contracts eSettlements Expenses General Ledger Maintenance Mgt. Project Costing eProcurement eSupplier Connection Purchasing Inventory Mobile Inventory Management Supplier Contract Management Warehouse Management Benefits Administration Payroll Human Resources ePay eProfile eBenefits Taleo On-Boarding eCompensation Workforce Communications Helpdesk for Human Resources Hyperion OBIEE Taleo WorkForce UPK
Agenda PUM & Maintenance in PeopleSoft Installation & Usage of PUM PUM Overview PUM vs Maintenance Bundles Installation & Usage of PUM Installation Creating and Applying Change Packages Migration and Remediation Strategies Analysis and Rollback Strategies, Non PUM Updates & Our Challenges Q&A
PUM & Maintenance in PeopleSoft
PUM Overview PeopleSoft Update Manager (PUM) is the new methodology by which Oracle delivers the maintenance packages PUM is a flexible, streamlined patching process that enables project teams to identify, understand, and selectively apply maintenance updates aligned to requirements and schedule. From PeopleSoft 9.2 all lifecycle management maintenance processes are delivered using only PUM, including the delivery of new features PeopleTools patching is not included in the PeopleSoft Update Manager
PUM vs Maintenance Bundles PeopleSoft 9.2 or higher versions(People Tools 8.53.02 ) PeopleSoft 9.1 or lower versions Speaker points: Our initial Tools installation was on 8.53.03, then the project applied 8.53.08 in advance of applying PUM3, and finally applied 8.53.13 prior to go-live. There were patches to the PUM process that were released 8.53.08 that we wanted to deploy in advance of applying PUMs 2 & 3. The download timeframe for the OVA files can take time, as these are files that come in groups. It should be planned well in advance, and sometimes the file downloads failed.
PUM vs Maintenance Bundles PeopleSoft 9.2 or higher versions(People Tools 8.53.02 ) PeopleSoft 9.1 or lower versions Available on MOS to download as a cumulative package Available on MOS to download by Bundle (Update) number
PUM vs Maintenance Bundles PeopleSoft 9.2 or higher versions (People Tools 8.53.02 ) PeopleSoft 9.1 or lower versions Available on MOS to download as a cumulative package Available on MOS to download by Bundle (Update) number Only current PUM Image (PI) is available at any given time by Application. It contains all updates released till PI release date All bundles released till date for a Application are available to download and are needed to become current Speaker points: Lessons learned on the cumulative image: Needs proper planning when applying standalone patches across image versions, as there can be additional post-requisites that are imported from the PUM image, depending on the version that is used. Latest version of the object will be imported from the latest image Retain old image files as they will no longer be available from Oracle (e.g., once PUM 8 is released, PUM 7 will no longer be available). Retain gold copy backup of original DMO/SYS install at the code baseline that will be used for the other environments, including PRD, prior to the start of any PUM applications.
PUM vs Maintenance Bundles PeopleSoft 9.2 or higher versions(People Tools 8.53.02 ) PeopleSoft 9.1 or lower versions Available on MOS to download as a cumulative package Available on MOS to download by Bundle (Update) number Only current PUM Image (PI) is available at any given time by Application. It contains all updates released till PI release date All bundles released till date for a Application are available to download and are needed to become current Provides ability to selectively apply bug fixes Cannot choose a fix from Bundle – All or Nothing Speaker points: The PUM process will include the pre and post requisites (technical and functional) that have not been applied. The PUM process will look at the maintenance log to see what exists/does not exist in the target instance. The additional content may impact project timelines (e.g., retrofit impact, regression testing impact, business process (re)design impact). We need to periodically schedule an auditing of how accurate the maintenance logs are across the instances. For the selective approach there is a high level of administrative overhead. In the need to track what has been applied and to reconcile that, the review of the maintenance logs is key. Patches will need to move in lockstep and will need to be handled as a ‘mini-upgrade’. For example, a defined window needs to be structured for a group of patches to move from DMODEVTST—QA.
PUM vs Maintenance Bundles PeopleSoft 9.2 or higher versions(People Tools 8.53.02 ) PeopleSoft 9.1 or lower versions Available on MOS to download as a cumulative package Available on MOS to download by Bundle (Update) number Only current PUM Image (PI) is available at any given time by Application. It contains all updates released till PI release date All bundles released till date for a Application are available to download and are needed to become current Provides ability to selectively apply bug fixes Cannot choose a fix from Bundle – All or Nothing One change package for all selected list of fixes Each Bundles need to be a separate change package Speaker points: Comment on methodology: for AHS, we PUM first in the DMO instance, then externally migrate PS project from DMO to DEV. This is done to provide a large developer pool the flexibility of not having conflicts with object locking by having the sysadmin do the migration. Change assistant will apply the DMS, SQL scripts, etc. The prescribed way is to use Change Assistant to migrate the patch across the environments. For standalone bug fixes, it is recommended to apply these as individual projects. In the event that a patch fails testing, it will be easier to back it out as opposed to pulling out the individual objects for that patch from a cumulative package.
PUM vs Maintenance Bundles PeopleSoft 9.2 or higher versions(People Tools 8.53.02 ) PeopleSoft 9.1 or lower versions Available on MOS to download as a cumulative package Available on MOS to download by Bundle (Update) number Only current PUM Image (PI) is available at any given time by Application. It contains all updates released till PI release date All bundles released till date for a Application are available to download and are needed to become current Provides ability to selectively apply bug fixes Cannot choose a fix from Bundle – All or Nothing One change package for all selected list of fixes Each Bundles need to be a separate change package Automatically ensures pre and post requisites Manual tracking is needed to ensure Pre and Post requisites Speaker points: - Pre-and post requisites change with new PUM images/releases and can have an impact on timeline (see prior notes).
PUM vs Maintenance Bundles PeopleSoft 9.2 or higher versions(People Tools 8.53.02 ) PeopleSoft 9.1 or lower versions Available on MOS to download as a cumulative package Available on MOS to download by Bundle (Update) number Only current PUM Image (PI) is available at any given time by Application. It contains all updates released till PI release date All bundles released till date for a Application are available to download and are needed to become current Provides ability to selectively apply bug fixes Cannot choose a fix from Bundle – All or Nothing One change package for all selected list of fixes Each Bundles need to be a separate change package Automatically ensures pre and post requisites Manual tracking is needed to ensure Pre and Post requisites Lists objects impacted for a Bug Fix even before applying List of objects impacted is not readily available Speaker notes: - Advantage to facilitate planning the patch deployment window and estimates for retrofits.
Installation & Usage of PUM
Installation The PeopleSoft Update Image is a virtual machine running on Oracle VM VirtualBox People Tools Talk about install first (OS, DB, PT) The latest PUM Images are offered for download on the Oracle Support site. Download the files to a physical machine running Oracle VM VirtualBox and create the .ova file (PUM image). Run a checksum (fciv) against the concatenated file to make sure you haven’t corrupted the files during download. Oracle 11gR2 Guest Operating System No additional license requirements for a PeopleSoft Virtual Box Images used for non-production activities such as patches and updates or demonstrations.
Steps to set up the PUM Virtual Machine Install Virtual Box and import PeopleSoft Image Install the current version of the Oracle VM Virtual Box Download the PeopleSoft Update Image files Import the PeopleSoft Update Image VM in Oracle VirtualBox Manager One can start, stop and manage the virtual machine from VM VirtualBox Manager Lessons Learned: 1. Before starting the VM make sure you select bridged adapter in the settings so you will be able to communicate with it from outside of the host system. 2. Also make sure to install extension pack for your version of the Oracle VM VirtualBox software. 3. 4. Don’t try to use WiFi if you are running this from a laptop or similar device. 5. If you need the Virtual Machines to start on boot in the event of maintenance set up VMVBoxService to run headless VM’s. You will still be able to connect and manage the VM’s with remote tools. http://vboxvmservice.sourceforge.net/
Steps to set up the PUM Virtual Machine Start VM and configure network settings 5. Start the PeopleSoft Update Image VM in Oracle VirtualBox Manager 6. During first boot you will need to provide network, database and other server selections i.e. IP address, subnet mask, gateway, DNS Server, and hostname 7. Once started make sure you can connect to the VM with the various protocols i.e. SMB, App Designer, PIA (web). Lessons Learned: 1. Plan ahead on how best to add the images to your network infrastructure – (DHCP vs. static) In our experience using static IP was only way to successfully initialize the stack. 2. Again we recommend that you use a static IP when setting this up so uou may have to contact your network administrator since you will need IP, Subnet Mask, Gateway, and DNS Server information of your local network in order to communicate with the VM outside of the host machine. 2. If you will be sharing access to the PUM image with others you will need to provide them with the /etc/hosts file entry and/or tnsnames.ora file entry if they will connect w/ app designer or third party tools.
Steps to set up the PUM Virtual Machine Configure Source and Target DB connections 7. Install the database client on host machine 8. Install People Tools (Application Designer and Change Assistant) on the host machine 9. Configure Change Assistant and the Environment Management Hub Lessons Learned: We recommend you always test your connection from the host machine using App Designer and your favorite SQL tool. This way you are sure that tnsnames.ora is setup properly. Careful not to fat finger your passwords Always use the Test Connection button to help ensure connectivity Verify that your folder structures are set up correctly and map drive to the pi_home samba share is connected.
Steps to set up the PUM Virtual Machine Clear agent cache and launch PSEMAgents PeopleSoft Update Manager Agents PSEMAgent Web (Linux) $PS_HOME/PSEMAgent/StartAgent.sh Sending pulse from 'com.peoplesoft.emf.peer:id=1' PSEMAgent Application (Linux) $PS_HOME/PSEMAgent/StartAgent.sh Sending pulse from 'com.peoplesoft.emf.peer:id=2' PSEMAgent Batch (Windows File Server) %PS_HOME%\PSEMAgent\StartAgent.bat Sending pulse from 'com.peoplesoft.emf.peer:id=3' Workstation Change Assistant (Update Manager) Lessons Learned: 1. We chose to share PS_HOME between development environments so it is necessary to clear agent cache and reconfigure agents before initialization 2. Best practice is to bring up the agent on you’re a. Web server (gateway) b. App Server c. Batch Server (File Server) d. Lastly launch change assistant and follow the update manager workflow. 3. Make sure your peer id’s are in proper sequence if not stop agents, clear cache, and relaunch in proper sequence 23
Steps to set up the PUM Virtual Machine Clearing PSEMAgent Cache Lessons Learned: Don’t‘ forget to clear cache on the batch file server and workstation as well. (Windows Machines)
Creating Change Package Launch PeopleSoft Change Assistant and select Update Manager Lessons Learned: PeopleSoft Change Assistance would be used to create and apply PUM Packages Launch the Change Assistant Wizard, select Update Manager
Creating Change Package Define a New Change Package Speaker points: Select Define a New Change Package to create a new change Package and click on finish Select target PeopleSoft environment
Creating Change Package Open Browser and Connect to the PI Database Speaker points: Verify Target and source instance names and click on ‘Open Browser and Connect to the Database’ button
Creating Change Package Define Change Package Speaker points: Navigate to ‘PeopleTools > LifeCycle Tools > Update Manager > Define Change Package’ Select target database on this screen, this allows PUM to identify and filter BUGs that are already applied. Provide a name for the change package. Lessons Learned: Before you create your first package come up with a naming convention that will make it easy to identify and track packages through the life cycle. We simply added and AH (Adventist Health) + bug ID since we install our packages individually… When we first started however we were creating huge packages with many bug id’s bundled together but that became problematic as we began migrating through our development environments since many patches did not pass. There was no way for us to pull out the patches that didn’t pass so it was difficult to run compare reports since we could not carry the original PUM project forward. Define a traceable naming convention for the change packages and PeopleSoft Projects
Creating Change Package Choose Package type Speaker points: All Updates Not Yet Installed – would create a package with all updates that are not already applied to target instance Specific Update – would need the BUG number as a input Update By Criteria – one can filter the bug fixes based on criticality of Bugs Updates By objects Name – would list all Bug fixes that include a People Tools Object
Creating Change Package Review Bug Details Speaker points: This step would provide list of Bug numbers selected It would also include the pre-requisite and post-requisite Bugs that are not already applied Once complete this step takes you back to the change assistant
Applying Change Package Select Change Package from Drop Down, Click Apply Change Package and Finish Speaker points: With the focus now back on change assistant select the change package you created from the drop down and click Finish At this point your PUM package zip file is created and downloaded to the local instance of your change assistant in the predefined download directory to be executed.
Applying Change Package Completed PUM Package Speaker points: Keep an eye on the steps and make sure each one completes successfully. Remediate any failures…good idea to check for known PUM issues on MOS.
Migration and Remediation
Migration Strategies Option 1 - Creating a Change Package Apply PUM to the development environment using Change Assistant Remediate code line and scripts Create a custom change package and apply it to higher environments for testing using Change Assistant
Migration Strategies Option 1 - Creating a Change Package Option 2 - Using a 3rd Party Migration Tool Apply PUM to the demo environment Create a project using the migration tool Use the migration tool to migrate the project to Dev Remediate and create a new migration set in Dev for testing in higher environments Rollback option may be available
Migration Strategies Option 1 - Change Package Option 2 - Migration Tool Option 3 - PUM followed by remediation Apply PUM to the development environment using Change Assistant Remediate code line and scripts Apply PUM to higher environments using Change Assitant followed by migration of remediated objects
Migration Strategies Option 1 - Change Package Option 2 - Migration Tool Option 3 - PUM followed by remediation Based on the tools available, phase of the Project and team composition we opted with PUM followed by remediation strategy.
Analysis and Rollback Create a list of Bug fixes that are part of technical or functional pre and post requisites Perform High level Analysis of how a change package impacts PeopleSoft based of the objects impacted Plan and Define retrofit windows appropriately as per project phase Take backup of Project, Script target data and source file like SQRs and Cobols Perform both technical and functional testing based of objects and code line impacted and business processes impacted If Project team opts to not proceed or proceed with a subset of fixes rollback changes to the instances using backups and repeat the process
Strategies, Non PUM Updates & Our Challenges
Strategies Proactive Application Periodically download current PUM Image & apply using ‘All Updates Not Yet Installed’ Introduces the most change, requires the most remediation and testing Once every 9 to 12 months or when count of number of fixes in not applied state for the modules implemented is too high and creates risk for Reactive fix application.
Strategies Proactive Application Reactive Application When an issue is identified and its fix is pointed to be available in PUM Image Use search capabilities to find the fix, create a change Package and apply fix
Strategies Proactive Application Reactive Application Selective Application Review critical fixes and new features Review the updates posted relevant to the Implemented modules for each PUM release Apply once for every 3 to 6 months
Strategies Proactive Application Reactive Application Selective Application All of these strategies are equally important and have to be included in the Project Plan
Non PUM Updates PeopleSoft Release Patchset (PRP) Oracle delivers new PeopleSoft Update Manager (PUM) Images approximately every 10 weeks, however there are updates outside a PI also. PeopleSoft Release Patchset (PRP) Patches that are delivered between PI releases. Should be applied to current PeopleSoft Update Image Once applied select the fixes from PI and apply to target database.
Non PUM Updates PeopleSoft Release Patchset (PRP) Oracle delivers new PeopleSoft Update Manager (PUM) Images approximately every 10 weeks, however there are updates outside a PI also. PeopleSoft Release Patchset (PRP) Proof of Concept (POC) Patches Patches created for escalated critical reasons for a single customer Apply directly to the DEV environment as a customization Apply the patch from the subsequent PUM image as soon as it is available
Non PUM Updates PeopleSoft Release Patchset (PRP) Oracle delivers new PeopleSoft Update Manager (PUM) Images approximately every 10 weeks, however there are updates outside a PI also. PeopleSoft Release Patchset (PRP) Proof of Concept (POC) Patches It is important to keep maintenance log consistent with the fixes applied to environments
Planning Adventist Health will have successive rollouts of PeopleSoft to multiple hospitals/regions over the course of multiple years. A key challenge is not only when to plan PUM deployment windows within the existing rollout but also how these may impact concurrent deployments.
Planning
Lockstep migrations from DMODEVTSTQA Planning Lockstep migrations from DMODEVTSTQA DMO DEV TST QA Speaker notes: Patches will need to move in lockstep and will need to be handled as a ‘mini-upgrade’. For example, a defined window needs to be structured for a group of patches to move from DMODEVTST—QA.
Lessons Learned and Takeaways Maintain a “Gold” Demo instance that is at the same patch level as your Production environment. Proper planning is required when applying patches across PUM images. The latest version of the objects will be imported from the latest image. This may impact retrofits, regression testing and business processes. Use individual packages in the event that patches do not work. Use the same naming convention for packages. Maintain an archive of PUM images that have been applied. Speaker Notes: When we first started however we were creating huge packages with many bug id’s bundled together but that became problematic as we began migrating through our development environments since many patches did not pass. There was no way for us to pull out the patches that didn’t pass so it was difficult to run compare reports since we could not carry the original PUM project forward. Before you create your first package come up with a naming convention that will make it easy to identify and track packages through the life cycle. We simply added and AH (Adventist Health) + bug ID since we install our packages individually
Lessons Learned and Takeaways Validate that the list we expected to apply is in the database’s maintenance log at the end of each move. Periodically schedule an audit of the maintenance logs to validate any selective patches that have been applied. Plan ahead on how best to add the images to your infrastructure – (DHCP vs. static). Make sure you select bridged adapter in the settings of your VM so you will be able to communicate with it from outside of the host system. Use the same migration method (PUM versus project migration) Patches will need to move in lockstep and will need to be handled as a ‘mini-upgrade’. For example, a defined window needs to be structured for a group of patches to move from DMODEVTST—QA.
Q&A