HP-UX Patch Program 1 HP-UX Customer Patch Panel HP-UX Successful Patching Strategies

HP-UX Patch Program 2 HP World – HP-UX Customer Patch Panel presented by : HP (Laurie Schoenbaum) Nestlé (Terri Mando) Brigham Young University (John Payne) Philips Research Labs (Donie Collins) Beckman Coulter ( Chris Maehara)

HP-UX Patch Program 3 Nestlé Presented by: Terri Mando

HP-UX Patch Program 4 background Nestlé is Switzerland's largest industrial company and the world's largest food company. Nestlé USA headquartered in Glendale, CA technology used to stay competitive in the market place 135 HP servers HP-UX 10.20, 11.0 and D, K, L, N, and V-class servers Located in Arizona, California, and Ohio Nestlé background

HP-UX Patch Program 5 background (Cont.) all remote system administration servers assigned to application groups Per application: Test, Development, QA, and Production servers SA assigned to application group 24x7x356 support primary and backup 5-16 servers per SA CSS support on SAP, mostly PSS support customized ASE, no onsite support Nestlé background

HP-UX Patch Program 6 patching strategy twice a year proactive patching stringent formal change management process phased rollout “12 step program” patch depot management strategy one patch depot per OS use of make_bundles use of “cleanup” Nestlé patch strategy

HP-UX Patch Program 7 patching strategy patch selection Mission Critical (CSS) support contract delivers proactive patch bundles quarterly “conservative” change strategy (MCSCM) custom patch bundles are “delta” bundles added to existing patch depots quarterly only select patches applicable to environment patch dependencies handled by HP support (RASE) Nestlé patch strategy

HP-UX Patch Program 8 patching strategy patch warnings HP support personnel track and provide recommendations on patch warnings recommendations are individually assessed for applicability to environment rarely has a patch been removed due to a patch warning handled in next proactive patch cycle Nestlé patch strategy

HP-UX Patch Program 9 change management documented change management process change requests required formal approval process business critical systems have a 4 hour maintenance window Nestlé change management

HP-UX Patch Program 10 patching strategy patch application/12 step program pre patching change management scheduling conflicts? health check commit patches and cleanup SD log files swinstall –p (review logs, resolve issues) patch application stop applications swinstall (Do it!) Nestlé patch application

HP-UX Patch Program 11 patching strategy patch application/12 step program post patching review logs health check verify applications change management Nestlé patch application

HP-UX Patch Program 12 conclusions- recommendations Nestlé conclusions written procedures provides consistency pre-patching, patching, post-patching plan (12 step program) provides a framework allows tasks to be automated minimize time spent patching proactive patching!! definition of success is not having a problem

HP-UX Patch Program 13 conclusions- recommendations Nestlé conclusions available on the Interex Patch SIG website ( index.html) index.html “Patching: A 12-Step Program” patch_preview.sh patch_do-it.sh “Patch Depot Management” document

HP-UX Patch Program 14 Brigham Young University Presented by: John Payne

HP-UX Patch Program 15 background Brigham Young University has grown from a small pioneer academy to one of the world’s largest private universities, with more than 29,000 students from 100 countries IT supports payroll, student information, courses online and other content related to the university 50 HP-UX systems HP-UX 10.20, 11.0 and A500/rp2470s, rp8410, K-class, R-class, L-class, N-class Brigham Young University background

HP-UX Patch Program 16 background (Cont.) 1primary system administrator 24x7 with 4 hour response no on-site HP support Brigham Young University background

HP-UX Patch Program 17 patching strategy philosophy quarterly proactive patching HP-UX exception based on release of SupportPlus media goal: no unscheduled downtime switch from reactive to proactive maintenance to improve supportability 3 month test cycle in lab before rolling to production Brigham Young University patch strategy

HP-UX Patch Program 18 patching strategy patch warnings QPK bundles reduce probability of a patch warning security patches may be applied reactively Brigham Young University patch strategy

HP-UX Patch Program 19 patching strategy patch application clusters of redundant applications maximizes system availability non-redundant applications require off hour planned outages problems generally logged with the ITRC call manager Brigham Young University patch strategy

HP-UX Patch Program 20 change management formal change request process all system changes are logged Brigham Young University change management

HP-UX Patch Program 21 conclusions- recommendations quarterly proactive patching as virtually eliminated unscheduled downtime and reactive patching quality of patches in QPK helps to stabilize systems – reduces risk adequately test before rolling to production eliminating the need for system administers from working nights would be a plus! Brigham Young University conclusions

HP-UX Patch Program 22 Philips Research Labs Presented by: Donie Collins

HP-UX Patch Program 23 background division of Philips Electronics technical computing support for 3000 users 1600 are researchers of various sciences 1400 are from product division R&D departments work in partnership with other IT departments within Philips Philips Research Labs background

HP-UX Patch Program 24 Philips Research ICT Infrastructure: Server Based Computing (NXA) file servers H.A. GigaBit Ethernet Ethernet 100BaseT/10BaseT Network switches Unix batch- and compute- servers for compute and memory intensive CAD applications Unix login-server (gateway to Unix for PC desktops) Windows NT/2000 PC with X-server Laptop W2000 with X-server X-terminal (decreasing) Windows Terminals Servers for PC based applications Unix Admin/license servers Unix Backup servers load balancing &redundancy NFS/CIFS

HP-UX Patch Program 25 background (Cont.) 150 HP9000 servers and workstations standard system models and configurations 10 system administrators Personalized System Support (PSS) HP on-site hardware engineer 99.97% uptime goal Philips Research Labs background

HP-UX Patch Program 26 patching strategy philosophy if its not broken, don’t fix it; reactive patch philosophy execute security_patch_check weekly proactive with security patches keep all systems at same patch level per OS one patch depot per OS strive for only 3 patches; highest rated patches use QPK bundles to reduce individual point patches Philips Research Labs patch strategy

HP-UX Patch Program 27 patching strategy (cont) moving in direction of proactive maintenance with a “Enterprise Technical Server Environment (ETSE)” includes QPK, HWE bundles and TCOE 6 month delivery cycle reduces management of point patches Philips Research Labs patch strategy

HP-UX Patch Program 28 patching strategy patch selection Philips Research Labs patch strategy use IT Resource Center subscribe to patch digest use patch database to download patches ITRC tools identify dependencies QPK and HWE bundles

HP-UX Patch Program 29 patching strategy patch warnings ITRC tools send proactive notification of patches with warnings warnings are examined for applicability and action is taken do nothing turn off functionality install superseding patch remove patch Philips Research Labs patch strategy

HP-UX Patch Program 30 patching strategy patch application SD-UX tools manage depots and installation 3 step rollout install on test system roll to a few production systems complete rollout system redundancy reduces planned and unplanned downtime Philips Research Labs patch strategy

HP-UX Patch Program 31 change management Philips Research Labs change management proactive configuration management cfg2html tool in-house monitoring tools and EMS component monitoring

HP-UX Patch Program 32 conclusions- recommendations successful with ITRC tools and security_patch_check tool for patch selection patch proactive notifications looking to ETSE to reduce system administration time for patch management make better use of QPK take advantage of the continuous improvements with ITRC patch tools patch installation is labor intensive and time consuming across 150 systems Philips Research Labs conclusions

HP-UX Patch Program 33 Beckman Coulter Presented by: Chris Maehara

HP-UX Patch Program 34 background Beckman Coulter makes products that are used in hospital laboratories, physicians' offices and group practices. The company provides a variety of systems for medical research, drug discovery and biotechnology applications. business supported by various HP-UX and NT servers Oracle applications, SAMBA, and Veritas for system backup MC/ServiceGuard used for high availability and to reduce planned downtime. Beckman Coulter background

HP-UX Patch Program 35 background (Cont.) 13 HP-UX servers L-class, N-class, two V2600s HP-UX 11.0 and system administrators 24x7 Critical System Support (CSS) No onsite support 100% uptime goal Beckman Coulter background

HP-UX Patch Program 36 patching strategy philosophy quarterly proactive patching rolling upgrades using MC/ServiceGuard 4 stage rollout master depot of patches for each supported OS release cleanup command used patches kept for 1 year text file kept in separate directory for all patches ever applied standard configurations minimize complexity Beckman Coulter patch strategy

HP-UX Patch Program 37 patching strategy patch selection CSS contract delivers proactive patch bundle CPM (ITRC tool) delivers proactive notifications CPM sends notifications of newly released patches based on system configurations weekly review of CPM notifications patches added to patch depot matrix of patch dependencies use of SD master patch depot minimizes issues with patch dependencies Beckman Coulter patch strategy

HP-UX Patch Program 38 patching strategy patch warnings patch warnings reviewed prior to patch application generally, patches with warnings left as is Beckman Coulter patch strategy

HP-UX Patch Program 39 patching strategy patch application perform rolling upgrades copy all patches to be applied to a software depot, regardless of whether or not the patches are from a download or a CD. ensures no corrupted patches keep two versions of patch in depot only latest patch will install Beckman Coulter patch strategy

HP-UX Patch Program 40 patching strategy patch application use SD GUI to install more user friendly can make modifications if necessary without exiting the operation after installation, review log files verify successful installation verify configured cleanup patches Beckman Coulter patch strategy

HP-UX Patch Program 41 change management change requests required sign-off by business leads MeasureWare and ITO monitors systems and changes Beckman Coulter change management

HP-UX Patch Program 42 conclusions- recommendations Beckman Coulter conclusions well planned/tested rollouts regular scheduled proactive patch applications proactive better than reactive always read “special installation” instructions do not “force install” a patch use SD to resolve patch dependencies

HP-UX Patch Program 43 summary all customers had some kind change management process for patching all customers did some level of testing of patches prior to rolling into production all customers are using some level of proactive patching customers used a combination of HP support services, ITRC tools, and SupportPlus patch bundles (QPK) HA and/or redundant environments aid with reducing downtime security patches are “classed” differently patches with warnings are rarely removed from a system all customers summary

HP-UX Patch Program 44 questions? HP (Laurie Schoenbaum) Nestle (Terri Mando) Brigham Young University (John Payne) Philips Research Labs (Donie Collins) Beckman Coulter (Chris Maehara) all customers summary