1. HP World 2005 Real Life HP- UX Patching Strategies Steven E Protter Senior Systems Administrator I.S.N. Corporation

2. HP-UX Patching: Outline Presenter information –Qualifications and experience. –Warning !! –How I got here.

3. HP-UX Patching: Outline Patching Philosophy –If it isn’t broke, don’t fix it (A real life mess) –Generally Accepted principles –Three Star approach –Explanation of the star system –Security concerns –No strategy fits all

4. HP-UX Patching: Outline What is a patch? –Why a systems administrator should care –The depot file –What might be in a patch

5. HP-UX Patching: Outline Where to get a patch –Support Plus CD –ITRC patch database –Custom designed by HP

6. HP-UX Patching: Outline Tools to help with patching –security_patch_check –Custom Patch Manager (CPM) –ITRC forums –Building a bundle in the ITRC patch database.

7. HP-UX Patching: Outline Building a custom patch library –Including patches to cut # of boots –Including non-patch depot software –Removing superseded releases & patches. –A real life run through

8. Nuts & bolts

9. Qualifications and Experience 14 ½ Years at the Jewish United Fund Software AG and Oracle DBA A decade of systems administration experience Survived an actual loss of data disaster. Five years as a Linux systems administrator

10. HP-UX Patching: Warning Today is August 14, 2005 My body has no idea what time zone it is in.

11. HP-UX Patching: How I got here Left Tel Aviv August 2. Drove from NY to San Francisco via the Grand Canyon. Traveled over 7,000 miles to be here.

12. HP-UX Patching: How I got here

14. HP-UX Patching: Philosophy If it isn’t broke, don’t fix it –HP-UX 11.00 rollout. –Recommended patches were not installed –Omniback II was unable to run Enterprise backups. –System had to be booted three times in prime time during the first day of production.

15. HP-UX Patching: Philosophy If it isn’t broke, don’t fix it –This strategy can not work. –HP-UX is too complex to not have patches. –Its not classroom theory, its real life experience.

16. HP-UX Patching: Philosophy If “it isn’t broke don’t fix it was a valid strategy, we’d still have to get to work like this:

17. HP-UX Patching: Generalities Immediately after a cold OS installation you install the following: –Diagnostics –Gold Base Depot (Core Os defects) –A Gold Applications bundle –Hardware enablement bundle. –Gold Quality Pack depot

18. HP-UX Patching: Extras Immediately after the general installation: –Install security patches –Install patches required for the applications –Install patches to deal with real situations –Tune the kernel

19. HP-UX Patching: 3 Star approach Only three star patches –Three star patches are widely tested and the least likely to have problems. –Caveat Patcher: Three star patches have been recalled. –Quarterly bundles are three star patches. –Some critical security patches are not three star patches. If you wait too long, you may incur the security problem.

20. HP-UX Patching: Star System From Charles Keenan: HP-UX CSE –1 Star: Functional testing by HP to verify that a patch fixes the problem it is supposed to fix. No unwanted side effects discovered. –2 Star: Patch has been installed in a certain number of customer environments with no problems reported. –3 Star: Patch has been stress- and performance- tested by HP in a simulated customer mission- critical environments using common application stacks. Not all patches undergo this testing. –WARNING: patch contains warnings. You may still need to use it.

21. HP-UX Patching: Security!? Your support contract may require you to install security patches. Your continued employment may require you to install security patches. Government regulation may require you to install security patches. There are good tools to find out what security patches you need.

22. HP-UX Patching: No size fits all You need a strategy that keeps your systems running smoothly. You need a strategy that meets your organizations needs.

23. Real Life Strategy

24. HP-UX Patching: JUF Jewish United Fund has security concerns. When Homeland security goes orange, we got regular security patrols. $200 million in annual revenue depended on the HP-9000 servers.

25. HP-UX Patching: JUF A third server was purchased for more thorough testing. Quarterly bundles, applications, security patches and other priority patches were bundled an installed in the sandbox.

26. HP-UX Patching: JUF 2-4 weeks in the sandbox. This box could be booted during business hours. 2-4 weeks in the development (12 user) server. Bi-weekly maintenance. 2-4 weeks of monitoring after release into production (200 users).

27. HP-UX Patching: JUF Every Friday whether there was work scheduled or not a make_tape_recovery backup was made. Copies of these backups went off site. We regular ran recovery tests on the sandbox

28. “Ignite is Your Friend.” Steven E Protter Senior Systems Administrator, I.S.N. Corporation

29. “Ignite is Free.” Hewlett-Packard Corporation

30. HP-UX Patching What is a patch? –A fix for an OS defect –Enable new hardware and software –Deliver new or enhanced functionality –Provide useful utilities Charles Keenan: HP-UX CSE

31. HP-UX Patching Patch naming convention –PHCO: A patch for commands and libraries –PHKL: A kernel patch (boot time!) –PHNE: Networking patch –PHSS: Other HP-UX subsystems. Charles Keenan: HP-UX CSE

32. HP-UX Patching Cool tricks and commands I –swlist –l product –a is_patch –Lists the patches –swlist –l product *,c=patch | more –swlist –l file PHCO_24630 Charles Keenan: HP-UX CSE

33. HP-UX Patching Cool tricks and commands II –swlist –l fileset –a patch_state –x show_superseded_patches=true *,c=patch | more Charles Keenan: HP-UX CSE

34. HP-UX Patching Cool tricks and commands III –swlist –l patch –x show_superseded_patches=true OS- Core.CMDS-AUX Charles Keenan: HP-UX CSE

35. HP-UX Patching Cool tricks and commands V –swlist -l patch –swlist -l patch | grep -v ^\#

36. HP-UX Patching Never do this: –The –q –qq option –These options tell the SD/UX program to ignore warnings and errors. This is such a bad thing someone else had to tell me what these options were. Never use them.

37. HP-UX Patching Cool tricks and commands IV –cleanup –c 1 # commits patches getting back /var space –cleanup -p -d # preview –cleanup –p –d /tmp/protter.depot # full path required Steven E Protter via hp education or forums.itrc.hp.com & Bill Hassell

38. HP-UX Patching: Outline Why a systems administrator should care: –Your system might stop working –You might want to take a vacation or day off –Because a lot of experienced Administrators say you should

39. HP-UX Patching: Where to get ITRC Patch database Quarterly patch bundles Custom patches ITRC Custom patch manager

40. HP-UX Patching: Building a patchset http://itrc.hp.com Click patch/firmware database Click HP-UX Choose your patches Select dependencies Download Ignite Backup and installation

41. HP-UX Patching: Building a patchset

46. HP-UX Patching: Download options

47. HP-UX Patching: Download notes: Individual patches are ascii, you must remember this when you ftp them from a pc. Use sftp to get them from your pc to your HP- UX box to avoid ascii/binary heck…. zip,gzip or tar packages are binary. A quick story about ascii/binary

48. HP-UX Patching: Real Life!! While recovering from a complete loss of data the development staff uploaded an ftp of their programs from one of the developers C drives. No oracle applications would compile. I was tired, but asked, are you sure you did the upload binary? Answer: Of course, I’ve been doing this for years.

49. HP-UX Patching: Real Life!! 20 man hours were invested. An HP Support call was opened because nobody trusted the disk integrity. Oracle tar was opened and escalated three times. They had us write a new simple program with the motif gui. A light bulb went off over my head. Try the ftp again. I like good movies, can I watch? Problem solved.

50. HP-UX Patching: Building a patchset Why I like the ftp download option –Sometimes those zip downloads just stop –I can leave ftp to run and not worry about keeping a browser going –Gives me time for a snack or a nap –Gives me time for planning or backup –The bundle comes with a script to build a custom patch depot

51. HP-UX Patching: Patch Download Options Run a browser on an HP-UX Box –Advantage: No binary/ascii problem. –Disadvantage: Management might not let you. Snarf –Third party program can be run on one designated HP-UX box to gather patches for others. –Still, management might not let you do this.

52. HP-UX Patching: Patch Download Options Have a patch box –A PC dedicated to the task or an old HP- UX box in the DMZ which would allow for ftp access. Disable or swremove unneeded services. –Make sure every transfer step on files ending in the extension.depot is ascii or the installation will fail.

53. Tools to help with patching

54. HP-UX Patching: Building a patchset security_patch_check –Originally released as a patch –Comes with Bastille –Mostly gives you patches you can find in the patch database –Makes me feel warm and fuzzy

55. HP-UX Patching: Building a patchset CPM: Custom Patch Manager –A feature of itrc.hp.com –Comes with a usual script for patch and application inventory –Uploads system data for analysis

56. HP-UX Patching: Building a patchset Quarterly Patch bundles –Advantage: Well tested widely used. Not bleeding edge –Advantage: Easy to sell to management –Disadvantage: Security, DP 5.x patches may not be included. –Some Oracle applications need two star patches.

57. Real Life Run Through

58. HP-UX Patching: Real Life Objectives –Deploy the maximum number of patches and software with the minimum number of system boots. Minimize downtime. –Remove patches from the patch set which are superseded. –Minimize disk space used for patches –Insure we have a back out plan.

59. HP-UX Patching: Real Life Work Plan –make_tape_recovery (Ignite is my best friend) –security_patch_check –ITRC Patch database –Check www.hp.com/go/software –Prepare a large custom depot

60. HP-UX Patching: Real Life Important points –Read the patch notes –Try to avoid using recalled patches –Have a backup plan –Test patches in a server that can tolerate down time.

61. HP-UX Patching: Real Life Good Stuff –My depot is too big and contains patches that are superseded a few times, what to do? –cleanup –p –d # preview –cleanup –p

62. HP-UX Patching: Real Life Example, my /home/spring.2005.depot –cd /home/spring.2005.depot –du –sk shows 2488634 kb (2.4 GB) –There are three versions of secure shell –cleanup –p –cleanup –p –d $PWD

63. HP-UX Patching: Real Life Example, my /home/spring.2005.depot –cleanup –d $PWD –Did not clean up software depots, they need to be handled differently. –du –sk now reports: 2332936 2.3 GB –Its not a lot of space but everything helps.

64. HP-UX Patching: Real Life Cleaning up the installed software –This is a manual process. –cd /home/spring.2005.depot –swremove -d -x enforce_dependencies=true Secure_Shell @ $PWD

65. HP-UX Patching: Real Life Cleaning up the installed software –swremove the unwanted software –swremove -d -x enforce_dependencies=true Secure_Shell,r=A.03.91.002 @ $PWD –swcopy the latest revision into the depot

66. HP-UX Patching: Real Life Cleaning up and revising the installed software –swcopy the latest revision into the depot –cd /home/secsh (location is where you actually downloaded the depot) –swcopy -s ${PWD}/T1471AA_A.04.00.000_HP- UX_B.11.11_32+64.depot \* @ /home/spring.2005.depot

67. HP-UX Patching: Final stuff How to set up a patch depot on an NFS share –Add the patch location to the /etc/exports configuration file –exportfs –av # verbose re-export of shares –cd /depot_location –swreg –l depot /depot_location/patch.depot –From remote machine: –swinstall –x autoreboot=true –s hostname:/patch.depot \*

68. HP-UX Patching: Real Life Done for today!!!!

69. HP-UX Patching: Real Life Questions and hopefully answers

70. “Never be afraid to ask a question” Steven E Protter Senior Systems Administrator I.S.N. Corporation

71. Thank you for coming