BMC BladeLogic Windows Patching Troubleshooting Lazar Gersh

Introduction Windows Patching Life Cycle Patch Catalog: contains patch information Patch Analysis Job: analyzes targets Patch Remediation Job: Downloads / Ensures that payload is available Builds BLPackages and Deploy Jobs Creates a Batch Job to bundle Deploy Jobs Batch or Deploy Job: installs patches

Patch Catalog Shavlik metadata is downloaded Source: Configuration / Patch Global Configuration / Shavlik URL Configuration Destination: <fileserver>/patch/catalog/catalog_XXX/ ISSUE: Metadata cannot be downloaded due to connectivity problem Proxy: Configuration / Patch Global Configuration / All Operating Systems Firewall: Validate that access to Shavlik URLs is allowed Shavlik site: Validate that the files are available Shavlik metadata and utilities are passed to Windows Helper Server (WHS) ISSUE: ACL issues where WHS cannot be reached or access not authorized Shavlik metadata is parsed ConvertXML.exe: Shavlik metadata is decrypted BLPatchCheck2.exe: Patch information is extracted ISSUE: One of commands fails due to corrupted metadata or OS related issue Catalog is updated with the information from the Shavlik metadata Information from XML file is filtered based on Catalog Product Filters ISSUE: Patch Object cannot be added/updated due to RBAC issue Logs to collect Catalog Update Job log, appserver.log, WHS rscd.log

Patch Analysis Life Cycle Shavlik metadata transferred Fileserver location: <fileserver>/patch/catalog/catalog_XXX Target location: <agent>/tmp/WindowsCatalog_XXX_[target] Note: debug agent log of respective server mentions the actual path Analysis scan performed [N/A to 8.0+] BLPatchCheck2 (no flags): usage Ensures supported version of executable is present BLPatchCheck2 0: analysis scan with HFNetChk6b Available patch bulletins, KBs, supersedence, detection logic [N/A to 8.0+] BLPatchCheck2 1: payload validation with PD5 Download URL, silent install command Note: For BLPatchCheck2, if CAB is passed, XML will be extracted first Analysis report generated Results: <agent>/tmp/WindowsCatalog_XXX_[target]/shavlik_results.xml Debug: C:\Trace.txt (Trace.txt.old from previous analysis scan)

Troubleshooting Job failed Job succeeded Shavlik metadata was not copied to the target One of BLPatchCheck2 phases did not complete Results were not passed back to the appserver Job succeeded Analysis reported unexpected patch as missing Analysis did not show missing expected patch Analysis report conflict with 3rd-Party tool Job hung, timed out, failed otherwise Gathering the right information

Shavlik metadata was not copied to the target Check if metadata exists in the expected fileserver location (Fileserver location: <fileserver>/patch/catalog/catalog_XXX/hfnetchk6b.xml) Check if this nsh location is accessible from the appserver If file does not exist, Catalog Update Job may need to be rerun Issue copying the file to the target (Target location: <agent>/tmp/WindowsCatalog_XXX_[target]) Agent issue ACL / RBAC authorization issue Agent not started or host unreachable OS permission issue UAC or mapped to invalid local Administrator Logs to collect Patch Analysis Job log From fileserver and target: rscd.log

One of BLPatchCheck2 phases did not complete [N/A to 8.0+] BLPatchCheck2 (no flags): usage Old agent installed showing invalid usage information Server properties not refreshed where RSCD_DIR may have changed System PATH is too long causing issues executing nexec commands BLPatchCheck2 0: analysis scan with HFNetChk6b Unregistered stPatchAssessment.dll Cannot extract XML from CAB Command completed with failure (more in Trace.txt) [N/A to 8.0+] BLPatchCheck2 1: payload validation with PD5 Unregistered stPackager.dll HFNetChk6b and PD5 files are not in sync Logs to collect Patch Analysis Job log From target: rscd.log and Trace.txt

Results were not passed back to the appserver Check if correct results file can be generated (Results: <agent>/tmp/WindowsCatalog_XXX_[target]/shavlik_results.xml) Rerun the Analysis Job and watch agent tmp directory Permission issue where results file cannot be created BLPatchCheck2 0 phase did not complete correctly Manual run: BLPatchCheck2 0 hfnetchk6b.xml results.xml Copy hfnetchk6b.xml to <agent>/sbin Check agent log for actual command switches which were passed Review results.xml (if exists) and trace.txt files for clues Check if results file can be copied back to the appserver Perform nsh copy directly from appserver Firewall issue where traffic is not allowed Logs to collect Patch Analysis Job log From target: rscd.log and Trace.txt appserver.log

Analysis reported unexpected patch as missing Review the "reason" to why the patch is reported missing Found in Patch Analysis Results or Trace.txt Validate the reason on the target system Verify on patch vendor (ex: MS) site that the logic for reason is valid List mode (Include filter) used during Analysis Disables patch supersedence (open Catalog / Bulletin ID for info) May show old patch as missing Reason happens to be valid for old patch or product (IE6 versus IE7) Patch installation was incomplete Deploy Job failed to install the patch Deploy Job was not configured to reboot or failed to reboot the server Deploy Job timed out before installing this patch More in <agent>/Transactions/log/bldeploy-xxx.log Logs to collect Bulletin and KB number of patch(es) in question Patch Analysis Job run results From target: Trace.txt and bldeploy-xxx.log (if applicable)

Analysis did not show missing expected patch Validate that the patch is considered by Analysis Job Check that the patch is present in the Catalog Catalog is up-to-date and contains correct filter Patch is not obsolete (otherwise, a newer patch is considered) Check that correct patch type is selected in analysis options Security Patch, Security Tool, Non-Security Patch, Service Pack Check that the patch is not excluded Validate that analysis scan checks for this patch Search Trace.txt for Bulletin ID If expected Bulletin not found Validate that scan finished completely (Trace.txt ends gracefully) Validate Shavlik files in catalog/catalog_xxx folder If expected Bulletin found, review the logic stack with explanation Check vendor site for applicability Is target on the correct SP? Logs to collect Bulletin and KB number of patch(es) in question Patch Analysis Job run results From target: Trace.txt

Analysis Report conflict with 3rd-Party tool Identify a clear list of patches and expectation Falls under one of two previous slides BladeLogic reports patches for installed products, not new products Example: IE7 is installed – IE8 will not be offered Example: SP1 installed – SP2 patches will not be offered Validate that KB number is an actual patch and not an article Example: MS10-068 (KB983539) – patches: KB981550 and KB982000 Live Browse / Hotfixes versus Patch Analysis Only reports Security Patches Metadata found in <fileserver>/templates Shavlik Technologies: scan engine and metadata If the patch is not released, Analysis will not report it http://www.shavlik.com/support/xmlsubscribe.aspx http://forum.shavlik.com/

Job hung, timed out, failed otherwise Information to gather Patch Analysis Job log appserver.log appserver details (if the job is in hung state) Found in: Configuration / Infrastructure Management Capture 2 exports 10 minutes apart Fileserver: rscd.log Hanging target (if identified): rscd.log and Trace.txt

Knowledge Base List of articles illustrating some of discussed areas (1/2): Slide: Patch Catalog KA322967 - Windows Patch Catalog Update error: Error occurred while downloading: http://xml.shavlik.com/data/hfnetchk6b.xml, retry completed KA324358 - Update Catalog error: Error while add/update patch in depot: [object], A depot object of the same type with the name '[object]' already exists in this group. Slide: Shavlik metadata was not copied to the target KA326503 - Patch Analysis Job error: Cannot copy shavlik metadata file from '//fileserver/storage/[catalog]/hfnetchk6b.xml' to '//target/[RSCD]/tmp:...: No such file or directory) Slide: One of BLPatchCheck2 phases did not complete KA326912 - Windows Patch Analysis warning: BlPatchCheck2 version is less than 1.3, Scan options will be ignored. KA325280 - Windows Patch Analysis error: The version of shavlik SDK on target is not supported by appserver. Patch Analysis will not continue KA312351 - Windows Patch Analysis error: Unable to load the Shavlik Scan Engine dll. Error: 80040154 KA322916 - Windows Patch Analysis error: 80004005 Assessment file is missing or unable to download or decab KA346838 - Windows Patch Analysis error: Error executing Analyzer: ExitCode = -1 or -1073741510 KA346824 - Windows Catalog or Patch Analysis error: Error executing BLPatchCheck2 (Exit code: -1073741790) Slide: Results were not passed back to the appserver KA323631 - Windows Patch Analysis error: Failed to copy //target/agent/shavlik_resultsXXXXX.xml to /app/bladelogic/tmp/application_server/shavlik_resultsXXXXX.xml KA352322 - Windows Patch Analysis error: com.bladelogic.shared.xml.XMLServiceException: File /[appserver]/tmp/shavlik_results_xxx.xml is empty. Cannot parse.

Knowledge Base List of articles illustrating some of discussed areas (2/2): Slide: Analysis reported unexpected patch as missing KA324245 - Windows Patch Deploy error: BlPatchCheck2.exe failed. Return Code: 1 KA317306 - Windows Patch Deploy error: Error 80004005 Could not load patch details file KA312236 - Patch Analysis or Deploy error: "Unable to load the Shavlik Packager Engine dll. Error: 80070005" or "Exit Code = 11" KA322198 - Windows Patch Analysis inconsistent results with and without whitelist KA353940 - Partially superseded Bulletins show as superseded in the Catalog Slide: Analysis did not show missing expected patch KA326110 - Windows patches not reporting during Patch Analysis Job KA348230 - Windows Patch Analysis reports incorrect results: Caught one - COM (-2147467259), An invalid character was found in text content KA347121 - Windows Patch Analysis does not report new missing patches after upgrade to 8.1 KA323772 - Windows Patch Analysis does not report new patches Slide: Analysis Report conflict with 3rd-Party tool KA351516 - Live Browse Hotfixes does not report latest patches installed KA308992 - How long it will take for the patches to be reflected in BladeLogic/Shavlik after Microsoft has released them? KA325243 - Inconsistent results of BladeLogic Windows Patch Analysis versus Windows Live Update (or another tool) General: KA312026 - FAQ: Windows Patch Analysis, Deployment, Troubleshooting KA354095 - Deploy Job error: APPLY failed for server [target]. Exit code = -4001 KA322337 - Difference between Installed and EffectivelyInstalled KA316294 - Windows patch Deploy Job rebooted the server even though the blpackage said 'no package items require reboot' KA323327 - Windows Patch Deploy fails with exit code 17025

Reading Logs Rscd.log (fileserver) Locating unique folder with metadata for the Catalog: CM: access (D:/storage01/patch/catalog/catalog_2000018/hfnetchk6b.xml, 0) = 0 Rscd.log (target) Command that performs the analysis: CM: remote command: BLPatchCheck2 0 -pt 13 -s "C:/Program Files/BMC Software/BladeLogic/8.0/RSCD/tmp/WindowsCatalog_2_107_lg-win2k3/hfnetchk6b.xml" "C:/Program Files/BMC Software/BladeLogic/8.0/RSCD/tmp/WindowsCatalog_2_107_lg-win2k3/shavlik_results.xml“ Trace.txt Graceful completion of the scan – last two lines: LG-WIN2K3 … Generating output for = LG-WIN2K3 LG-WIN2K3 … Scanned Server Count = 1, MAX count = 1 Detection logic stack: LG-WIN2K3 … TESTING MS11-025 Bulletin ID = MS11-025 Filecount = 1 Regcount = 0 LG-WIN2K3 … File C:\Program Files\Common Files\Microsoft Shared\VC\msdia80.dll ErrNum=0 LG-WIN2K3 … File C:\Program Files\Common Files\Microsoft Shared\VC\msdia80.dll (8.0.50727.4053) C 2 8.0.50727.6195 (AC 5) LG-WIN2K3 … A file was tested LG-WIN2K3 … Did not pass file tests

Common Notations rscd.log: //target/<rscd_dir>/rscd.log in debug mode Set agent to debug: open C:\WINDOWS\rsc\log4crc.txt Change "info1" to "debug" in: <category name="rscd" priority="info1" appender=“<rscd_dir>/rscd.log" debugappender="stderr"/> Restart the agent. attachment too large for email: upload to ftp.bmc.com/incoming/[create_folder_with_ticket]/ bldeploy log: //target/<rscd_dir>/Transactions/log/bldeploy-xxx.log matching Deploy Job

This slide is designed to be an opening or closing slide This slide is designed to be an opening or closing slide. This will allow the presenter to have a presentation cued up in slideshow mode without being on the title slide. The audience can take their seats, leave, or have open discussion with this slide up.