1. BMC BladeLogic Windows Patching Troubleshooting
Lazar Gersh

2. Introduction Windows Patching Life Cycle
Patch Catalog: contains patch information
Patch Analysis Job: analyzes targets
Patch Remediation Job:
Downloads / Ensures that payload is available
Builds BLPackages and Deploy Jobs
Creates a Batch Job to bundle Deploy Jobs
Batch or Deploy Job: installs patches

3. Patch Catalog Shavlik metadata is downloaded
Source: Configuration / Patch Global Configuration / Shavlik URL Configuration
Destination: <fileserver>/patch/catalog/catalog_XXX/
ISSUE: Metadata cannot be downloaded due to connectivity problem
Proxy: Configuration / Patch Global Configuration / All Operating Systems
Firewall: Validate that access to Shavlik URLs is allowed
Shavlik site: Validate that the files are available
Shavlik metadata and utilities are passed to Windows Helper Server (WHS)
ISSUE: ACL issues where WHS cannot be reached or access not authorized
Shavlik metadata is parsed
ConvertXML.exe: Shavlik metadata is decrypted
BLPatchCheck2.exe: Patch information is extracted
ISSUE: One of commands fails due to corrupted metadata or OS related issue
Catalog is updated with the information from the Shavlik metadata
Information from XML file is filtered based on Catalog Product Filters
ISSUE: Patch Object cannot be added/updated due to RBAC issue
Logs to collect
Catalog Update Job log, appserver.log, WHS rscd.log

4. Patch Analysis Life Cycle
Shavlik metadata transferred
Fileserver location: <fileserver>/patch/catalog/catalog_XXX
Target location: <agent>/tmp/WindowsCatalog_XXX_[target]
Note: debug agent log of respective server mentions the actual path
Analysis scan performed
[N/A to 8.0+] BLPatchCheck2 (no flags): usage
Ensures supported version of executable is present
BLPatchCheck2 0: analysis scan with HFNetChk6b
Available patch bulletins, KBs, supersedence, detection logic
[N/A to 8.0+] BLPatchCheck2 1: payload validation with PD5
Download URL, silent install command
Note: For BLPatchCheck2, if CAB is passed, XML will be extracted first
Analysis report generated
Results: <agent>/tmp/WindowsCatalog_XXX_[target]/shavlik_results.xml
Debug: C:\Trace.txt (Trace.txt.old from previous analysis scan)

5. Troubleshooting Job failed Job succeeded
Shavlik metadata was not copied to the target
One of BLPatchCheck2 phases did not complete
Results were not passed back to the appserver
Job succeeded
Analysis reported unexpected patch as missing
Analysis did not show missing expected patch
Analysis report conflict with 3rd-Party tool
Job hung, timed out, failed otherwise
Gathering the right information

6. Shavlik metadata was not copied to the target
Check if metadata exists in the expected fileserver location
(Fileserver location: <fileserver>/patch/catalog/catalog_XXX/hfnetchk6b.xml)
Check if this nsh location is accessible from the appserver
If file does not exist, Catalog Update Job may need to be rerun
Issue copying the file to the target
(Target location: <agent>/tmp/WindowsCatalog_XXX_[target])
Agent issue
ACL / RBAC authorization issue
Agent not started or host unreachable
OS permission issue
UAC or mapped to invalid local Administrator
Logs to collect
Patch Analysis Job log
From fileserver and target: rscd.log

7. One of BLPatchCheck2 phases did not complete
[N/A to 8.0+] BLPatchCheck2 (no flags): usage
Old agent installed showing invalid usage information
Server properties not refreshed where RSCD_DIR may have changed
System PATH is too long causing issues executing nexec commands
BLPatchCheck2 0: analysis scan with HFNetChk6b
Unregistered stPatchAssessment.dll
Cannot extract XML from CAB
Command completed with failure (more in Trace.txt)
[N/A to 8.0+] BLPatchCheck2 1: payload validation with PD5
Unregistered stPackager.dll
HFNetChk6b and PD5 files are not in sync
Logs to collect
Patch Analysis Job log
From target: rscd.log and Trace.txt

8. Results were not passed back to the appserver
Check if correct results file can be generated
(Results: <agent>/tmp/WindowsCatalog_XXX_[target]/shavlik_results.xml)
Rerun the Analysis Job and watch agent tmp directory
Permission issue where results file cannot be created
BLPatchCheck2 0 phase did not complete correctly
Manual run: BLPatchCheck2 0 hfnetchk6b.xml results.xml
Copy hfnetchk6b.xml to <agent>/sbin
Check agent log for actual command switches which were passed
Review results.xml (if exists) and trace.txt files for clues
Check if results file can be copied back to the appserver
Perform nsh copy directly from appserver
Firewall issue where traffic is not allowed
Logs to collect
Patch Analysis Job log
From target: rscd.log and Trace.txt
appserver.log

9. Analysis reported unexpected patch as missing
Review the "reason" to why the patch is reported missing
Found in Patch Analysis Results or Trace.txt
Validate the reason on the target system
Verify on patch vendor (ex: MS) site that the logic for reason is valid
List mode (Include filter) used during Analysis
Disables patch supersedence (open Catalog / Bulletin ID for info)
May show old patch as missing
Reason happens to be valid for old patch or product (IE6 versus IE7)
Patch installation was incomplete
Deploy Job failed to install the patch
Deploy Job was not configured to reboot or failed to reboot the server
Deploy Job timed out before installing this patch
More in <agent>/Transactions/log/bldeploy-xxx.log
Logs to collect
Bulletin and KB number of patch(es) in question
Patch Analysis Job run results
From target: Trace.txt and bldeploy-xxx.log (if applicable)

10. Analysis did not show missing expected patch
Validate that the patch is considered by Analysis Job
Check that the patch is present in the Catalog
Catalog is up-to-date and contains correct filter
Patch is not obsolete (otherwise, a newer patch is considered)
Check that correct patch type is selected in analysis options
Security Patch, Security Tool, Non-Security Patch, Service Pack
Check that the patch is not excluded
Validate that analysis scan checks for this patch
Search Trace.txt for Bulletin ID
If expected Bulletin not found
Validate that scan finished completely (Trace.txt ends gracefully)
Validate Shavlik files in catalog/catalog_xxx folder
If expected Bulletin found, review the logic stack with explanation
Check vendor site for applicability
Is target on the correct SP?
Logs to collect
Bulletin and KB number of patch(es) in question
Patch Analysis Job run results
From target: Trace.txt

11. Analysis Report conflict with 3rd-Party tool
Identify a clear list of patches and expectation
Falls under one of two previous slides
BladeLogic reports patches for installed products, not new products
Example: IE7 is installed – IE8 will not be offered
Example: SP1 installed – SP2 patches will not be offered
Validate that KB number is an actual patch and not an article
Example: MS (KB983539) – patches: KB and KB982000
Live Browse / Hotfixes versus Patch Analysis
Only reports Security Patches
Metadata found in <fileserver>/templates
Shavlik Technologies: scan engine and metadata
If the patch is not released, Analysis will not report it

12. Job hung, timed out, failed otherwise
Information to gather
Patch Analysis Job log
appserver.log
appserver details (if the job is in hung state)
Found in: Configuration / Infrastructure Management
Capture 2 exports 10 minutes apart
Fileserver: rscd.log
Hanging target (if identified): rscd.log and Trace.txt

13. Knowledge Base
List of articles illustrating some of discussed areas (1/2):
Slide: Patch Catalog
KA Windows Patch Catalog Update error: Error occurred while downloading: retry completed
KA Update Catalog error: Error while add/update patch in depot: [object], A depot object of the same type with the name '[object]' already exists in this group.
Slide: Shavlik metadata was not copied to the target
KA Patch Analysis Job error: Cannot copy shavlik metadata file from '//fileserver/storage/[catalog]/hfnetchk6b.xml' to '//target/[RSCD]/tmp:...: No such file or directory)
Slide: One of BLPatchCheck2 phases did not complete
KA Windows Patch Analysis warning: BlPatchCheck2 version is less than 1.3, Scan options will be ignored.
KA Windows Patch Analysis error: The version of shavlik SDK on target is not supported by appserver. Patch Analysis will not continue
KA Windows Patch Analysis error: Unable to load the Shavlik Scan Engine dll. Error:
KA Windows Patch Analysis error: Assessment file is missing or unable to download or decab
KA Windows Patch Analysis error: Error executing Analyzer: ExitCode = -1 or
KA Windows Catalog or Patch Analysis error: Error executing BLPatchCheck2 (Exit code: )
Slide: Results were not passed back to the appserver
KA Windows Patch Analysis error: Failed to copy //target/agent/shavlik_resultsXXXXX.xml to /app/bladelogic/tmp/application_server/shavlik_resultsXXXXX.xml
KA Windows Patch Analysis error: com.bladelogic.shared.xml.XMLServiceException: File /[appserver]/tmp/shavlik_results_xxx.xml is empty. Cannot parse.

14. Knowledge Base
List of articles illustrating some of discussed areas (2/2):
Slide: Analysis reported unexpected patch as missing
KA Windows Patch Deploy error: BlPatchCheck2.exe failed. Return Code: 1
KA Windows Patch Deploy error: Error Could not load patch details file
KA Patch Analysis or Deploy error: "Unable to load the Shavlik Packager Engine dll. Error: " or "Exit Code = 11"
KA Windows Patch Analysis inconsistent results with and without whitelist
KA Partially superseded Bulletins show as superseded in the Catalog
Slide: Analysis did not show missing expected patch
KA Windows patches not reporting during Patch Analysis Job
KA Windows Patch Analysis reports incorrect results: Caught one - COM ( ), An invalid character was found in text content
KA Windows Patch Analysis does not report new missing patches after upgrade to 8.1
KA Windows Patch Analysis does not report new patches
Slide: Analysis Report conflict with 3rd-Party tool
KA Live Browse Hotfixes does not report latest patches installed
KA How long it will take for the patches to be reflected in BladeLogic/Shavlik after Microsoft has released them?
KA Inconsistent results of BladeLogic Windows Patch Analysis versus Windows Live Update (or another tool)
General:
KA FAQ: Windows Patch Analysis, Deployment, Troubleshooting
KA Deploy Job error: APPLY failed for server [target]. Exit code = -4001
KA Difference between Installed and EffectivelyInstalled
KA Windows patch Deploy Job rebooted the server even though the blpackage said 'no package items require reboot'
KA Windows Patch Deploy fails with exit code 17025

15. Reading Logs Rscd.log (fileserver)
Locating unique folder with metadata for the Catalog:
CM: access (D:/storage01/patch/catalog/catalog_ /hfnetchk6b.xml, 0) = 0
Rscd.log (target)
Command that performs the analysis:
CM: remote command: BLPatchCheck2 0 -pt 13 -s "C:/Program Files/BMC Software/BladeLogic/8.0/RSCD/tmp/WindowsCatalog_2_107_lg-win2k3/hfnetchk6b.xml" "C:/Program Files/BMC Software/BladeLogic/8.0/RSCD/tmp/WindowsCatalog_2_107_lg-win2k3/shavlik_results.xml“
Trace.txt
Graceful completion of the scan – last two lines:
LG-WIN2K3 … Generating output for = LG-WIN2K3
LG-WIN2K3 … Scanned Server Count = 1, MAX count = 1
Detection logic stack:
LG-WIN2K3 … TESTING MS11-025
Bulletin ID = MS11-025
Filecount = 1
Regcount = 0
LG-WIN2K3 … File C:\Program Files\Common Files\Microsoft Shared\VC\msdia80.dll ErrNum=0
LG-WIN2K3 … File C:\Program Files\Common Files\Microsoft Shared\VC\msdia80.dll ( ) C (AC 5)
LG-WIN2K3 … A file was tested
LG-WIN2K3 … Did not pass file tests

16. Common Notations rscd.log:
//target/<rscd_dir>/rscd.log in debug mode
Set agent to debug:
open C:\WINDOWS\rsc\log4crc.txt
Change "info1" to "debug" in:
<category name="rscd" priority="info1" appender=“<rscd_dir>/rscd.log" debugappender="stderr"/>
Restart the agent.
attachment too large for
upload to ftp.bmc.com/incoming/[create_folder_with_ticket]/
bldeploy log:
//target/<rscd_dir>/Transactions/log/bldeploy-xxx.log matching Deploy Job

17. This slide is designed to be an opening or closing slide
This slide is designed to be an opening or closing slide. This will allow the presenter to have a presentation cued up in slideshow mode without being on the title slide. The audience can take their seats, leave, or have open discussion with this slide up.