1. Symantec NetBackup OpenStorage Logs and Troubleshooting Short Course

2. Sample Agenda Introduction 1 NetBackup Overview 2 Processes and logs 3
Legacy logs vs. Unified logs 4
Parsing (following the breadcrumbs) 5
Symantec NetBackup OpenStorage

3. Sample Agenda 1 Functional Log Gathering 2 Basics, from the bottom up
Training from Symantec 3
Materials for this preso 4
? & 5
Symantec NetBackup OpenStorage

4. NetBackup from 5000 feet…
Symantec NetBackup OpenStorage

5. NetBackup Terminology
Master Server
The ‘brains’ of NetBackup
Maintains the ‘catalog’ (the list of backup images), schedules jobs, maintains information about all devices and systems
Media Server
The data movers to/from devices
Responsible for performing the backups, duplications, restores. These are the servers where the backup devices are ‘attached’.
Client
The machine being backed up, e.g., file server, application server.
Symantec NetBackup OpenStorage

6. NetBackup Terminology
Catalog
The list of all backups, information, retention period, etc.
Backup Image
Warning: NetBackup Image != OpenStorage Image
NBU Backup images are composed of 1 to N image fragments.
Each NBU image fragment == OST image.
Backup Policy
Specifies what to back up on the client (e.g., files, database), when to perform the backup (generally multiple schedules, e.g., weekly full, daily incremental backup), where to send the backups (e.g., disk, tape) and how long to retain the backup image before expiring/deleting it.
Symantec NetBackup OpenStorage

7. NetBackup Terminology
Disk Volume
NetBackup’s name for an LSU
Disk Pool
A collection of Disk Volumes (LSUs)
Storage Unit
A means of specifying which media servers move the data to what device (disk pool, tape library) and to what extent to throttle job activity.
STU is the acronym, DSU acronym for Disk Storage Unit.
Storage Lifecycle Policies
Highly popular new way (introduced in NBU 6.5) to define a plan for how many backup copies to make, destination and retention.
E.g., back up to disk, make 1 copy to tape, another copy to off-site disk.
Symantec NetBackup OpenStorage

8. Before we get started….. I’m assuming a few things:
You’ve got logging cranked up in verbosity
Do this via the GUI under Server properties OR
Vi the bp.conf file (add VERBOSE=5), for legacy logs
Use the vxlogcfg command to up the settings for some or all unified logs
You know how to run NBU to do backups, restores, and opt_dups
You have all of your NBU services up and running (EMM is kind of a big deal)
You like to read lots of logs and take great joy in reading process flow diagrams
You have sufficiently caffeinated/B12’d/sugared yourself up
You are not allergic to some perl and NetBackup command lines
Symantec NetBackup OpenStorage

9. Crank it up… Legacy Logs OR Unified Logs Edit bp.conf
Add “VERBOSE=5” to the file OR
You can also simply go to your Master Server GUI and up this in the Properties settings for the Master, it will propagate to the Media servers
Unified Logs
The vxlogcfg command:
vxlogcfg -a -p <ProductID> -o <OID> -s DebugLevel=6
vxlogcfg -a -p <ProductID> -o <OID> -s DiagnosticLevel=6
ProductID is for NetBackup
OID (Originator ID) – there is an exhaustive list of these in the Troubleshooting Guide (can be “ALL”)
Run both commands for all OIDs you need to up log levels
Symantec NetBackup OpenStorage

10. Processes and Logs (Stuff we do and things we leave behind)
Symantec NetBackup OpenStorage

11. NetBackup Backup Processes
These diagrams are available in our Troubleshooting Guide
Symantec NetBackup OpenStorage

12. That’s a nice eyechart, what’s it mean?
Backup
Three things to think about:
Bpbrm
Bpbkar
Bptm
There is a lot going on, but these are the main processes
Symantec NetBackup OpenStorage

13. NetBackup Restore Processes
These diagrams are available in our Troubleshooting Guide
Symantec NetBackup OpenStorage

14. That’s a nice drawing of a spiderweb, what’s it mean?
Restore
Here, the focus will move to the bprestore, bprm, bpdm, and tar processes
Again, the logs will have much more to say about this…
Symantec NetBackup OpenStorage

15. NetBackup Forestry NetBackup Logs Named and produced per process
Two types:
Legacy Logging (older style)
Unified Logging (newer style)
Different processes are logged to each type of log, you will need some information to follow your job around
Time Window
Process ID
JobID from NetBackup
Image Name from NetBackup (this is a personal favorite, not necessarily needed)
Write that down…
Symantec NetBackup OpenStorage

16. Logs and where to find them (hint: near saws)
Legacy logs
Ascii text
Placed in the (/usr/openv/netbackup/logs or <INSTALLDIR>\netbackup\logs ) directory
Subdirectories named for the process that logs to them
Each log is named for its timestamp within the process directory
Sifting through these can be tedious
Unified logs
Unix: /usr/openv/logs Windows: <INSTALLDIR>\netbackup\logs
Comma separated values lists
Viewable as text, but must be viewed using “vxlogview” command for detail
Symantec NetBackup OpenStorage

17. Viewing logs Legacy logs A good text editor is your friend
Vi/vim/elvis/pico/emacs/etc… on unix
I like TextEdit on Windows
These are basic ascii logs, arm yourself
Process ID
JobID in NBU
Time frame
NetBackup Image name
Cd into your process directory and open the log that fits your timeframe
Start reading…..
Symantec NetBackup OpenStorage

18. Viewing logs Unified logs Vxlogview is powerful, use it wisely
There are a lot of options, but you need only a few to be effective
These are not necessarily readable without the vxlogview command
Many of the processes that are prefixed with “nb” log to these logs
These were meant to make logging easier and to have less jumping around between logging directories for the admin/support
Symantec NetBackup OpenStorage

19. Parsing logs Legacy logs
Cd into the directory of your favorite NBU process
You will want to search for “<16>” and “<32>” in the logs
A timeframe is a great help here
You can also use grep/sed/awk/perl/vbs to search the logs for JobID/PID/imagename/time
Scripts for tearing out the 16/32 messages are nice for doing a fast “rip and glance” approach to narrow your search
Symantec NetBackup OpenStorage

20. QaD script to search the Legacy logs for 16/32
#!/bin/bash ls /usr/openv/netbackup/logs | while read x do echo "$x:" echo "" cd $x for FILE in log* echo "${FILE}:" perl -ane 'print if /\<16\>|\<32\>/;' $FILE done cd /usr/openv/netbackup/logs
Symantec NetBackup OpenStorage

21. Parsing logs (s’more) Unified logs
Cd into the unified logs directory (/usr/openv/logs on unix)
As of and in 7.0, all processes that log here have directories
Cd into your favorite process directory and have a look around using:
“vxlogview –G . –d all > out”
This will generate a LOT of logs, you can minimize it by clearing the old files you don’t want in your timeframe out of the directory
Also, “vxlogview –G . –d all –hoursago <#hours>”
If you can narrow the window to “between then and now”, this is helpful
Also try the –b and –e arguments for begin and end times for processing
Redirect these commands to files for parsing more like legacy logs
The vxlogview man page is very helpful, even if it is a bit wordy….
Symantec NetBackup OpenStorage

22. Time to cut some trees
In the past, tar, gzip and Windows zip were all good to have on hand (they still are…)
Tar and gzip or simply archive and zip together all the log directories for all logs and send them along to Symantec
“Why do you need ALL of them?” - The core issue may not be the plugin or anything to do with OST
“Where can I put them?” - We do have a Support ftp site, that you will be given info to login to and add your logs to your incident directory when needed
“Can’t I just send you snippets?” -Nope, sorry. I know it’s easier, but we have to see NBU running, from start to end, all the daemons, all the bits and pieces
Nowadays, we have a utility, nbcplogs
/usr/openv/netbackup/bin/support/nbcplogs
And…. NBSU!!!
Symantec NetBackup OpenStorage

23. Basics from the bottom up
Check to be sure NBU is running
Sometimes it’s not…
Check EMM to be sure IT is running
This can cause weird issues with socket connect failures
Check your filesystems for space (EMM needs scratch space)
Ping all of the hosts in the config being used (master to media, and back, media to storage and back)
Network misconfigs or random firewall rule changes cause havoc
Use bpstsinfo –pi
If the plugin won’t load, it won’t backup
Use AdvanceDisk as a baseline check of NBU vs. Plugin issue
Symantec NetBackup OpenStorage

24. Coordination between Support Organizations
TSANet
All STEP/OST Partners have an account on TSANet
This is basic info at this point, but both sides can add detail to their contact information
This is a first step at handing off a case
All STEP/OST Partners also have customer numbers and can open support cases with Symantec on behalf of shared customers
Symantec NetBackup OpenStorage