1. 1 Symantec NetBackup OpenStorage Troubleshooting Short Course Symantec NetBackup OpenStorage

2. Sample Agenda Symantec NetBackup OpenStorage 2 Introduction 1 NetBackup Overview 2 Logs Review 3 NBU tools 4 GEN_DATA 5

3. Agenda Symantec NetBackup OpenStorage 3 1 2 When to bring it to Symantec and what to take 3 ? &….. 4 Null Plugin External tools

4. NetBackup from 5000 feet… 4 Symantec NetBackup OpenStorage

5. NetBackup Terminology Master Server – The ‘brains’ of NetBackup – Maintains the ‘catalog’ (the list of backup images), schedules jobs, maintains information about all devices and systems Media Server – The data movers to/from devices – Responsible for performing the backups, duplications, restores. These are the servers where the backup devices are ‘attached’. Client – The machine being backed up, e.g., file server, application server. 5 Symantec NetBackup OpenStorage

6. NetBackup Terminology Important Considerations – This is a networked application – Many a slip betwixt the cup and lip OR connections can hide many problems – Versions across the domain, the master must ALWAYS be at the highest level – Network tools are good to know and carry – The world is larger than a data center, WAN links can be painful – Correlation is NOT causation NBU may be displaying the error, but it may not be the cause of it 6 Symantec NetBackup OpenStorage

7. NetBackup Relationships RMMS – Remote Media Management System Monitors the relationship between media servers and disk volumes (A-HA!) Reports back to EMM with capacity info DPS – Disk Polling Service Runs once minute Non-tuneable interval, but the timeout can be tuned via touchfiles EMM – Enterprise Media Management database Maintains state of media servers and storage servers Resides on Master server 7 Symantec NetBackup OpenStorage

8. Before we get started….. I’m assuming a few things: – You’ve got logging cranked up in verbosity Do this via the GUI under Server properties OR Vi the bp.conf file (add VERBOSE=5), for legacy logs Use the vxlogcfg command to up the settings for some or all unified logs – You know how to run NBU to do backups, restores, and opt_dups – You have all of your NBU services up and running (EMM is kind of a big deal) – You like to read lots of logs and take great joy in reading process flow diagrams – You have sufficiently caffeinated/B12’d/sugared yourself up 8 Symantec NetBackup OpenStorage

9. Logs Review Legacy Logs – Edit bp.conf – Add “VERBOSE=5” to the file OR – You can also simply go to your Master Server GUI and up this in the Properties settings for the Master, it will propagate to the Media servers Unified Logs – The vxlogcfg command: vxlogcfg -a -p -o -s DebugLevel=6 vxlogcfg -a -p -o -s DiagnosticLevel=6 –ProductID is 51216 for NetBackup –OID (Originator ID) – there is an exhaustive list of these in the Troubleshooting Guide (can be “ALL”) –Run both commands for all OIDs you need to up log levels 9 Symantec NetBackup OpenStorage

10. NBU Tools 10 Symantec NetBackup OpenStorage

11. GEN_DATA Generating data with bpbkar – Directly out of memory – Eliminates latency on client, comes directly from media server or client – VERY fast, VERY granular controls – Tuneable for deduplication rate, repetitiveness, compression, encryption, etc. – Good for: Checking if the client is causing timeouts or issues with the path to storage Finding any bottlenecks at the storage server Testing a storage server’s capacity (network and storage-wise) Isolating the non-client parts of the config – Currently only available on unix/linux systems, no windows 11 Symantec NetBackup OpenStorage

12. Null Plugin Available in NetBackup 7.5 (coming soon) – Availble in Windows: install_path\NetBackup\bin\support Unix/linux: user/openv/netbackup/bin/support – Simply place in the ost-plugins directory to use – This allows you to create an STU, as type null_stu This storage unit writes to a null device, isolating the client/media server relationship – There are a few caveats: ANY data written is not recoverable, yes, it’s a fast backup.. That’s not recoverable. There is an 18 hour window from install, you must restart RMMS to reset the timer 12 Symantec NetBackup OpenStorage

13. NBSU NetBackup Support Utility – EXCELLENT tool, please refer back to our previous session for this one! 13 Symantec NetBackup OpenStorage

14. Other Tools – Wireshark GREAT tool for seeing network traffic and filtering conversations and determining latencies – Appneta Subscription based, systems and config assessment tool used by NBU support – Cain and Abel Network traffic tool that can replay and rebuild network conversations across protocols – bpclntcmd Internal NBU command, helps to find resolved hostnames to determine connectivity http://www.symantec.com/business/support/index?page=content&id=TECH1769 – bptestnetconn Test for connectivity between nodes and resolution issues within the config http://www.symantec.com/business/support/index?page=content&id=HOWTO43773 14 Symantec NetBackup OpenStorage

15. More Other Tools – bpstsinfo –pi Determines plugin loading, functionality and version The Swiss Army knife of NBU This can also be used with other arguments to determine connectivity and communication with the storage server – ldd List Dynamic Dependencies Run against shared object lib (plugin) If anything is missing = uh-oh… – Dependency Walker depends.exe ldd for Windows www.dependencywalker.com 15 Symantec NetBackup OpenStorage

16. When to bring it to Symantec Run through the troubleshooting we’ve covered previously Use Symantec to help you to run in parallel SYMC support KNOWS NBU, coordinate with them on any shared customer 16 Symantec NetBackup OpenStorage

17. When SYMC will call you…. As soon as there is indication of a plugin issue – Any “20600xx” errors in logs – Any plugin error or indication of plugin not functioning as expected 17 Symantec NetBackup OpenStorage

18. Coordination between Support Organizations TSANet – www.tsanet.org www.tsanet.org – All STEP/OST Partners have an account on TSANet – This is basic info at this point, but both sides can add detail to their contact information – This is a first step at handing off a case – All STEP/OST Partners also have customer numbers and can open support cases with Symantec on behalf of shared customers 18 Symantec NetBackup OpenStorage

19. Thank you! Copyright © 2010 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. 19