1. www.novell.com Practical NDS ® iMonitor: Case Studies in Novell eDirectory ™ Diagnosis Duane Buss Senior Software Engineer Novell, Inc. dbuss@novell.com Tom Doman Senior Software Engineer Novell, Inc. tdoman@novell.com Steve McLain Senior Software Engineer Novell, Inc. smclain@novell.com Gary J. Porter Senior Network Analyst MindWorks, Inc. porter@myrealbox.com

2. Vision…one Net A world where networks of all types—corporate and public, intranets, extranets, and the Internet—work together as one Net and securely connect employees, customers, suppliers, and partners across organizational boundaries Mission To solve complex business and technical challenges with Net business solutions that enable people, processes, and systems to work together and our customers to profit from the opportunities of a networked world

4. Deployed Versions Novell eDirectory ™ and Novell Directory Services ® (NDS ® ) Product VersionBuild VersionPlatforms NetWare 5.1 SP4 (NDS 7)DS.nlm v7.57NetWare 5.1 NetWare 5.1 SP 4 (NDS 8)DS.nlm v8.79NetWare 5.1 eDirectory 8DS.nlm & DS.dlm v8.79NetWare 5.0,Win NT/2K eDirectory 8.5.xDS v85.23NetWare 5.x,Win,Solaris NetWare 6 (eDirectory 8.6)DS.nlm v10110.20NetWare 6 eDirectory 8.6.1DS v10210.43NW 5.1,NW 6,Win,Solaris,Linux NetWare 6 SP1 (eDirectory 8.6.2)DS.nlm v10310.17NetWare 6 eDirectory 8.6.2DS v103xx.xxNW 5.1,NW 6,Win,Solaris,Linux eDirectory 8.7DS v10410.xxNW 5.1,NW 6,Win,Solaris,Linux,AIX

5. Differences between eDirectory and Novell Directory Services (NDS) NetWare 6 NetWare NDSeDirectory NOS directory focused on managing NetWare ® servers A cross-platform, scalable, standards-based directory used for managing identities that span all aspects of the network—eDirectory is the foundation for eBusiness NetWare 5

6. Introduction Historical diagnostic tools Problems vs. symptoms eDirectory diagnostic case studies using iMonitor  The case of the unknown object  The case of the attribute that just wouldn’t sync  The case of the inconsistent replica  The case of the security-minded administrator  The case of the inconsistent entry  The case of the under-performing agent

7. Historical NDS Diagnostic Tools Diagnostic tools  DSTrace  DSBrowse  DSRepair  DSDiag  NDS Manager Tool access  Server console  Remote console  Telnet  pcAnywhere  Insight Board

8. The eDirectory Utility Knife Multiple tools in one You don’t leave the tool to go to another one You don’t have to access several different servers Agent Summary Agent Configuration NDS Trace NDS Repair DirXML ™ Tools Report Tool Search Tool Agent Health Object and Schema Browse Agent Synchronization Partition List Known Servers Agent Process Status Agent Activity, Verb and Event Statistics Plus: Inbound and Outbound Connection Monitor Error Information

9. What Kind of Problems Can I Diagnose? What problems are you seeing?  Time sync issues  Synchronization issues  NDS agent version problems  Communication issues  Schema issues  Improperly moved/removed servers  Inconsistent object/database  Agent process errors  Performance issues  DirXML ™ issues  Distributed issues  Many others…

10. Disclaimer The following case studies are based on real-world scenarios and depict systems which have been, at times, brutally battered, beaten, or otherwise mistreated. Viewer discretion is advised. The names of the perpetrators have been changed to protect the guilty. Their crimes include  Inappropriate usage of undocumented/advanced support switches in NDS Repair  Hardware failure  Improper removal of servers and/or replicas from the tree  Incorrect system configuration  Overzealous administration  Running pre-release (beta) code in production  Attempting to correct symptoms of the real problem  Poor network infrastructure and/or monitoring

11. Case #1 unknow object the case of the

12. Unknown Object Causes An object referenced by a mandatory attribute has been deleted Object is only a forward reference Object is an External Reference and the object has not yet been Backlinked, or the real object is unknown Object has Auxiliary Classes and you are viewing the object on a non-Aux Class compatible replica Object is being deleted Object is actually damaged (rare) Schema inconsistencies (rare) Ghost Objects (extremely rare)

13. Unknown Object—Missing Mandatory Detecting the case  Examine the attributes “Unknown Base Class” and “Unknown Auxiliary Class”  Compare the mandatory attributes required in the schema to the attributes on the object

14. Unknown Object—Missing Mandatory

15. Resolving the issue  Don’t panic  Is the missing attribute missing on all replicas or just some of the replicas If the attribute is missing on all replicas, add the missing attribute using LDAP, ConsoleOne ®, or iManage (the object will remain unknown) If the object is consistent on some replicas but not others use iMonitor to resend that one object from the consistent replica to the other replicas  As a last resort, remove the object, then recreate it

16. Unknown Object—Forward Reference Detecting the case  Entry information flags show “Reference”  The replica type shown in the entry information is something other than subordinate  The object may not have all attributes  Walking the replica ring shows the object is not unknown on all replicas

17. Unknown Object—Forward Reference

18. Resolving the issue  Don’t panic, forward references happen all the time in the course of synchronization and will become known when the actual object successfully synchronizes  Check for and resolve any schema and object sync problems, then wait for the sync operation to finish  In rare cases use “Single Object Send” to send the entry from a consistent replica to all other replicas

19. Unknown Object—External Reference Detecting the case  Entry information flags show “Reference”  There are not ‘real’ server names in the replica frame  The partition type is subordinate  The attribute list is abbreviated although the authenticated user has full rights to the object being viewed

20. Unknown Object—External Reference

21. Resolving the issue  Don’t panic—this is not generally a problem  External References are only viewable in iMonitor or DSBrowse  If the entry information flags show “Temporary Reference,” by design, this server may never receive the base class of the real object  Check and resolve any errors shown in “Agent Process Status” in the External Reference section  Start the “Reference Check” background process and wait for it to complete

22. Unknown Object—Aux Class Detecting the case  Check the version of the servers in the replica ring  Examine the “AuxClass Object Class Backup,” “auxClassCompatibility,” and “Object Class” attributes

23. Unknown Object—Aux Class

24. Resolving the issue  Don’t panic  Not a problem, it is safe to ignore these unknowns  Upgrade older servers to 8.x or later version of eDirectory and apply appropriate service patches

25. Unknown Object—Deleted Detecting the case  Entry information flags don’t show “Present”  There may be obituary attributes on the object  These objects are only visible in utilities such as iMonitor

26. Unknown Object—Deleted

27. Resolving the issue  This object will generally finish deleting without manual intervention  Wait for sync to finish  Run the “Purger” background process

28. Unknown Object—Ghost Object Detecting the case  Entry information flags show “Reference”  Walking the replica ring shows the object is unknown on all replicas

29. Unknown Object—Ghost Object

30. Resolving the issue  Delete the object if it is not needed

31. Case #2

32. Attribute Mismatch?

33. Filter Desired Attributes

34. Getting to Replica Synchronization

35. Replica Synchronization

36. Getting to Entry Synchronization

37. Entry Synchronization

38. Take Action?

39. Schema Definition

40. Release Version 8.6 and Later

41. Houston, We Have a Problem Obituary Report

42. Entry with Obituaries

43. Houston, We Have a Problem Unknown Objects

44. Houston, We Have a Symptom

47. Case #3 replica the case of the inconsiste t

48. Houston, What Exactly Is the Problem? Using NDS Trace

49. Analyzing NDS Trace Data Target NDS Agent Update Packet

50. Switching to Trace on Another Server

53. Houston, What Exactly Is the Problem? Using NDS Trace

54. More Info on -609

55. NDS Error Information

56. Inspect William Object

57. Inspect Schema Class Definition

58. Mandatory

59. Filter Desired Attributes

60. Compare the Object Around the Replica Ring

61. Aaaaaaha!

62. Quicker Check of Synchronization

63. Resynchronize All Data from the Master

64. Case #4

65. Obituary Report One Entry Still Has Not Been Fully Processed

67. Report Configuration

68. Report Configuration Server Information

69. Report Server Information

71. Obviously, the 609-6 NDS Agent Is Up

72. Agent Process Status Limber

74. Inspect the Server Entry

75. This is an External Reference

76. This is a real copy of the 609-6 object Aaaaaaha! An Overzealous Security-Minded Administrator

77. Case #5

78. Agent Synchronization Error Replica Synchronization Detail

82. Inconsistent Object Go to NDS Repair

83. Single Object Repair

85. Troubleshooting Guidelines Don’t panic Look for root causes, not symptoms After taking steps to correct a problem, make sure the correct background processes run to completion Get training General rules 1.Solve communication problems first 2.Solve synchronization issues first (Schema and Object) 3.Make sure your system is correctly time-synced 4.Run the correct (latest) support pack

86. Case #6

87. eDirectory Performance Factors Replica type and placement NDS version mix Overall tree design— IO115—Directory or Database: Choosing the Right Tool for the Job TUT223—Avoiding the Top eDirectory Issues TUT329—Novell eDirectory Deployment TUT33—eDirectory In Depth Load and application distribution Hardware and network capability Database cache settings Index definitions Bindery usage

88. Agent Activity

90. Agent Configuration

91. Fault to Hit Ratio Current vs. Maximum New in eDirectory 8.5

92. Profiling Data

93. Conclusion Historical diagnostic tools Problems vs. symptoms eDirectory diagnostic case studies using iMonitor  The case of the unknown object  The case of the attribute that just wouldn’t sync  The case of the inconsistent replica  The case of the security-minded administrator  The case of the inconsistent entry  The case of the under-performing agent