Practical NDS ® iMonitor: Case Studies in Novell eDirectory ™ Diagnosis Duane Buss Senior Software Engineer Novell, Inc. Tom Doman Senior Software Engineer Novell, Inc. Steve McLain Senior Software Engineer Novell, Inc. Gary J. Porter Senior Network Analyst MindWorks, Inc.

Vision…one Net A world where networks of all types—corporate and public, intranets, extranets, and the Internet—work together as one Net and securely connect employees, customers, suppliers, and partners across organizational boundaries Mission To solve complex business and technical challenges with Net business solutions that enable people, processes, and systems to work together and our customers to profit from the opportunities of a networked world

Deployed Versions Novell eDirectory ™ and Novell Directory Services ® (NDS ® ) Product VersionBuild VersionPlatforms NetWare 5.1 SP4 (NDS 7)DS.nlm v7.57NetWare 5.1 NetWare 5.1 SP 4 (NDS 8)DS.nlm v8.79NetWare 5.1 eDirectory 8DS.nlm & DS.dlm v8.79NetWare 5.0,Win NT/2K eDirectory 8.5.xDS v85.23NetWare 5.x,Win,Solaris NetWare 6 (eDirectory 8.6)DS.nlm v NetWare 6 eDirectory 8.6.1DS v NW 5.1,NW 6,Win,Solaris,Linux NetWare 6 SP1 (eDirectory 8.6.2)DS.nlm v NetWare 6 eDirectory 8.6.2DS v103xx.xxNW 5.1,NW 6,Win,Solaris,Linux eDirectory 8.7DS v10410.xxNW 5.1,NW 6,Win,Solaris,Linux,AIX

Differences between eDirectory and Novell Directory Services (NDS) NetWare 6 NetWare NDSeDirectory NOS directory focused on managing NetWare ® servers A cross-platform, scalable, standards-based directory used for managing identities that span all aspects of the network—eDirectory is the foundation for eBusiness NetWare 5

Introduction Historical diagnostic tools Problems vs. symptoms eDirectory diagnostic case studies using iMonitor  The case of the unknown object  The case of the attribute that just wouldn’t sync  The case of the inconsistent replica  The case of the security-minded administrator  The case of the inconsistent entry  The case of the under-performing agent

Historical NDS Diagnostic Tools Diagnostic tools  DSTrace  DSBrowse  DSRepair  DSDiag  NDS Manager Tool access  Server console  Remote console  Telnet  pcAnywhere  Insight Board

The eDirectory Utility Knife Multiple tools in one You don’t leave the tool to go to another one You don’t have to access several different servers Agent Summary Agent Configuration NDS Trace NDS Repair DirXML ™ Tools Report Tool Search Tool Agent Health Object and Schema Browse Agent Synchronization Partition List Known Servers Agent Process Status Agent Activity, Verb and Event Statistics Plus: Inbound and Outbound Connection Monitor Error Information

What Kind of Problems Can I Diagnose? What problems are you seeing?  Time sync issues  Synchronization issues  NDS agent version problems  Communication issues  Schema issues  Improperly moved/removed servers  Inconsistent object/database  Agent process errors  Performance issues  DirXML ™ issues  Distributed issues  Many others…

Disclaimer The following case studies are based on real-world scenarios and depict systems which have been, at times, brutally battered, beaten, or otherwise mistreated. Viewer discretion is advised. The names of the perpetrators have been changed to protect the guilty. Their crimes include  Inappropriate usage of undocumented/advanced support switches in NDS Repair  Hardware failure  Improper removal of servers and/or replicas from the tree  Incorrect system configuration  Overzealous administration  Running pre-release (beta) code in production  Attempting to correct symptoms of the real problem  Poor network infrastructure and/or monitoring

Case #1 unknow object the case of the

Unknown Object Causes An object referenced by a mandatory attribute has been deleted Object is only a forward reference Object is an External Reference and the object has not yet been Backlinked, or the real object is unknown Object has Auxiliary Classes and you are viewing the object on a non-Aux Class compatible replica Object is being deleted Object is actually damaged (rare) Schema inconsistencies (rare) Ghost Objects (extremely rare)

Unknown Object—Missing Mandatory Detecting the case  Examine the attributes “Unknown Base Class” and “Unknown Auxiliary Class”  Compare the mandatory attributes required in the schema to the attributes on the object

Unknown Object—Missing Mandatory

Resolving the issue  Don’t panic  Is the missing attribute missing on all replicas or just some of the replicas If the attribute is missing on all replicas, add the missing attribute using LDAP, ConsoleOne ®, or iManage (the object will remain unknown) If the object is consistent on some replicas but not others use iMonitor to resend that one object from the consistent replica to the other replicas  As a last resort, remove the object, then recreate it

Unknown Object—Forward Reference Detecting the case  Entry information flags show “Reference”  The replica type shown in the entry information is something other than subordinate  The object may not have all attributes  Walking the replica ring shows the object is not unknown on all replicas

Unknown Object—Forward Reference

Resolving the issue  Don’t panic, forward references happen all the time in the course of synchronization and will become known when the actual object successfully synchronizes  Check for and resolve any schema and object sync problems, then wait for the sync operation to finish  In rare cases use “Single Object Send” to send the entry from a consistent replica to all other replicas

Unknown Object—External Reference Detecting the case  Entry information flags show “Reference”  There are not ‘real’ server names in the replica frame  The partition type is subordinate  The attribute list is abbreviated although the authenticated user has full rights to the object being viewed

Unknown Object—External Reference

Resolving the issue  Don’t panic—this is not generally a problem  External References are only viewable in iMonitor or DSBrowse  If the entry information flags show “Temporary Reference,” by design, this server may never receive the base class of the real object  Check and resolve any errors shown in “Agent Process Status” in the External Reference section  Start the “Reference Check” background process and wait for it to complete

Unknown Object—Aux Class Detecting the case  Check the version of the servers in the replica ring  Examine the “AuxClass Object Class Backup,” “auxClassCompatibility,” and “Object Class” attributes

Unknown Object—Aux Class

Resolving the issue  Don’t panic  Not a problem, it is safe to ignore these unknowns  Upgrade older servers to 8.x or later version of eDirectory and apply appropriate service patches

Unknown Object—Deleted Detecting the case  Entry information flags don’t show “Present”  There may be obituary attributes on the object  These objects are only visible in utilities such as iMonitor

Unknown Object—Deleted

Resolving the issue  This object will generally finish deleting without manual intervention  Wait for sync to finish  Run the “Purger” background process

Unknown Object—Ghost Object Detecting the case  Entry information flags show “Reference”  Walking the replica ring shows the object is unknown on all replicas

Unknown Object—Ghost Object

Resolving the issue  Delete the object if it is not needed

Case #2

Attribute Mismatch?

Filter Desired Attributes

Getting to Replica Synchronization

Replica Synchronization

Getting to Entry Synchronization

Entry Synchronization

Take Action?

Schema Definition

Release Version 8.6 and Later

Houston, We Have a Problem Obituary Report

Entry with Obituaries

Houston, We Have a Problem Unknown Objects

Houston, We Have a Symptom

Case #3 replica the case of the inconsiste t

Houston, What Exactly Is the Problem? Using NDS Trace

Analyzing NDS Trace Data Target NDS Agent Update Packet

Switching to Trace on Another Server

Houston, What Exactly Is the Problem? Using NDS Trace

More Info on -609

NDS Error Information

Inspect William Object

Inspect Schema Class Definition

Mandatory

Filter Desired Attributes

Compare the Object Around the Replica Ring

Aaaaaaha!

Quicker Check of Synchronization

Resynchronize All Data from the Master

Case #4

Obituary Report One Entry Still Has Not Been Fully Processed

Report Configuration

Report Configuration Server Information

Report Server Information

Obviously, the NDS Agent Is Up

Agent Process Status Limber

Inspect the Server Entry

This is an External Reference

This is a real copy of the object Aaaaaaha! An Overzealous Security-Minded Administrator

Case #5

Agent Synchronization Error Replica Synchronization Detail

Inconsistent Object Go to NDS Repair

Single Object Repair

Troubleshooting Guidelines Don’t panic Look for root causes, not symptoms After taking steps to correct a problem, make sure the correct background processes run to completion Get training General rules 1.Solve communication problems first 2.Solve synchronization issues first (Schema and Object) 3.Make sure your system is correctly time-synced 4.Run the correct (latest) support pack

Case #6

eDirectory Performance Factors Replica type and placement NDS version mix Overall tree design— IO115—Directory or Database: Choosing the Right Tool for the Job TUT223—Avoiding the Top eDirectory Issues TUT329—Novell eDirectory Deployment TUT33—eDirectory In Depth Load and application distribution Hardware and network capability Database cache settings Index definitions Bindery usage

Agent Activity

Agent Configuration

Fault to Hit Ratio Current vs. Maximum New in eDirectory 8.5

Profiling Data

Conclusion Historical diagnostic tools Problems vs. symptoms eDirectory diagnostic case studies using iMonitor  The case of the unknown object  The case of the attribute that just wouldn’t sync  The case of the inconsistent replica  The case of the security-minded administrator  The case of the inconsistent entry  The case of the under-performing agent