2. NetWare5 and Advanced NDS Management - course designed for: Dynamic Mutual Funds, 40 King St. W. Toronto

3. Introductions Jim Gillett Contract Instructor jimeg@rogers.com –CNA 3,4,5 CNE 4,5, –CNI –MCP MCSE MCSB –MCT Class 9:00 am – 4:00 pm

4. Course Outline Day One- –Overview of NDS and eDirectory –Good Preventive Maintenance Measures for the NDS Database –NDS Management Tools & Logic Source Day Two- –Understanding How NDS Processes Management Tasks –The Relationship between NetWare Servers and NDS Day Three- –NDS Troubleshooting

5. NDS and eDirectory eDirectory (NDS) Benefits (printout) Tree Design Partitions Replicas Synchronization

6. Good Preventive Maintenance Measures for the NDS Database

7. Network Physician(?) Basic NDS Health Check –NDS version –Time Synchronization –Partition & Replica Continuity Complete NDS Checkup –Basic Checks plus - –NDS background Processes: External references Obituaries Remote server IDs Unknown objects NDS schema –SET Parameters

8. Your NDS Tree- Static or Dynamic? Static NDS Tree –Simple changes daily –New Partition or Server every few months MONTHLY CHECKUPS Dynamic NDS Tree –Partition or Server weekly –Redesigning the Tree –Major upgrading WEEKLY CHECKUPS

9. Health Check ToolKit NDS Manager (workstation-based) DSRepair (server-based) DS Diagnostics (server-based, reports) DS Trace (granular, view background processes) Use these tools to monitor, diagnose, repair and update NDS replicas and versions. –Each tool has advantages and useful features.

10. Additional useful tools CRON command-line Scheduler Server logs and Records – Server name, – NetWare revision – NDS version – IP,IPX numbers – Partitions and replicas – Hardware upgrades – Server abends, failures, error messages( *.err; *.log) – Addition of software, patches, service packs – Peripheral devices

11. Exercises – 60 minutes Exercise #1 Work with NDS Manager Create a report with DsDiag. Schedule an event with CRON. Work with DSRepair Create several DS Trace script files and observe results.

12. NDS Management Tools & Logic Source

13. NDS Error Guidelines 1 st be sure physical connectivity exists between servers to allow synchronization Be patient; allow time for NDS to synch after changes Use error codes and server documentation Usually NDS error conditions have occurred because of an inability of a synchronization process to complete.

14. Troubleshooting Method Clearly define problem Define possible causes (check server records, error logs etc.) Assess possible solutions Backup then try a solution Allow time for effect Document the solution Plan how to avoid a repetition

15. Troubleshooting Tools DSRepairidentify and repair DSBrowseidentify (NW5) NDS Manageridentify and repair DSTraceidentify and verify DSViewidentify (NW4) Logic Sourceresearch Knowledge Baseresearch

16. DSRepair Syntax >> DSREPAIR -[switch] –A Advanced mode –LLog file –UUnattended –RC Creates a database backup (dsrepair.dib) –PFlags unknown objects as reference –XK2(killer switch 2) Arbitrarily removes all replicas and turns all objects into ext. references –XK3(killer switch 3) Further to XK2, clears the ‘backlinked’ status of all ext. references.

17. The DSRepair Log Files Time Synchronization Log –Server name –DS version –Replica Depth –Time source, In Synch, Delta Replica Synchronization Log –Partition Name and Replicas Stored on this Server –Time of last sync & Error Codes if any

18. DSRepair Advanced Options Repair Local Database Server Information Replica & Partition Operations (many options) Check Volume Objects Check External references (backlinks, obituaries) Schema and New Epoch Database Dump file

19. DSView (NW4) & DSBrowse (NW5) Allows detailed view of NDS objects and attributes With DSBrowse, can delete objects or “receive” or “send” attributes or values of a specific object (take time and try out both tools)

20. NDS Manager Explain and/or Demonstrate –Partition continuity –Replica synchronization –Send updates –Receive updates –Check and/or Update DS version –Delete server –Remove server –Assign new master replica –Context-sensitive Help files

21. DS Trace Flag Types * + - ! Viewing Filters ( +, - ) Force Actions( * ) Tunable Settings ( ! ) Set TTF= ON/OFF Set DS Trace = ON/OFF –See DSTrace printout for all the detailed switches

22. Research Tools Knowledge Base –At support.novell.com –Manuals and TIDs Logic Source for NDS –Advanced NDS Documentation –On Cdrom (check now)

23. Exercises When you try to add a replica of the [Root] partition to a server, you get an error. (Exercise #2- 90 min.) Using troubleshooting method and NDS management and research tools, determine the cause and potential solutions Implement the solution Document the problem and its resolution

24. End of Day One See you tomorrow!! You may leave when ready! - course designed for: Dynamic Mutual Funds, 40 King St. W. Toronto

25. Understanding How NDS Processes Management Tasks

26. Why Be Patient? Because the Directory is a distributed (split up), and replicated (copied) database it takes some time for changes to be updated to all copies or replicas of the database. The term ‘loosely consistent’ can properly be applied to the NDS Directory.

27. Types of Changes Simple –Normal day-to-day administration of the Directory tree. Creating leaf objects Modifying attributes of leaf objects –Can be performed at any time, multiple locations Complex –Higher level administration tasks Partitioning Adding replicas to server Merging partitions Redesigning tree Moving containers (Ous) –Be sure one operation is completely synchronized before starting another

28. Handling NDS Wisely Some Guidelines: –Do health check before any complex operation –Control complex NDS management tasks centrally –Perform complex tasks during off-hours if possible –Backup database using DSRepair –RC command –Allow time for the replicas to synchronize normally –Verify that one operation or part of an operation fully completes before proceeding to another

29. Partitions

30. Creating Partitions, Adding Replicas Be sure that the servers holding the master replicas needed to perform this operation are available. (If a server receives a replica of a certain partition it will also receive a ‘subordinate reference’ to any child partitions). Ensuring that all needed servers are on-line will allow the operation to complete promptly. Allow time for replica lists to be updated and any changes to be propagated to all servers holding replicas of the updated partitions.

31. Demonstration Run DSRepair for 0 errors then: – create a new partition –Create a child partition of the new partition –Unload NDS from a server holding a replica of the parent partition (unload DS.nlm) –Merge child partition with parent Set DSTrace=on and Set DSTrace=*H –Observe Directory screen display Run DSRepair and notice any errors The point? Simply that you need to allow one NDS operation to completely finish (synchronize) before attempting another.

32. Looking Behind the Scenes DS Diagnostics allows you to generate NDS status reports –Load DSDIAG.nlm –Select preferences –Select generate report –Select parameters of report –Generate desired report and examine

33. Looking Behind the Scenes (II) DSTrace with –ST switch creates more detailed statistics on screen- Some useful switches>>

34. What are the Background NDS Synchronization Processes? Replica synchronization Janitor Flat cleaner Purger Obituary Limber

35. Replica Synchronization Maintains consistency of data between replicas 3 levels –immediate (10 secs hold time, most events) –Slow (30 secs hold time, attribute modifications, etc.) –Heartbeat (or skulker) every 30 minutes, automatic, default –To display Set dstrace=on Set dstrace=+s (or, +skulker, or +sync) Set dstrace=+misc Set dstrace=+part Set dstrace=*H (to force sync process)

36. Janitor process Makes sure deleted objects are purged from each replica (using the obituary and flat cleaner processes) Verifies information re: replicas on server, status of server and time synchronization. Periodically optimizes database Runs on database initialization and again every 2 minutes

37. Flat cleaner Scheduled by janitor to occur every 60 minutes Purges unused bindery (NW3) and external reference objects and attributes Removes obituaries (deleted objects) if in purgeable state To observe: Set dstrace=+J Set dstrace=+misc Set dstrace=*F

38. Purger Process Purges unused objects and attributes Processes obituaries through their different states Triggered by synch process To observe: Set dstrace=+J Set dstrace=*H

39. Obituary Process 4 step process of deleting objects –1 (Flag=0000) Object deleted- initially just on local replica, others not yet notified –2 (Flag=0001) Notified- all replicas informed of change –3 (Flag=0002) OK to purge- all replicas respond, change implemented on their side –4 (Flag=0004) Purgeable- really gone

40. Limber Process Verifies server credentials such as- –Network address –NetWare version –Distinguished names of servers on Replica List –Name of the Tree –Authentication of credentials To observe- –Set dstrace=+Limber –Set dstrace=*L

41. Schema Synchronization At initialization, then every 240 minutes Ensures updates, additions to schema are sent to all replicas To observe- –Set dstrace=+schema –Set dstrace=*SS

42. Backlinker Process Takes care of external references External reference are quick pointers to servers holding replicas of a certain object, stored on a server that has referenced that object but does not have a replica containing that object itself. They are not needed if the server later receives a replica containing that object, or if partitioning changes make it invalid. The Backlinker process runs 2 hours after initialization and every 780 minutes after. To observe-Set dstrace=+blink (or +backlink) - Set dstrace=*B

43. NDS database is initialized when- Volume SYS: is mounted DS.nlm is unloaded and reloaded Is forced by typing –Set dstrace= *. To view completion type –Set dstrace=+init –Set dstrace=+misc –Set dstrace=+J

44. Exercises Do exercises 3 and 4 (printouts, 60 minutes) Use DSTrace to view & confirm results. Be sure all obituaries are resolved. Using NDS Manager –View partitions –Create partitions –Merge partitions –Move containers

45. Types of Replicas Master –First, needed for partitioning operations Read/Write –Additional replica(s), fault tolerance Read/Only –Cannot directly make changes or provide authentication Subordinate Reference –Created by system, holds replica list or ring

46. External References Basically a very simple pointer or placeholder to an object that is referenced by a server for which there is no replica on the server. An external reference just holds a few attributes of the object. Its purpose is to cache references to external objects for quicker access. A backlink is an attribute of an external reference which points to the location of the real replica holding the object and all its attributes.

47. Partition & Replica guidelines Do not create unnecessary partitions Partition at upper layers based on location Maintain fewer than 3500 objects per partition For fault tolerance have at least 3 replicas of each partition (esp. [ROOT] partition) Partition for bindery services if applicable (NW3) Use WAN Traffic Manager if applicable to off-load synchronization to low-use periods

48. NetWare5- Faster Synchronization NetWare4 uses Sync-up-to Vector or timestamp –Replicas communicate and synchronize one at a time,around the ring –Read-only replicas and subordinate references can trigger synch process –Replicas are searched sequentially for changes –One object per synch packet. NetWare5 uses Transitive Vector –Each server contains a list of timestamps of all replicas in the ring, not just those on the one server. –Read-only replicas and subordinate references do not trigger synch process –Changes are cached for each replica, no sequential search –Multiple objects per packet.

49. Replica States & Exercises see replica states printout (support.novell.com) Exercises (30 min.)- –Add a replica to a server –Delete a replica –Change a replica type –Receive updates (local replica overwritten; cannot perform on a master replica) –Send updates (local changes sent)

50. The Relationship Between NetWare Servers and NDS

51. Removing a Server from the Tree Holiday Group Tour –where’s Mary?? Problem! –She’s got our funds! She knows schedule! What does server do? –Does it hold master replica(s)? Move them. –Is it a time source server? Assign another. –Important apps? (GroupWise) Move them. Inform group before removing. –(Remove NDS) Is the removal temporary? –Use placeholders.(Nwconfig-NW5; Install-NW4)

52. How to Remove a Server 1. Run DSRepair 2. Remove Master replicas from server 3. Change if time source server 4. If removal is temporary use a placeholder. 5. Remove NDS using NWConfig. 6. Verify all references removed. Can use ‘DSRepair –dsremove’ if necessary.

53. The –dsremove switch ‘NWConfig –dsremove’ allows arbitrary removal of NDS from a server (caution!) –Does not verify replica exists –Does not synchronize first –Does not stop on any errors –Does not require admin log-in –Might not remove all references as cleanly as normal method. – Exercise 6 – 30 minutes

54. Changing Server Credentials Can change server name and ID. Server ID (formerly internal ipx #) random 8-digit hex number- may want to standardize Change one at a time, restart server Correct volume names and license assignments Limber process verifies server IDs Use DSTrace ON, +limber, *L, *H to observe results ***Exercise #7, 30 minutes***

55. The Limber Process

56. Upgrading Server Hardware Backup the server files Run DSMaint, Install(NW4) or NWConfig(NW5) –First record Server name and ID –Select “Save local DS information prior to hardware upgrade” or “Prepare NDS for hardware upgrade”. NDS is now locked. –Do not perform any other replica or server changes during this time. – A file will be created called either Backup.ds or Backup.nds. –Copy this backup file from sys:\system to a workstation. –Upgrade the hard drive, with same version and server packs, into a temporary tree. Log in from a workstation. –Copy the backup.nds file to sys:\system then remove NDS and restore from the backup file using DSMaint, Install, or NWConfig ***Exercise #8 (30 minutes)***

57. When It Crashes If a hard drive crashes- –Hard drive failure, overheating, power supply or other hardware failure. Documentation needed –Server name, ID, replicas, services etc. Can use ‘DSMaint –PSE’ switch –Premium support engineer –Place server references on a placeholder object –‘Delete’ crashed server and volumes –Replace failed server –Use DSMaint to restore server references, tape for file system.

58. Placeholder ***Exercise #9 (30 minutes)***

59. Backups and Restores First-line defence- –Partition replicas are an ‘on-line’ dynamic backup –Maintain at least 3 of each partition Tape Backups- –the NDS Directory, File System and Server-specific Info(5 files) (5 files- servdata.nds,dsmisc.log,volsinfo.txt,startup.ncf,autoexec.ncf) –Partial restore or entire tree –Restore NDS before file system so rights can be restored. TCOPY (tcopy2.exe) and TBACKUP (tback3.exe) useful tools Database Dump- –Run DSRepair –RC (Use CRON) –Creates sys:system\dsrepair.dib, must run on each server.

60. Backup Guidelines Use Novell-certified backup hardware & software Use latest versions of TSA’s, drivers, etc. Use latest service packs, patches. Document your network diligently. Back up regularly and test the integrity of backups. Always restore NDS before the file system. If re-install operating system, re-apply service packs, patches etc. before restoring NDS and file system. ***Exercise #10 (30 minutes)***

61. End of Day Two One more to go!! You may leave when ready! - course designed for: Dynamic Mutual Funds, 40 King St. W. Toronto

62. NDS Troubleshooting

63. Uncommon Issues Inconsistent leaf objects Time un-synched Server 54 where are you? Who are you and how did you get here? Schema mis-matches Dead but not gone (obituaries)

64. Looking closer……. DSView- NW4.11, read-only, run on each server DSBrowse- NW5, can delete, send or receive updates NetWare Administrator- easy to read, but how to be sure reading from server interested in checking?? Unload ds.nlm from other close servers, as NWAdmin uses info from first server to reply. Check number of subordinates in each container, make notes, then check the view from another server and compare total number of objects.

65. Restoring consistency- ‘Receive updates’ on problem server –Replaces replica on server with over-write from another server containing a trusted replica ‘Send updates’ to other servers in ‘ring’ –Non-destructive –Updates all the other replicas ***Exercise #11 (30 minutes)***

66. Time Synchronization Why needed. What makes a timestamp? Time and IPX Time and IP Mixed IP/IPX Resolving time synchronization issues

67. (timesync) Why needed? (1) Used to accurately order changes in file system. Used to accurately record message times Used by applications to record events. Especially used by NDS to maintain consistency of updates across multiple replicas. The order of modifications must be accurately recorded so changes made on one object in two replicas can be prioritized correctly.

68. (timesync)What makes a timestamp? 3 parts- –1. UTC or Universal Time Coordinated (GMT) –2. Replica number where event took place –3. Event ID- a consecutive number from 1 to 65,535 to keep track of multiple events in the same second.

69. (timesync) Time and IPX Timesync is an easy yet comprehensive Novell proprietary time synchronization service –With NW4 it used IPX –With NW5 it can use either IP or IPX or both. –With IPX, there were 4 types of time servers Single reference-default authoritative time source Secondary- default all but 1 st server, accept time corrections Reference- form ‘committee’ with 2 to 7 Primary time servers Primary- communicate with Reference servers and vote on time –Uses timesync.cfg, simple text file for set parameters SAP, configured lists,directory tree mode, etc.

70. (timesync) Time and IP Still uses timesync.cfg Set timesources to NTP Internet sources Specify and use standard NTP port 123 Load monitor>server parameters-or-edit the timesync.cfg directly Set timesync configured sources=ON Set timesync time sources= :123 Single reference, Reference and Primary Servers all get their time directly from internet sources. If the connection fails to the internet they will use their own clocks temporarily. Pure IPX servers must be secondary time servers and require IP/IPX compatibility mode or IP and IPX on one server.

71. Time Un-synched? If time is set backwards on an authoritative time server, NDS realizes a problem and declares a state called “synthetic time” Uses timestamp IDs and allows timestamps to remain accurate relative to each other but compresses events compared to real-time until NDS time converges with real time. (automatic process) Displays “synthetic time” warnings at the server console.

72. Resolving “Synthetic Time” 1. Do Nothing- (preferred) –If time reversal was not too long, just wait. NDS will eventually ‘catch up’ to itself and the messages will stop. 2. Declare a ‘New Epoch’ –Load DSRepair –A –Perform at off-peak hours –Choose advanced options> replica and partition operations> repair time stamps and declare a new epoch

73. Don’t Filter Out NTP port 123 Your firewall must permit UDP ports 123 and 524 for NTP and timesync packets out; also all UDP in (unless firewall supports dynamic filtering) for timesync packets and port 123 in for NTP across the firewall. **Exercise #12 (30 mins)**

74. Server 54, Where Are You? -625 error –transport failure Check physical connections Use ping / ipxping Check router logs (dropped pkts) Don’t span a slow WAN Check DS.nlm loaded on each server Don’t filter ports- SAP 278, 26b; all RIP; TCP/UDP 524;UDP 123; TCP/UDP 427; TCP 2302;UDP2645 They are used by- NTP, SAP, SLP, RIP,CMD, NCP Server IDs and public key can become corrupted – 632 error:system failure-add / repair replica or use DSRepair –XK3 to re-backlink ext references. ** *Exercise #13 (30 minutes)***

75. Who are you and how did you get here? All objects have several mandatory attributes or properties. If a mandatory attribute of an object becomes corrupt, has no value, or cannot be located, the object becomes ‘unknown’. When product add-ons or upgrades are installed, the ‘schema’ of NDS may be extended and ‘viewing snap- ins’ added to allow a wider range of objects & properties.

76. How to get rid of unknown objects Be sure NetWare Administrator has the required.dll’s (snap- ins) so it can ‘see’ the objects. If you are sure the unknown objects no longer exist (ex.- volume objects from a deleted server) simply delete them. If a leaf object can be easily re-created there is also no harm in deleting it. Using NDSManager or DSRepair, run the ‘Receive update’ option on the corrupted replica, or ‘Send updates’ from an uncorrupted replica. Run DSRepair –P which will flag all unknown objects as new objects. Their attributes will then be automatically updated the next time the replica ring synchronizes.

77. How to get rid of renamed objects If replica A1 can’t communicate with replica A2 for a period of time, it is possible that 2 different objects could accidentally receive the same name. When communication is restored, NDS will see the clash and rename one object with a no._no. designation (ex.- 1_2) Solution- Be sure all replicas are now ‘ON’ and communicating then either delete or rename one of the objects. (***Exercise 14 – 30 mins.***)

78. Schema mis-matches The schema is the layout of NDS, what objects and properties are allowed and can be understood. Every time you upgrade NetWare, or add a program like ZENworks, the schema is extended. All servers in the tree hold a copy of the ‘schema’ pattern and if the schema is extended this update is passed to all servers. This process occurs 15 seconds after initialization and then every 240 minutes after. For manual schema updates, DSRepair –A then Advanced Options>>Global schema operations. (***Exercise #15 –30 mins ***)

79. Dead but not gone (obituaries) When an object is- –Deleted –Renamed –Moved The actual purging of the object only occurs after- –Master replica says ‘Dead!’ (Flag 0000) –All servers notified (Flag 0001) –Servers report OK to purge (Flag 0002) –Entry actually purgeable (Flag 0004)

80. Dead but not gone (obituaries) There are various types of obituaries based on the type of operation that was performed on the object(s). If obituaries get ‘stalled’ you may need to clean them up. How? >>

81. Dead but not gone (obituaries) If an obituary is not cleanly removed, determine which server has not acknowledged the process by running dsrepair –A and looking at the log reports. –Find the object that was deleted but not OK’d to purge –Find where it was moved from –Find which server has not responded to the obituary process –Clear up the problem with that server. Options to clear up the problem- –Restore the offending server’s communications with network –If server no longer exists, delete using NDSManager –Run DSRepair with the –XK3 switch on the offending server

82. The XK3 switch Resets all external references on a server to cause the properties to be refreshed by the backlink process. In some cases will clear up stuck obituaries by allowing the process to go beyond Flag 0002. Note: If an offending obituary does not respond to these steps, contact Novell Technical Support for further assistance to solve the problem. ***Exercise #16 (30 min.)***

83. Practice Exercises As time permits- –Exercises 17 to 26

84. End of Day Three OK all you NDS Experts! You may leave when ready! Thank-you for your contribution! - course designed for: Dynamic Mutual Funds, 40 King St. W. Toronto