Linux Technology Center 18 April 2003 © 2003 IBM LDAP Content Synchronization Kurt D. ZeilengaJong Hyuk Choi OpenLDAP ProjectIBM Research Title slide

LDAP Content Synchronization2 Agenda  What is a directory? What is X.500? What is LDAP? What is the OpenLDAP Project?  LDAP Content Synchronization  Questions and Answers Agenda slide

LDAP Content Synchronization3 What is a Directory?  A database organized and/or optimized for lookup, searching, browsing, and other “read” activities.  Examples: address books web catalogs whois(1) dict(1) operating system / network information services Domain Name System

LDAP Content Synchronization4 What is X.500?  The set of ITU standards covering electronic directory services.  X.500 defines the OSI Directory Service Hierarchical Distributed Replicated Schema driven

LDAP Content Synchronization5 What is LDAP?  Lightweight Directory Access Protocol an IETF Proposed Standard  A protocol for accessing X.500 directory services over the Internet (or networks using Internet Protocols).

LDAP Content Synchronization6 What is OpenLDAP?  OpenLDAP Software “community developed LDAP software”  OpenLDAP Project “a collaborative effort to develop a robust, commercial-grade, fully featured, and open source LDAP suite of applications and development tools”  OpenLDAP Foundation Provides a legal umbrella for the OpenLDAP Project 

LDAP Content Synchronization7 LDAP Overview Closing slide Request Response The Directory User

LDAP Content Synchronization8 Polling for content changes Closing slide Request The Directory User Response X

LDAP Content Synchronization9 Problems with simple “polling”  Cannot reliably detect whether the entry named by a DN is the same entry previously named by that DN. DNs are not stable identifiers!  A complete copy of each entry is transferred with each poll  Changes not apparent until next poll

LDAP Content Synchronization10 Stable Identifiers  Add an Universally Unique Identifier (UUID) to each entry. Introduce entryUUID operational attribute  Add knowledge of superior’s UUID for each entry. Introduce parentUUID operational attribute

LDAP Content Synchronization11 Reducing traffic  Each entry in LDAP has create and modify time stamps (&(FILTER)(|(createTimeStamp>=TIME)(modifyTimeStamp>=TIME)) where FILTER is our original assertion and TIME is the greatest seen in previous content.  All entries created/modified before TIME are not transferred.  Does not detect deleted (or modified ‘out of scope’) entries.  Does not detect unchanged entries now ‘in scope’  Search operations are not necessarily isolated from directory modifications. One cannot assume all updates made before TIME appeared in the previous content.

LDAP Content Synchronization12 We need to extend LDAP to support a lightweight content synchronization mechanism!

LDAP Content Synchronization13 Target Applications  Heterogeneous Data Store Synchronization “meta directory” applications  Local content shadowing Directory-enabled services (web, , etc.) Mobile clients  LDAP Replication Single master Partial (Sparse and/or Fractional) Replication

LDAP Content Synchronization14 Design Goals  Less chatty than “simple polling”  Maintain consistency Eventually convergent (inconsistencies are transient)  Support “polling” for changes  Support “listening” for changes  Support partial synchronization  Do not require pre-arranged synchronization agreements  Do not require server to maintain per client state information  Do not require server to maintain histories of changes  Lightweight / Simple

LDAP Content Synchronization15 Non-goals  Support bi-directional synchronization  Support very small clients (e.g., PDAs)  Attribute (or finer) granularity

LDAP Content Synchronization16 Synchronization Overview Old New Changed Content Renamed Content Added Content Deleted Content Unchanged Content

LDAP Content Synchronization17 Basic “Refresh Mode” Protocol Design Send Entries Initial Content Request Done with Cookie Send “Changed” Entries Content Refresh Request Done with Cookie Repeat Send “Present” Messages

LDAP Content Synchronization18 First things first…  Stable object identifiers LDAP Distinguished Names are not stable Solution: add entryUUID and parentUUID attributes to every entry  Time stamps LDAP create/modify time stamps are generally good enough, but semantics are unclear Solution: add change sequence numbers (CSNs) to every entry, well define the semantics

LDAP Content Synchronization19 The Cookie  Synchronization State Indicator  Greatest committed change sequence number (CSN) in context which is not greater than any outstanding CSN e.g., Change Sequence Numbers != Commit Sequence Numbers

LDAP Content Synchronization20 Bandwidth

LDAP Content Synchronization21 Refresh: updates+present v. updates+deletes  If server has no histories, it cannot determine what to “delete”  If server maintains “tombstone” information for deleted entries but no content change history, it must send deletes for all new tombstones as well as all out-of-scope updated entries.  If the server can reliably determine which entries previously present in the shadow copy which are no longer present in the content and the number of such entries is less than equal the number of unchanged entries, the server may send “deletes” instead of “present” messages.

LDAP Content Synchronization22 OpenLDAP Implementation  No naming/change history  Each entry has: entryUUID/parentUUID entryCSN - updated on entry modify namingCSN - updated on entry (not subtree) rename  Context has: Outstanding CSNs - ordered list Outstanding Naming CSNs – ordered list Allows server to detect some “no deletes” cases.

LDAP Content Synchronization23 “Refresh & Persist Mode” Protocol Design Send Entries Content Request Refresh Done Send “Changed” Entries and/or “Deleted” messages Cancel Done with Cookie / Acknowledge Cancel New “Cookie” Repeat

LDAP Content Synchronization24 Special Cases  Operational Attributes Must either be treated as part of the entry’s content if transferred. structuralObjectClass hasSubordinates subschemaSubentry  Alias and Referral objects No dereferencing Treat as normal object  Collective Attributes Not treated as content of entries of the collection Client must ‘watch’ the collective attributes subentries  Access and other administrative controls Server must either generate appropriate changes to sync client OR force reload

LDAP Content Synchronization25 Dealing with subtree renaming…  Case 0: subtree stays out of scope No action  Case 1: subtree enters scope Send ‘update’ messages for each entry  Case 2: subtree leaves scope Refresh: U+P: don’t send ‘present’ messages for entries U+D: send ‘deletes’ for entries OR force reload Persist: send ‘delete’ messages for each entry OR force reload  Case 3: subtree remains in scope Send ONLY entry at base of subtree.

LDAP Content Synchronization26 Summary  The LDAP Content Synchronization Operation appears to met our needs: Supports both “Polling” and “Listening” modes “Polling” mode is less chatty than “simple polling” “Listening” mode is significantly less chatty than “simple polling” Eventual convergence is assured, inconsistencies are transient. No pre-arranged synchronization agreements are needed. Server need not maintain histories or per-client state information. Able to support our target applications: Meta Directory Content caching LDAP Replication

LDAP Content Synchronization27 Availability  IETF Technical Specification: draft-zeilenga-ldup-sync-xx.txt draft-zeilenga-ldup-uuid-csn-xx.txt (available soon) Last Call this summer (hopefully)  OpenLDAP Implementation: LDAP-sync client and server implementations committed to HEAD First-cut LDAP Replication engine committed to HEAD Release target: OpenLDAP 2.2 in Summer of 2003

LDAP Content Synchronization28 Questions? Closing slide