LDAP/TIO implementations -2- Overview of TIO-index implementations Henny Bekker The DAG, GIDS and Desire TIO/LDAP index servers

LDAP/TIO implementations -3- Agenda Overview of TIO-index implementations What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla General overview of LDAP/TIO-indexes –What are TIO indexes –The generic model Some specific implementations –The generic Desire TIO index server –The Ericsson DAG server –The GIDS server Open Issues –The scope and communication between LDAP/TIO index servers Exchanging TIO’s –Local access policy Access restrictions Security requirements Senario’s

LDAP/TIO implementations -4- Tagged Index Object’s General overview of LDAP/TIO-indexes What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla A TIO consists of: Meta information such as –A mime header defining the object –An object type identifier that uniquely identifies the subtree and scope –One or more URI's that will form the base of the created referrals –The security options and credentials such as a PGP or S/MIME key –The update type indicating the type of TIO (e.g. full or incremental) The payload –The tokenization types headers (e.g. Full, Token, RFC822 etc) Indicating which information is ‘tokenizated’ and which delimiters to use –The TAG list Containing multiple consecutive tags which might be grouped using a dash.

LDAP/TIO implementations -5- General overview of LDAP/TIO-indexes What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Content-Type: application/index.obj.tagged; dsi=" "; base-uri= "weetmuts.surfnet.nl:389/o=SURFnet, c=NL" Content-Length:6219 version: x-tagged-index-1 updatetype: total thisupdate: BEGIN IO-Schema sn: FULL cn: FULL. o: TOKEN END IO-Schema BEGIN Index-Info sn: 22/Arends -6/Bezemer -4/Bos -8/Neggers. -2-3,5-9,11,14-15,18-19/ / o: 1/SURFnet END Index-Info

LDAP/TIO implementations -6- Tagged Index Object’s (cont.) General overview of LDAP/TIO-indexes What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla What is it used for: Provide pointers to servers which most likely contains the requested information –The number of false hits is depending on the choice of attribute tokenization types –Performing phrase searches is depending on the tokenization of the fields Features a full or incremental update (which uses potentially less bandwidth)

LDAP/TIO implementations -7- The generic model General overview of LDAP/TIO-indexes What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla A TIO interface –For importing, deleting and in some cases exporting TIO’s from the index –Implementing authentication control A TIO searchable index –For searching the index on referrals to other information services –Accessible through the TIO query interface The LDAP query interface –Containing a LDAP gateway to the query interface of the TIO index –Can act as an LDAPv2 chaining server or as an LDAPv3 referral server

LDAP/TIO implementations -8- The generic Desire TIO index server Some specific implementations What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Sponsored by the European Community and build by SURFnet & DFN in cooperation with Dante. The server consists of: –The TIO index server Using the MySQL database engine for storing and searching the TIO’s Containing a TIO push/pull interface and a database for storing TIO’s. An HTTP frontend for direct access to the TIO index server by the NPS. –A Native Protocol Server (NPS) for access using the LDAP protocol For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. For connecting directory servers with a specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. –An LdapCrawler for gathering and converting LDIF files to TIO’s Currently no encryption of TIO’s implemented Currently only support for LDAPv2. (no characterset conversion problem)

LDAP/TIO implementations -9- The generic Desire TIO index server (cont.) Some specific implementations

LDAP/TIO implementations -10- The Desire LDAP/TIO index server (cont.) Some specific implementations

LDAP/TIO implementations -11- The Desire LDAP/TIO index server (cont.) Some specific implementations What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Unfortunately we don’t have yet any performance figures  –The package is on the brink of being completed –Presumably the GIDS index server will be faster The generic MySQL engine is slow compared to a dedicated TIO database. Current implementation –Available on Linux and (hopefully) on Digital Unix –The source code and executable for Linux of The LdapCrawler with an integrated LDIF2TIO converter The TIO index (using MySQL v3.23.6) The LDAP NPS implemented using the Open-LDAP v with an API to the TIO index

LDAP/TIO implementations -12- The Ericsson DAG server Some specific implementations What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Offspring of the TISDAG project –Aimed to provide a solution for an uniform telephone directory containing numbers without a centralized database The server consists of: –The DAG (Directory Access Gateway) index server Implemented using the TimesTen “In-Memory” database engine for storing and searching the TIO’s. –One or more CAP (Client Access Point) modules For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. –One or more SAP (Server Access Point) modules For connecting directory servers with a specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++.

LDAP/TIO implementations -13- The Ericsson DAG server (cont.) Some specific implementations

LDAP/TIO implementations -14- The Ericsson DAG server (cont.) Some specific implementations What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Unfortunately we aren’t allowed to present exact figures  –The next version is said to be much faster Performance figures –Response times Use LDAPv3 referral requests to measure the response time of the referral server without doing chaining or following referrals. The mean response time related to the number of parallel search queries. (measured with a large number of queries) –Number of queries/second (or minute??) –The number of parallel requests Related to the response time –Maximum number of entries in the TIO index Bounded by the memory size and the algorithm used to search the index

LDAP/TIO implementations -15- The GIDS server Some specific implementations What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Offspring of the TISDAG project –Second implementation of the TISDAG TIO index server The server consists of: –An index server Using a dedicated database engine for storing and searching the TIO’s. Is using a dedicated communication protocol (analogous with LDAP) to communicate with the CAP and SAP modules. –One or more CAP (Client Access Point) modules For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 and HTTP –One or more SAP (Server Access Point) modules For connecting directory servers with a specific communication protocols such as LDAPv2 and LDAPv3 –An LdapCrawler for gathering and converting LDIF files to TIO’s With support for LDAPv2 and LDAPv3 and character-set conversion

LDAP/TIO implementations -16- The GIDS server (cont.) Some specific implementations What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Performance figures –Response times Measured with LDAPv3 requests (an LDAPv3 bind, sending the query, receiving the message, doing an unbind operation) With one sequence of LDAPv3 requests, a mean time of approximately 23 msec. per LDAPv3 request. With 10 simultaneously LDAPv3 requests approximately 150 msec per LDAPv3 request. –Maximum number of queries/second Approximately 65 LDAPv3 queries/second With LDAPv2 the number will be lower because the server has to do chaining. –Maximum number of entries in the TIO index Bounded by the memory size and the algorithm used to search the index Current demo implementation (CH, DE, NL, NO & SE) 120K tokens of 450 different data sets. (which consumes about 35-Mbyte of memory).

LDAP/TIO implementations -17- The scope and communication between TIO index servers Open issues What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Scope –Centralized versus distributed LDAP/TIO engines Location of the TIO/LDAP-index server –Located close by (in network terms) the end users to minimize the RTT –Located close by the referred LDAP servers to minimize the RTT related to LDAPv2 chaining Exchanging TIO’s –Global TIO collection versus distributed collections on country level Distributed to country level Knowledge base or ‘where to find what’?? –Encrypted transport via HTTP –Push or pull ??

LDAP/TIO implementations -18- Local access policy Open issues What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla Security requirements –Personal data is subjected to privacy legislation –For public data other security requirements might be imposed –No unauthorized access to local directory servers Only accessible by local inhabitants and peer countries –All applications able to access the index should be known –Only a limited number of referrals might be returned –No ‘access denied’ messages Don’t show entries which are not accessible Access restrictions –Restrict access to the TIO/LDAP-index server –Restrict access to the LDAP servers containing the information Chaining versus LDAPv3 referral HTTP access control versus LDAP access control Access via HTTP proxies versus LDAP proxies

LDAP/TIO implementations -19- Senario’s Open issues Create trusted relation between country level TIO servers –Only peers will communicate with each other Besides the local LDAP clients –A peer will enforce their own local access rules The TIO index server should only be accessible by known clients –The LDAP query will be chained to the remote peers The TIO objects of the peer country should deliver referrals which will point to a known access point e.g. an LDAP proxy or the FLDSA An LDAP search requests from a known LDAP client must be chained to the known access point. –The number of known access points should be limited –The TIO objects cannot be duplicated between the peers

LDAP/TIO implementations -20- Senario’s (cont.) Open issues

LDAP/TIO implementations -21- Discussion..