1. April 1999HEPiX991 PostKEK A new mail system using DCE/DFS Akihiro Shibata Akihiro.Shibata@kek.jp Computing Research Center, High Energy Accelerator Research Organization (KEK)

2. April 1999HEPiX992 Contents _ PostKEK system –Requirements –System Design –Status –Summary and Discussion _ High Availability File service using DFS

3. April 1999HEPiX993 System requirements _ More than 1,000 users _ Non-stopping service through year. _ Security _ services –POP (IMAP in the future) –Mail exchanger (out-going mail gateway) –Remote login –Home directory –Mailing-List

4. April 1999HEPiX994 System Design _ Design based on distributing system using DCE (Distributed Computing Environment) DFS (Distributed Files System) _ High availability –Duplication of servers (SMTP, POP, telnetd,...) –Higher availability file service by DFS –Application fail-over (sendmail: mail spooling)

5. April 1999HEPiX995 System components _ 4-work stations –HITACHI 3500 (160MB memory) –OS : HI-UX/WE2 _ RAID disk –HITACHI A-6531 2 port controller Duplicated electric supply units 32Gbyte (2-arrays) –For spools and home directories (file service by DFS) _ DCE –HI-DCE Executive (OSF DCE ver 1.1 base)

6. PostKEK

7. April 1999HEPiX997 Why DCE/DFS? _ security –Integrated login No password encrypted data is shared in the DCE client Long password is possible (<128) –Access Control Lists (ACLs) more flexible access control than UNIX –Even root of DCE/DFS clients has no privilege for cell administrations –No plain password is sent among DCE cell _ availability –DCE server replication –Load balancing among replica of DFS

8. April 1999HEPiX998 Why DCE/DFS ? (2) _ Scalability –The resources are distributed and shared among many hosts. –Up-to hundreds hosts. _ Multi platforms are supported –AIX, Solaris, HPUX, Digital UNIX, Irix _ Uniform file access _ DFS Backup (local file system) –Snapshot of the home directory is held –The files and directories are able to be recovered to those of time when snapshot was taken.

9. April 1999HEPiX999 DCE/DFS servers Mlserva dced (M) cdsd (M) secd(M) fl server fxd (M) file exporter (prim) Mlservb dced (S) cdsd (S) secd(S) fl server fxd (S) file exporter (backup) mail1 fl server mail2

10. April 1999HEPiX9910 Mail servers Mlserva DNS (M) SMTP (M) Mlservb DNS (S) SMTP (S) mail1 SMTP : gateway POP3mail2 SMTP : gateway POP3 MX: post.kek.jp mail1, mail2 spooling

11. April 1999HEPiX9911 Modification from UNIX system _ Authentication –To access DFS user should be authenticated for DCE –Integrated login with DCE and UNIX about 20-30 lines modification for each source –POP server, –ftp server, –login _ qpopper (POP server) –Adding the file lock function to the source code –DCE login

12. April 1999HEPiX9912 High Availability services _ DCE servers replication (security server, CDS server) _ High Available file service by DFS –Home directories and mail spools will be explained in detail later _ Spooling mails –Swapping the SMTP servers. (application fail over) –Synchronized with swapping DFS server _ POP server, telnet server, SMTP server –More than 2 servers

13. High available service: server fail over Malservb DFS : stand-by SMTP: stand-byMalserva DFS: file exporter SMTP(M) : spool Malservb DFS : file exporter SMTP(M) : spoolMalserva DFS: SMTP: Both DFS and SMTP servers are swapped

14. April 1999HEPiX9914 Status of service _ Starts at May 5th 1998 _ Users 360 users at April 1st 1999 _ Mailing Lists –since December 1998 –20 lists _ POP(2 servers) –10000 accesses per day –500 accesses per hour (peek) _ login (2 servers) –200 accesses per day –30 users at the same time –60 users per day _ Mails – 2000 mails per day

15. April 1999HEPiX9915 Status of system running _ Swapping server –It tooks at most 15 minutes for server swapping 32 Gbyte disk (15 partitions) about 800 filesets (RW, Backup) –and about a further few minutes for propagating to clients depending on cache parameter of DFS and so on. _ running status –clients: 200 days without stopping –servers: 100 days without stopping

16. April 1999HEPiX9916 Summary and Discussion _ Stability and continuous services are required. _ Mail system using DCE answers the requirements –security –The high available system –The file sharing in secure –Possible to add a lots of clients _ High Availability File Service using DFS –Continuous service is possible even in maintenance. –Manipulated by hand. –It is helpful, if “hot” (automatic) fail-over is possible.

17. April 1999HEPiX9917 Discussion(2) _ IMAP –The later Supporting on Japanese Language. Almost 3 clients are available in the beginning of 1998 –Netscape ver 4.0, Airmail, pine Now, several mailers support on Japanese. –IMAP will be supported in near future.

18. April 1999HEPiX9918 Higher Availability (HA) File System with DFS

19. April 1999HEPiX9919 DFS fail over limitation _ Read Only replica –One Read/Write(R/W) server and many Read Only(RO) servers –Load sharing among RO servers –Very useful for application service or read most file service such as web pages, but useless for home directory service –Very useful for application service or read most file service such as web pages, but useless for home directory service. –In case of R/W server fail, one of backup RO replica could be a new RW server. But data consistency between old and new R/W server is not assured. _ R/W replica functionality is not implemented in DFS yet.

20. April 1999HEPiX9920 HA File Service _ Make down time of DFS file server shorter in trouble or in maintenance –Total fault tolerance or dynamic fail over –Prevent program failure or file damage in unexpected system down. _ HA products using 2port disks exist on NFS, but not on DFS.

21. April 1999HEPiX9921 HA file service - How _ Adopt two ports RAID –Multi port RAID is popular recently. –No data copy is needed between active and stand-by servers. (no data consistency or synchronization problem) _ Relation between Fx server and served Fileset. –Seems to be defined by ‘position’ of Serverentry in FLDB. –To change FileServer assignment for the fileset, it need just replacement of the Servername in the entry

22. April 1999HEPiX9922 DFS file access mechanism _ Two DFS server provides. –File server To store and export LFS and on-LFS data as filesets. –File location server Stores FLDB (fileset location database) which contain the data of location of fileset. FLDB contains information about fileset (name, ID number., physical location). _ FILESET –Sub-tree related files and directories. –The mount point the place where fileset is attached to DFS global filesystem the name of fileset in which data resides, ( not physical location)

23. DFS file access mechanism FLDB DFS client CDS server 2. Where is root.dfs? 3. Acsess to /:/ 4. Interpreting path user 5. Where is shibata.home? 6. Access to /:/usr/shibata ls /:/user/shibata root.dfs mounted to /:/ #mount point shibata.home mounted to /:/user/shibata DFS server 1. Where is FLDB ?

24. Commands Server A side (Original File Server)  become cell_admin # dfsexport -agg -detach - force Server B (New File Server) become cell_admin fts eds -ch # fts eds -ch # fts eds -prin hosts/ # dfsexport -agg hostA hostB fileset 1 agg1_A agg1_B

25. Server Entries server 1 server 2 sever 3 File set Entry for fileset 1 for fileset 2 for fileset 3 FLDB entry Server Entries server 2 server 1 sever 3 File set Entry for fileset 1 for fileset 2 for fileset 3

26. Usual Status DFS server B 2 port DISK DFS server Astand-by FLDB server user.shibata -> A Clinets ls ~shibata unexport Network

27. 2 port DISK DFS server A down DFS server B FLDB server user.shibata -> A Clinets ls ~shibata unexport user.shibata ->B Network Server A maintenance

28. April 1999HEPiX9928 HA - result (1) _ Works well. –Better than NFS HA. In usual NFS HA, IP swap trick is used. Caching of ARP table (IP-MAC) in clients, bridges or routers sometimes makes problem. _ Without ‘dfsexport -detach’ when the servers are swapped –File inconsistency between both servers were happened. –File damages may happen in server trouble.

29. April 1999HEPiX9929 HA - discussion _ This mechanism cannot offer load valance among servers but, useful for high availability –moderately useful in the present status. –Very helpful if dynamic fail over becomes possible. (even if it cannot prevent application abort or damages of file being opened at the time of the server down.) –Dynamic fail over is our dream. (Then we will not be called in early morning for a server down)