DIFFSYNC A file synchronization program CLASS PROJECT – EEL6883 BY ADAM RADWAN GARGI CHIPALKATTI

FILE SYNCHRONIZATION Process of making sure that two or more locations contain the same up-to-date files One-way (Mirroring) or two-way file synchronization Useful for mobile workers, multiple computer users; used for backup purposes Version control tools and File synchronization tools Various existing solutions, none of which were found by OpenPublic Networks, LLC to be robust enough for production use in a distributed environment

GOALS Develop a file synchronization program which is capable of working over a diverse range of geographical placement Must be capable of running securely over common public network backbone, such as the Internet Capable of two-way synchronization of files for changes to served web pages Ability for a non-functional server to gracefully degrade while other servers detect this and recover appropriately to keep services running

RSYNC Open Source software for Unix systems Mirroring takes place with only one transmission in each direction Used to synchronize local directories or directories via a remote shell such as RSH or SSH Two checksums: a weak rolling 32-bit checksum and a strong 128-bit MD4 checksum

UNISON Runs on Unix-like operating systems (Linux, Mac OS X, and Solaris), and on Windows Can synchronize files across platforms Detects 'conflicts' where a file has been modified on both sources, and displays these to the user Uses the rsync algorithm Designed to be robust in the event of a program or system crash or a communication failure

DIFFSYNC Uses open source software – rsync, inotify Uses PHP scripting language Built on top of a MySQL 5.1 triple-master replicated database system running in chrooted jail for security Custom SSH-based VPN for three independent point- to-point TUN links between the servers Apache-based web server for verification running in chrooted jail for security phpMyAdmin used for database management

DIFFSYNC FOR THREE SERVERS Akimbo Location: Orlando, FL IP: CPU: Pentium III 800 Mhz Storage: 20GB, 128 MB OS: Gentoo Linux Philomena Location: Ft. Lauderdale, FL IP: CPU: AMD Storage: 40GB, 256 MB OS: Gentoo Linux Podice Location: Fayetteville, NC IP: CPU: Pentium Ghz Storage: 80GB, 512 MB OS: Gentoo Linux

CREATION OF DIRECTORY & FILE ~/TestArea $ mkdir 1 Array ( [0] => Array ( [dir] =>./TestArea/ [event] => CREATE,ISDIR [file] => 1 ) [1] => Array ( [dir] =>./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [2] => Array ( [dir] =>./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 ) ~/TestArea $ nano./1/X Array ( [0] => Array ( [dir] =>./TestArea/1/ [event] => CREATE [file] => X ) [1] => Array ( [dir] =>./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] =>./TestArea/1/ [event] => MODIFY [file] => X ) [3] => Array ( [dir] =>./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X )

MOVE DIRECTORY & FILE ~/TestArea $ mv./1/X./1/Y Array ( [0] => Array ( [dir] =>./TestArea/1/ [event] => MOVED_FROM [file] => X ) [1] => Array ( [dir] =>./TestArea/1/ [event] => MOVED_TO [file] => Y ) ) ~/TestArea $ mv./1./2 Array ( [0] => Array ( [dir] =>./TestArea/ [event] => MOVED_FROM [file] => 1 ) [1] => Array ( [dir] =>./TestArea/ [event] => MOVED_TO [file] => 2 ) )

ATTRIB DIRECTORY & FILE ~/TestArea $ chmod 777./1 Array ( [0] => Array ( [dir] =>./TestArea/ [event] => ATTRIB [file] => 1 ) ~/TestArea $ chmod 777./1/X Array ( [0] => Array ( [dir] =>./TestArea/1/ [event] => ATTRIB [file] => X )

DELETE DIRECTORY [4] => Array ( [dir] =>./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => ) [5] => Array ( [dir] =>./TestArea/ [event] => DELETE,ISDIR [file] => 1 ) [6] => Array ( [dir] =>./TestArea/1/ [event] => DELETE_SELF [file] => ) [7] => Array ( [dir] =>./TestArea/1/ [event] => IGNORED [file] => ) ~/TestArea $ rm –r./1 Array ( [0] => Array ( [dir] =>./TestArea/ [event] => OPEN,ISDIR [file] => 1 ) [1] => Array ( [dir] =>./TestArea/1/ [event] => OPEN,ISDIR [file] => ) [2] => Array ( [dir] =>./TestArea/1/ [event] => DELETE [file] => X ) [3] => Array ( [dir] =>./TestArea/ [event] => CLOSE_NOWRITE,CLOSE,ISDIR [file] => 1 )

DELETE FILE ~/TestArea $ rm./1/X Array ( [0] => Array ( [dir] =>./TestArea/1/ [event] => DELETE [file] => X ) )

EDIT FILE ~/TestArea $ nano./1/X Array ( [0] => Array ( [dir] =>./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] =>./TestArea/1/ [event] => ACCESS [file] => X ) [2] => Array ( [dir] =>./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X ) Array ( [0] => Array ( [dir] =>./TestArea/1/ [event] => MODIFY [file] => X ) [1] => Array ( [dir] =>./TestArea/1/ [event] => OPEN [file] => X ) [2] => Array ( [dir] =>./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => X )

COPY FILE ~/TestArea $ cp./1/X./1/Y Array ( [0] => Array ( [dir] =>./TestArea/1/ [event] => OPEN [file] => X ) [1] => Array ( [dir] =>./TestArea/1/ [event] => CREATE [file] => Y ) [2] => Array ( [dir] =>./TestArea/1/ [event] => OPEN [file] => Y ) Array ( [3] => Array ( [dir] =>./TestArea/1/ [event] => ACCESS [file] => X ) [4] => Array ( [dir] =>./TestArea/1/ [event] => MODIFY [file] => Y ) [5] => Array ( [dir] =>./TestArea/1/ [event] => CLOSE_WRITE,CLOSE [file] => Y ) [6] => Array ( [dir] =>./TestArea/1/ [event] => CLOSE_NOWRITE,CLOSE [file] => X )

DIFFICULTIES ENCOUNTERED Must have a kernel module installed on the host server for inotify events to be recorded Problems detecting changes made to newly-created directories within the specified inotify watch point MySQL replication is sensitive to binary log changes when one node is disconnected Accurate prediction and parsing of inotify events for complete analysis of changes made to filesystem Not enough time in the semester to complete integration of all features into program

CONCLUSION Fundamentals of DiffSync program prove that file synchronization is possible for servers in remote locations Additional work will be completed to make this program ready for a production environment Any questions?

THE END THANK YOU!