Outline Subversion server Sandpit cluster What is it? Why is it useful? How to use Sandpit cluster 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Subversion: What is it? Version control system Successor to CVS Subversion server https://svn.win.tue.nl/repos/san Subversion client 1 C:\My Documents\ Subversion client 2 D:\My Documents\ education courses 2IC10 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www courses 2IC10 2IF20 … education courses 2IC10 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Subversion: Why is it useful? Sharing knowledge Now documents, code, etc. are only available on pc of individual group members These artifacts should be access-able for all Collaboration Writing documents together No need for mailing documents to each other over and over again Backup 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Subversion: How to use? Important actions: Check-out Update Add Remove Commit 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Check-out Create a local working copy of a sub-tree of the repository Subversion server https://svn.win.tue.nl/repos/san Subversion client C:\My Documents\ education courses 2IC10 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www courses 2IC10 2IF20 … 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Update Synchronize updates on the repository with your local copy. Subversion server https://svn.win.tue.nl/repos/san Subversion client C:\My Documents\ education courses 2IC10’ 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www courses 2IC10 ‘ 2IF20 … 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Add Put a (new) file under version control !! File will be added Subversion server https://svn.win.tue.nl/repos/san Subversion client C:\My Documents\ education courses 2IC10 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www courses 2IC10 New file 2IF20 … !! File will be added to repository during next commit. 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Remove Remove a file from version control (The history will remain !!). Subversion server https://svn.win.tue.nl/repos/san Subversion client C:\My Documents\ education courses 2IC10 file 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www courses 2IC10 file 2IF20 … !! File will be removed to repository during next commit. 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Move Move a file from one location in the repository to a different one. Subversion server https://svn.win.tue.nl/repos/san Subversion client C:\My Documents\ education courses 2IC10 file 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www courses 2IC10 file 2IF20 … !! File will be moved to repository during next commit. 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Commit Changes in your local working copy are synchronized with the repository Subversion server https://svn.win.tue.nl/repos/san Subversion client C:\My Documents\ education courses 2IC10 New file 2IF20 File … student_projects research personal jmuskens mchaudron projects robocop space4u www courses 2IC10 New file 2IF20 File … 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Our Repository https://svn.win.tue.nl/repos/san/ education courses 2IC10 2IF20 … student_projects research personal jmuskens mchaudron projects robocop space4u www 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Tools Windows client Unix client Tortoise SVNClient Integrated in Windows Explorer Unix client Command line (svn add/remove/…) Graphical client ? 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Things to remember Changes are not effectuated at the server until you do a commit You can only add, remove, … things when you have a local copy of a sub-tree of the repository You can have multiple local copies of possibly different sub-trees of the repository You can put files in a local working copy without putting them under version control History remains Once you add something to subversion, traces remain (even when you do a remove). 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Guidelines Make it available for all Also old project documentation Add sources, not the derived artefacts Don’t use versions in the document names Subversion does the versioning for you ! Don’t alter the existing directory structure How you structure your personal, course or project sub-tree is up to you Keep it clean 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

SANdpit SAN distributed & parallel integrated terminal to dig up information Richard Verhoeven 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Content Hardware Software Usage General, OpenPBS, LAM/MPI Courses Projects Users Data access 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Old cluster One master with 17 nodes (pacluster) 533 MHz Pentium III, 256 MB, 10 GB, 100 Mbit RedHat 7.3 with security fixes Failing power supplies, fans, disks, video cards Only 13 nodes kept running to some extend 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

New cluster One master with 17 nodes (sandpit) 3 GHz Pentium 4 with hyper-threading 400 GB disk in master, 40 GB disk per node 1 GB memory 2 Gigabit NICs 19” rack mounted 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Hardware 2x200 GB Internet 40 GB 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Software Gentoo Linux distribution Compile from source Optimized for platform Large collection of standard software Text editors (vi, emacs, nano) Compilers (gcc, g++, g77, gcj, icc, javac) Text processing (perl, LaTeX, ps2???) Graphic processing (netpbm, gstreamer) GUI apps (gnome, mozilla) More software is installed upon request 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

890 GB shared disk space for users Directory structure /home/user (190 GB) /local/user (190 GB) /wslocal/sb01/user /home/user /wslocal/sandman/user /local/user (30 GB) /wslocal/sb01/user sb01 .. sb17 sandpit / sandman 890 GB shared disk space for users 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

OpenPBS Portable Batch System Four queues for jobs Maui scheduler Jobs are submitted to the server with qsub Status of queue is available through qstat Scheduler allocates nodes to specific jobs. Four queues for jobs small (< 20 min, highest priority) medium (< 2 hours) large (< 12 hours) verylarge (< 3 days) Maui scheduler 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

LAM/MPI MPI – message passing interface LAM/MPI – open-source MPI implementation lamboot – create a multicomputer mpirun – run parallel program lamhalt – terminate multicomputer Under PBS Create script with lamboot – mpirun – lamhalt Create multicomputer from allocated nodes 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Courses Student accounts Course specific software Special PBS queue Only for students and instructors Only active queue during instruction hours Cluster remains accessible for others Getting started manual 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Projects Group of users Project related directory Guest accounts Software repository Documentation Guest accounts Faculty, university, national, international 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Users SAN members Faculty members Students of courses Copied from UNIX systems Faculty members mCRL group Students of courses Guest researchers Use ssh to login and rsh/ssh to move between nodes. 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Data Access Central directories are not accessible Use ssh and scp UNIX directories not NFS mountable (anymore) Window G: disk not SAMBA mountable (yet) Use ssh and scp 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Data Backup Cluster not under the backup policy of BCF Options: Data is often temporary Too much data for offsite backups Options: Copy data to different nodes Copy data regularly to central network disk Integrate backup into PBS scripts Check data into Subversion 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Usage policy No enforcement (yet) Fair use Otherwise By-passing PBS is possible No disk quota No accounting of processor time Fair use Clean up temporary data regularly Don’t abuse the resources Personal projects are allowed as long as they don’t interfere with education or research. Otherwise Enforcement will be added or account disabled 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Possible extension Use OpenMosix Add real-time boot configuration Extension to move processes between nodes Add real-time boot configuration Reboot nodes when job scheduler asks for it Use second network interface Small ring Second connection to the switch Add cross-compiler tool chains Familiar Linux for iPAQ TinyOS for sensor motes SymbianOS for mobile phones 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Moet weg tegen elk aannemelijk bod What happened with … the old cluster? Already reused in other projects Several nodes revisited to get them working Create heterogeneous network setup Ethernet, wifi, firewire, bluetooth, sensors Combine or upgrade test environments Give to family and friends Moet weg tegen elk aannemelijk bod 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking

Questions? 21-Nov-18 Johan Muskens TU/e Computer Science, System Architecture and Networking