1. Introduction to Boston University’s Scientific Computing Facility Doug Sondak sondak@bu.edu Scientific Computing and Visualization

2. Outline  Introduction  Hardware  Software  Account Information  Tutorials  Some Basic Unix Commands  How to Run Batch Information Services & Technology 2 9/7/2015

3. Introduction  BU’s Scientific Computing Facility (SCF) addresses high-performance computing and visualization needs of the BU community  Large resource requirements  CPU time  Memory  One group used 3 centuries of CPU time last year! Information Services & Technology 3 9/7/2015

4. Overview of SCV  Scientific Computing and Visualization  Computing  high-performance, parallel computers (SCF)  support  parallelization  optimization  tutorials  Visualization  support to create graphics, from publication figures to interactive 3D displays and movies  15’x8’ tiled display for 3D visualization  tutorials Information Services & Technology 4 9/7/2015

5. Current Hardware - BladeCenter  IBM BladeCenter  katana  currently 272 processors  heterogeneous mixture of blades  Intel Xeon and AMD Opteron  2.4-3.0 GHz  blades contain 4 or 8 processors  May use up to 32 processors at a time  parallel or serial  Two blades each have 8 processors and 96 GB memory Information Services & Technology 5 9/7/2015

6. Current Hardware – Blue Gene  IBM Blue Gene  1024 nodes, each with 2 processors  PowerPC 440 processors  700 MHz  32-bit  only runs MPI codes  each processor is relatively slow, so you need good scalability to make it worthwhile  May use 1024 processors during the day, all 2048 processors at night Information Services & Technology 6 9/7/2015

7. Current Hardware - pSeries  IBM p655  shared-memory machines  Power4 processors  9 machines, each with 8 processors  6 machines 1.1 GHz, 16 GB memory  3 machines 1.7 GHz, 8 GB memory  login node is twister Information Services & Technology 7 9/7/2015

8. Software  All machines run under Unix  C, C++, Fortran, Java, Perl, Python, Tcl  Matlab  all machines except Blue Gene  PCT (Parallel Computing Toolbox)  Maple  all machines except Blue Gene  Mathematica  all machines except Blue Gene Information Services & Technology 8 9/7/2015

9. Software (cont’d)  Gauss  blade center, pSeries  R  all machines except Blue Gene  VTK  Paraview  Maya Information Services & Technology 9 9/7/2015

10. Accounts and Policies  Accounts and allocations are based on “projects”  Must be faculty or research staff to apply for projects  Project PI then adds accounts for students, collaborators, etc.  May assign post-doc or admin. staff as “administrative contact”  Apply for project at http://www.bu.edu/tech/accounts/special/research/accounts/applications/  Click on “Boston University faculty and research staff may apply for a new project.”  If you’re not sure how much time to ask for, 1000 katana hours is considered modest  The form has fields for requesting time on each machine, but the time is actually all in one pot, and can be spent on any machine  This will give you accounts on all machines except for Blue Gene Information Services & Technology 10 9/7/2015

11. Accounts and Policies (cont’d)  For Blue Gene account, once you are awarded a “regular” account go back to  http://www.bu.edu/tech/accounts/special/research/accounts/applications/  There is a paragraph starting with “Blue Gene accounts.” Click on the “SCF User Information Page” link.  You will be requested for your user name and password  Click on the “Update Personal Information” link Information Services & Technology 11 9/7/2015

12. Disk Space  The disk space in your home directory is minimal  Most project PI’s request a “project” directory  project directory contains disk space allocated to the specific project  once your account has been activated go back to http://www.bu.edu/tech/accounts/special/research/accounts/applications/ and click “request a Project Disk Space allocation.”  a request of a few GB will be rubber-stamped  large requests (hundreds of GB) will require stronger justification Information Services & Technology 12 9/7/2015

13. “Buy-In” Nodes  Departments/groups may purchase their own BladeCenter nodes  alternative to buying and administering their own machines  Group that owns node(s) gets priority for batch runs  can be restricted to specific users if desired  May also have log-in node(s) for interactive use  log-in node restricted to specified groups/users Information Services & Technology 13 9/7/2015

14. Services  SCV has 3 components: 1.system administration  excellent administrators!  can often accommodate special requests 2.visualization  can help with creating visualizations, including animations, for presentations, publication, and display wall  wall has 3D capability Information Services & Technology 14 9/7/2015

15. Services (cont’d) 3.high-performance computing  code parallelization and optimization  MPI  OpenMP  PCT (Matlab)  help with any large-scale computing issues  algorithms  coding  There is no charge for SCV services Information Services & Technology 15 9/7/2015

16. Tutorials  SCV offers tutorials on a variety of computing and visualization topics  If you have interest in an area that we don’t presently cover, let us know!  Current offerings are in three areas  Programming  High-Performance Computing  Visualization Information Services & Technology 16 9/7/2015

17. Tutorials (cont’d)  Programming  Intro to Matlab  Intro to Fortran  Intro to C/C++  High-Performance Computing  Code Tuning (C, Fortran)  Tuning Matlab Codes  Intro to MPI  Parallelization with OpenMP  Matlab Parallel Computing Toolbox (PCT) Information Services & Technology 17 9/7/2015

18. Tutorials (3)  Visualization  Intro to Scientific Visualization  Scientific Visualization using Paraview  Scientific Visualization using VTK  Scientific Visualization using Matlab  Graphics Programming in C/C++: OpenGL and OpenSceneGraph  Graphics and Images for Presentation and Publication Information Services & Technology 18 9/7/2015

19. Some Basic Unix Information Services & Technology 19 9/7/2015

20. Login  From a terminal (e.g., Xwin-32, putty) ssh katana  this requests a “secure shell” on katana  you’ll be prompted for your username and password  You’re now logged on to the blade server “katana”  You’ll see something like “katana:~ %”  this is a Unix prompt, which is where you type in commands Information Services & Technology 20 9/7/2015

21. Directories  “directories” are analogous to PC “folders”  delimited by / rather than \ as on a PC  when you first log in, current directory is called your “home” directory  directory where you are located at any given time is your “working” directory  pwd means “print working directory”  type pwd  to create a subdirectory, use mkdir (make directory)  type mkdir sub1, where sub1 is the subdirectory name Information Services & Technology 21 9/7/2015

22. File Management  ls lists all files and directories in your current directory  type ls  you should see your new directory listed  some shorthand directory names: ~home directory.current directory..one level above current directory  cd dirname (change directory) moves you to dirname  type cd sub1  type pwd Information Services & Technology 22 9/7/2015

23. File Management (cont’d)  type ls ~  lists everything in your home directory  type ls..  since you’re one directory below your home directory, this is the same as ls ~  cp from to will copy a file  from is the file name you’re copying from  to is either a file name or a directory  if it’s a file name, the copy will be given that name  if it’s a directory, the file will retain the old name and be placed in the specified directory Information Services & Technology 23 9/7/2015

24. File Management (3)  copy the file /scratch/sondak/emacs_file to your sub1 directory  cp /scratch/sondak/emacs_file.  we will use it later  a path is a full set of directories leading to a file or directory  e.g., when you type pwd, it shows the path to your current directory  file suffixes generally don’t have any special meanings as they do under Windows  there are some cases where they do have meaning, such as.m for a Matlab program Information Services & Technology 24 9/7/2015

25. Emacs  text files are created with an editor  kind of like a fancy “notepad”  two most common Unix editors are vi and emacs  since I use emacs, that’s what we’ll use here  we’ll introduce some emacs commands using the file emacs_file  two ways to enter commands  menu  shortcuts Information Services & Technology 25 9/7/2015

26. Emacs Exercise  make sure you’re in your sub1 directory  type emacs emacs_file  the file will appear in an emacs window  try migrating around the file with the arrow keys  you can also navigate by clicking on the desired location  click on the 0 in 0.282  to delete a character use Delete button  hit Delete 3 times to delete 0.2  type 1.3  you’ve now changed 0.282 to 1.382 Information Services & Technology 26 9/7/2015

27. Emacs Exercise (cont’d)  Highlight 0.288 with the mouse  Don’t include spaces before or after  Edit -> Cut will delete the highlighted characters  can also use CTL-w  Oh no, we made a mistake! Edit -> Undo undoes the previous command. Try it; 0.288 should reappear.  can also use CTL-x u  can keep repeating undo, and it will undo last command, second-to- last command, third-to-last command, etc. Information Services & Technology 27 9/7/2015

28. Emacs Exercise (3)  The point is the location on the left side of the cursor; i.e., between the character on which the cursor resides and the character to its left.  The mark is similar to the point, but it’s location is set explicitly (i.e., doesn’t move with cursor). Information Services & Technology 28 9/7/2015

29. Emacs Exercise (4)  Suppose we want to delete all the characters between (inclusively) 0.288 and 0.407.  Set the cursor on the 0 in 0.288.  To set the mark, CTL-spacebar  The note “Mark set” will appear at the bottom of the screen.  Move the cursor to the right of 0.407  We place it to the right of the 7 rather than on it because the point is always to the left of the cursor  Edit -> Cut will delete all characters between the mark and the point  shortcut is CTL-w Information Services & Technology 29 9/7/2015

30. Emacs Exercise (5)  For now, let’s put the characters back in by using Edit -> Undo (or CTL-x u)  Move the cursor to the start of the current line using CTL-a  (CTL-e moves to end of current line)  Delete (“kill”) the line with CTL-k  Another CTL-k deletes the newline character Information Services & Technology 30 9/7/2015

31. Emacs Exercise (6)  “Meta” key is the Esc key  We will use M to refer to the Meta key  The Meta key is hit prior to the subsequent key(s), not simultaneously as with CTL  Place the cursor at the top of the file  M-> will move the cursor to the bottom of the file (don’t confuse notation with arrows I’m using for menu notation; this is “Meta greater-than”)  M-< will move it back to the top Information Services & Technology 31 9/7/2015

32. Emacs Exercise (7)  Whatever we deleted last is available in a buffer  Move the cursor to the beginning of the “M1rel” line  Edit -> Paste “yanks” the current buffer  can also use CTL-y Information Services & Technology 32 9/7/2015

33. Emacs Exercise (8)  CTL-h gets you into the Emacs help function  at prompt type ? to list types of help  I often use “a” (apropos)  Type CTL-h  Then type a at the prompt  Then type “kill” at the prompt  The window will split, listing all Emacs commands that have something to do with the word kill Information Services & Technology 33 9/7/2015

34. Emacs Exercise (9)  Type CTL-x 1 to get you back to the un-split screen  Split screens can be very handy, especially for cutting from one file and pasting in another  Type CTL-x 2 to split screen horizontally  can open different files in each half of the screen  toggle across split with CTL-o (“other”)  CTL-x 0 eliminates the half screen in which the cursor currently resides (try it)  this is a zero, not an “oh” Information Services & Technology 34 9/7/2015

35. Emacs Exercise (10)  To save the modified file, File -> Save (current buffer)  can also use CTL-x CTL-s  A note will appear at the bottom of the window saying the file has been saved  To save it under a new name, File -> Save Buffer As…  can also use CTL-x CTL-w  You’ll be prompted for the name at the bottom of the screen  Note that when you get these kinds of prompts, you can edit them using emacs commands  Type a file name and then move back and forth with arrow keys  File -> Exit Emacs (or CTL-x CTL-c) to quit  Previous version will appear with ~ suffix Information Services & Technology 35 9/7/2015

36. Katana Batch System Information Services & Technology 36 9/7/2015

37. Katana Batch System  If your run will take less than 10 minutes of CPU time, you can run interactively  Longer runs are submitted to the batch system  Sun Grid Engine  May use up to 32 processors at a time  can be 32 serial runs  can be one 32-processor parallel run  can be combination  Serial runs are limited to 24 hours  Parallel runs are limited to 5 hours Information Services & Technology 37 9/7/2015

38. Katana Batch System (cont’d)  Job Control File (JCF)  this is file you submit to batch system  line starting with #! defines Unix shell  lines starting with #$ are batch system commands  lines starting with # are comments  Here’s a “minimalist” JCF for a serial run: Information Services & Technology 38 9/7/2015 #!/bin/tcsh # run time limit hr:min:sec #$ -l h_rt=2:00:00 mycode

39. Katana Batch System (3)  JCF must have execute permission  chmod 755 myjcf  qsub myjcf submits your job to the queue  qstat –u username checks status of all your jobs Information Services & Technology 39 9/7/2015

40. Katana Batch System (4)  Create a trivial Matlab code with some output, e.g.,  I called mine “dumbcode.m”  Create a JCF Information Services & Technology 40 9/7/2015 a = 3; disp([‘a = ‘ int2str(a)]); #!/bin/tcsh # run time limit hr:min:sec #$ -l h_rt=0:02:00 matlab –r –nodisplay dumbcode

41. Katana Batch System (5)  Submit to batch system  When it’s done 2 files should be in your working directory  dumbcode.e######  dumbcode.o###### where ###### represents a 6-digit number  The.e file contains errors  The.o file contains output Information Services & Technology 41 9/7/2015

42. References  General SCV/SCF information  http://www.bu.edu/tech/research/ http://www.bu.edu/tech/research/  Blade Server  http://www.bu.edu/tech/research/computation/linux-cluster/katana-cluster/ http://www.bu.edu/tech/research/computation/linux-cluster/katana-cluster/  Account Application and Maintenance  http://www.bu.edu/tech/accounts/special/research/accounts/applications/ http://www.bu.edu/tech/accounts/special/research/accounts/applications/  Slides from Live SCV Tutorials  http://www.bu.edu/tech/research/training/presentations/list/ http://www.bu.edu/tech/research/training/presentations/list/  Web-based SCV Tutorials  http://www.bu.edu/tech/research/training/tutorials/list/ http://www.bu.edu/tech/research/training/tutorials/list/  Katana Batch System  http://www.bu.edu/tech/research/computation/linux-cluster/katana-cluster/runningjobs/ Information Services & Technology 42 9/7/2015

43. Human Help  If you ever have any questions, feel free to contact us  Doug Sondak, sondak@bu.edusondak@bu.edu  Kadin Tseng, kadin@bu.edukadin@bu.edu Information Services & Technology 43 9/7/2015