1. Welcome to our Nuclear Physics Computing System
Software
Hardware
Servers
login and interactive server
file transfer server
file server
online station
Batch system
Storage
2018/10/14
RCNP, Osaka Univ.

2. Information is here http://www.rcnp.osaka-u.ac.jp/Divisions/CN/cn2015/
English version is available partly.
main manual is at wiki (Japanese only)
2018/10/14
RCNP, Osaka Univ.

3. Software
Compilers
Intel Parallel Studio XE 2015 Cluster Edition(ifort/icc)
GCC (3.4.6, 4.1.2, 4.4.7, 4.9.3)
before use, run setup script for each version.
Libraries
LAPACK, BLAS
ROOT
CERNLIB
others
MPI
Intel MPI
Applications
emacs
LaTeX
2018/10/14
RCNP, Osaka Univ.

4. Hardware login server login-1, login-2
access to these servers in order to login miho from outside RCNP
interactive server miho-1, miho-2
interactive use
except heavy computing and file transfer
ftp server ftp-1, ftp-2
file transfer especially for heavy use
file server fs-1, fs-2
mainly for CIFS (SMB) server
online station aino-1, aino-2
for experiment at the cyclotron (DAQ)
batch system
2018/10/14
RCNP, Osaka Univ.

5. Interactive server (1) interactive use only
miho-1, miho-2
interactive login from outside RCNP
slogin
slogin
select miho-1 or miho-2 by MENU screen
interactive login from inside RCNP
slogin
slogin
interactive use only
DON’T USE long or heavy computing or file transfer
emacs may be runaway. please watch.
2018/10/14
RCNP, Osaka Univ.

6. Interactive server (2) Shell
login shell can be selected from tcsh and bash using “user management system”.
other shell can be invoked by hand.
zsh, mksh, sh, csh, ksh, dash
sx2name command
get username of miho from username of the super computer.
2018/10/14
RCNP, Osaka Univ.

7. file transfer server ftp-1, ftp-2
To avoid troubles concerning file transfer, file transfer severs are provided.
DON’T transfer files on miho for large files or long time.
File transfer
scp FILENAME2
scp FILENAME2
bbftp can be used (client and server)
the user computer needs to install bbftp software (free)
parallel file transfer supported and fast ( >600Mbps)
Please consult us for beginner
GridFTP is now prepareing.
2018/10/14
RCNP, Osaka Univ.

8. File server
fs-1, fs-2
have 2 x 10Gbps interfaces to fast access to clients and storage.
We achieved total performance of >8Gbps when we copy data from old system to current system using NFS.
NFS server for limited use.
CIFS (SMB) server
to access all files on miho
can be accessed from Windows, MAC and linux
even if you don’t use miho, it’s convenient to
share files between PCs
backup data on PCs.
can be written by document scanner (Xerox Multifunction Device)
2018/10/14
RCNP, Osaka Univ.

9. online station aino-1, aino-2 for experiment at the cyclotron (DAQ)
current experiment group has a priority.
have 10Gbps interface to fast access to clients and storage.
We achieved single performance of >1Gbps by scp.
have 8TB of local storage for emergency use
2018/10/14
RCNP, Osaka Univ.

10. batch system (1) hardware total 46 nodes, 1096 cores, 3136GB memory
normal nodes
E5-2697v GHz 12 cores x 2 CPU / node
24 cores x 44 nodes
64GB memory / node (2.6GB/core)
large memory nodes
E5-2680v GHz 10 cores x 2 CPU / node
20 cores x 2 nodes
160GB memory / node (20GB/core)
2018/10/14
RCNP, Osaka Univ.

11. batch system (2) job scheduler qsub, qdel, qstat,,,
default memory size is 1GB, which is intended to small value.
default cpu (core) is 1.
can use batch nodes interactively. ( qsub –I )
can send mail when start, end and abort.
parallel job can be executed using MPI.
scheduling by I/O requirement
qsub –l highio=1
The jobs which declare “highio” should be assigned only 1 job per node.
2018/10/14
RCNP, Osaka Univ.

12. storage (1) total capacity is 5PB physically. High performance
total 160 Gbps ( 20GB/s )
GPFS
High reliability
RAID 6 + hot spare
All components are redundant.
High function
snapshot
/home 1 snapshot / day
other preparing ( How do you need ?? )
backup is now preparing
/home 1 backup / 7 days
others ?? ( How do you need ?? )
2018/10/14
RCNP, Osaka Univ.

13. storage (2) current usage All file systems looks the same.
df –a -a is required to show all file systems.
quota
/home 150GB / user
other quota is effective but not set limits. you can see used space easily.
current usage
～１３００TB／５０００TB
2018/10/14
RCNP, Osaka Univ.

14. storage (3) each file systems home directory
/home/USERNAME quota 150GB / user
temporary area
/tmp don’t use intendedly. not shared
/tmp-common 10TB, shared by all machine.
/tmp-cifs 2TB. temporary area for cifs user.
mail/web
/Maildir/USERNAME
/HTMLpub/USERNAME
operation area
/miho common library, etc
/cn work area for CN group
/archive archived files
/compaq old files in compaq era
2018/10/14
RCNP, Osaka Univ.

15. storage (4) each file systems data area
/np1a experiment of the cyclotron (except cagra)
/np1a/cagra cagra experiment
/np1b LEPS
/np1c other
/np2 theory group
/acc accelerator group
/bnpc BNPC
2018/10/14
RCNP, Osaka Univ.

16. storage (5) each file systems super computer
500TB storage us used for JLDG project
miho mounts the disk of the super computer
The same directory structure as the super computer
/sc/rcnp/home
/sc/rcnp/short (/ext/rcnp/short)
/sc/rcnp/work (/ext/rcnp/work)
/sc/rcnp/work2 (/ext/rcnp/work2) not allocated for new user
/sc/rcnp/work3 (/ext/rcnp/work3) not allocated for new user
2018/10/14
RCNP, Osaka Univ.